Original: B I A DAM : F AST A DAPTIVE B ILEVEL O PTIMIZATION M ETHODS
Revised: B I A DAM : F AST A DAPTIVE B ILEVEL O PTIMIZATION M ETHODS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Bilevel optimization recently has attracted increased interest in machine learning due to its many applications such as hyper-parameter optimization and mate learning.
Revised: Bilevel optimization recently has attracted increased interest in machine learning due to its many applications such as hyper-parameter optimization and meta learning.

Original: Although many bilevel optimization methods recently have been proposed, these methods do not consider using adaptive learning rates.
Revised: Although many bilevel optimization methods recently have been proposed, these methods do not consider using adaptive learning rates.

Original: It is well known that adaptive learning rates can accelerate many optimization algorithms including (stochastic) gradient-based algorithms.
Revised: It is well known that adaptive learning rates can accelerate many optimization algorithms including (stochastic) gradient-based algorithms.

Original: To fill this gap, in the paper, we propose a novel fast adaptive bilevel framework for solving bilevel optimization problems that the outer problem is possibly nonconvex and the inner problem is strongly convex.
Revised: To fill this gap, in the paper, we propose a novel fast adaptive bilevel framework for solving bilevel optimization problems that the outer problem is possibly nonconvex and the inner problem is strongly convex.

Original: Our framework uses unified adaptive matrices including many types of adaptive learning rates, and can flexibly use the momentum and variance reduced techniques.
Revised: Our framework uses unified adaptive matrices including many types of adaptive learning rates, and can flexibly use the momentum and variance reduced techniques.

Original: In particular, we provide a useful convergence analysis framework for the bilevel optimization.
Revised: In particular, we provide a useful convergence analysis framework for the bilevel optimization.

Original: Specifically, we propose a fast singleloop adaptive bilevel optimization (BiAdam) algorithm based on the basic momentum technique, which achieves a sample complexity of ˜ O
Revised: Specifically, we propose a fast singleloop adaptive bilevel optimization (BiAdam) algorithm based on the basic momentum technique, which achieves a sample complexity of ˜ O

Original: ( ϵ − 4 ) for finding an ϵ -stationary point.
Revised: ( ϵ − 4 ) for finding an ϵ -stationary point.

Original: Meanwhile, we propose an accelerated version of BiAdam algorithm (VR-BiAdam) by using variance reduced technique, which reaches the best known sample complexity of ˜ O
Revised: Meanwhile, we propose an accelerated version of BiAdam algorithm (VR-BiAdam) by using variance reduced technique, which reaches the best known sample complexity of ˜ O

Original: ( ϵ − 3 ) without relying on large batch-size.
Revised: ( ϵ − 3 ) without relying on large batch-size.

Original: To the best of our knowledge, we first study the adaptive bilevel optimization methods with adaptive learning rates.
Revised: To the best of our knowledge, we first study the adaptive bilevel optimization methods with adaptive learning rates.

Original: Some experimental results on data hypercleaning and hyper-representation learning tasks demonstrate the efficiency of the proposed algorithms.
Revised: Some experimental results on data hypercleaning and hyper-representation learning tasks demonstrate the efficiency of the proposed algorithms.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Bilevel optimization is known as a class of popular hierarchical optimization, which has been applied to a wide range of machine learning problems such as hyperparameter optimization Shaban et al.
Revised: Bilevel optimization is known as a class of popular hierarchical optimization, which has been applied to a wide range of machine learning problems such as hyperparameter optimization Shaban et al.

Original: (2019), meta-learning Ji et al.
Revised: (2019), meta-learning Ji et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2021a)
Revised: (2021a)

Original: and policy optimization Hong et al.
Revised: and policy optimization Hong et al.

Original: In the paper, we consider solving the following stochastic bilevel optimization problem, defined as
Revised: In the paper, we consider solving the following stochastic bilevel optimization problem, defined as

Original: [Equation]
Revised: [Equation]

Original: where F ( x ) =
Revised: where F ( x ) =

Original: f ( x, y ∗ ( x )) =
Revised: f ( x, y ∗ ( x )) =

Original: E ξ (cid:2) f
Revised: E ξ (cid:2) f

Original: ( x, y ∗ ( x ); ξ ) (cid:3) is a differentiable and possibly nonconvex function, and g ( x, y )
Revised: ( x, y ∗ ( x ); ξ ) (cid:3) is a differentiable and possibly nonconvex function, and g ( x, y )

Original: =
Revised: =

Original: E ζ (cid:2) g ( x, y ; ζ ) (cid:3) is a differentiable and strongly convex function in variable y , and ξ and ζ are random variables follow unknown distributions D and M , respectively.
Revised: E ζ (cid:2) g ( x, y ; ζ ) (cid:3) is a differentiable and strongly convex function in variable y , and ξ and ζ are random variables follow unknown distributions D and M , respectively.

Original: Here X ⊆ R d and Y ⊆
Revised: Here X ⊆ R d and Y ⊆

Original: R p are convex closed sets.
Revised: R p are convex closed sets.

Original: Problem (1) involves many machine learning problems with a hierarchical structure, which include hyper-parameter optimization Franceschi et al.
Revised: Problem (1) involves many machine learning problems with a hierarchical structure, which include hyper-parameter optimization Franceschi et al.

Original: (2018), metalearning Franceschi et al.
Revised: (2018), metalearning Franceschi et al.

Original: policy optimization Hong et al.
Revised: policy optimization Hong et al.

Original: and neural network architecture search Liu et al.
Revised: and neural network architecture search Liu et al.

Original: Since bilevel optimization has been widely applied in machine learning, some works recently have been begun to study the bilevel optimization.
Revised: Since bilevel optimization has been widely applied in machine learning, some works recently have been begun to study the bilevel optimization.

Original: For example, Ghadimi & Wang (2018)
Revised: For example, Ghadimi & Wang (2018)

Original: ; Ji et al.
Revised: ; Ji et al.

Original: (2021) proposed a class of double-loop methods to solve the problem (1).
Revised: (2021) proposed a class of double-loop methods to solve the problem (1).

Original: However, to obtain an accurate estimate, the BSA in Ghadimi & Wang (2018) needs to solve the inner problem to a high accuracy, and the stocBiO in Ji et al.
Revised: However, to obtain an accurate estimate, the BSA in Ghadimi & Wang (2018) needs to solve the inner problem to a high accuracy, and the stocBiO in Ji et al.

Original: (2021) requires large batch-sizes in solving the inner problem.
Revised: (2021) requires large batch-sizes in solving the inner problem.

Original: [Table]
Revised: [Table]

Original: Subsequently, Hong et al.
Revised: Subsequently, Hong et al.

Original: (2020) proposed a class of single-loop methods to solve the bilevel problems.
Revised: (2020) proposed a class of single-loop methods to solve the bilevel problems.

Original: More recently, Chen et al.
Revised: More recently, Chen et al.

Original: ; Khanduri et al.
Revised: ; Khanduri et al.

Original: ; Guo & Yang (2021); Yang et al.
Revised: ; Guo & Yang (2021); Yang et al.

Original: (2021) presented some accelerated single-loop methods by using the momentum-based variance reduced technique of STORM Cutkosky & Orabona (2019).
Revised: (2021) presented some accelerated single-loop methods by using the momentum-based variance reduced technique of STORM Cutkosky & Orabona (2019).

Original: Although these methods can effectively solve the bilevel problems, they do not consider using the adaptive learning rates and only consider the bilevel problems under unconstrained setting.
Revised: Although these methods can effectively solve the bilevel problems, they do not consider using the adaptive learning rates and only consider the bilevel problems under unconstrained setting.

Original: Since using generally different learning rates for the inner and outer problems to ensure the convergence of bilevel optimization problems, we will consider using different adaptive learning rates for the inner and outer problems with convergence guarantee.
Revised: Since using generally different learning rates for the inner and outer problems to ensure the convergence of bilevel optimization problems, we will consider using different adaptive learning rates for the inner and outer problems with convergence guarantee.

Original: Clearly, this can not follow the exiting adaptive methods for single-level problems.
Revised: Clearly, this can not follow the exiting adaptive methods for single-level problems.

Original: Thus, there exists a natural question:
Revised: Thus, there exists a natural question:

Original: How to design the effective optimization methods with adaptive learning rates for the bilevel problems ?
Revised: How to design the effective optimization methods with adaptive learning rates for the bilevel problems ?

Original: In the paper, we provide an affirmative answer to this question and propose a class of fast singleloop adaptive bilevel optimization methods based on unified adaptive matrices, which including many types of adaptive learning rates.
Revised: In the paper, we provide an affirmative answer to this question and propose a class of fast singleloop adaptive bilevel optimization methods based on unified adaptive matrices, which including many types of adaptive learning rates.

Original: Moreover, our framework can flexibly use the momentum and variance reduced techniques.
Revised: Moreover, our framework can flexibly use the momentum and variance reduced techniques.

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: 1) We propose a fast single-loop adaptive bilevel optimization algorithm (BiAdam) based on the basic momentum technique, which achieves a sample complexity of ˜ O
Revised: 1) We propose a fast single-loop adaptive bilevel optimization algorithm (BiAdam) based on the basic momentum technique, which achieves a sample complexity of ˜ O

Original: ( ϵ − 4 ) for finding an ϵ -stationary point.
Revised: ( ϵ − 4 ) for finding an ϵ -stationary point.

Original: Meanwhile, we propose a single-loop accelerated version of BiAdam algorithm (VRBiAdam) by using the momentum-based variance reduced technique, which reaches the best known sample complexity of ˜ O
Revised: Meanwhile, we propose a single-loop accelerated version of BiAdam algorithm (VRBiAdam) by using the momentum-based variance reduced technique, which reaches the best known sample complexity of ˜ O

Original: ( ϵ − 3 ) .
Revised: ( ϵ − 3 ) .

Original: Moreover, we provide a useful convergence analysis framework for both the constrained and unconstrained bilevel programming under some mild conditions (Please see Table 1).
Revised: Moreover, we provide a useful convergence analysis framework for both the constrained and unconstrained bilevel programming under some mild conditions (Please see Table 1).

Original: The experimental results on hyper-parameter learning demonstrate the efficiency of the proposed algorithms.
Revised: The experimental results on hyper-parameter learning demonstrate the efficiency of the proposed algorithms.

Original:  P RELIMINARIES 2. N OTATIONS
Revised:  P RELIMINARIES 2. N OTATIONS

Original: U{ 1 , 2 , · · · , K } denotes a uniform distribution over a discrete set { 1 , 2 , · · · , K } .
Revised: U{ 1 , 2 , · · · , K } denotes a uniform distribution over a discrete set { 1 , 2 , · · · , K } .

Original: ∥ · ∥ denotes the ℓ 2 norm for vectors and spectral norm for matrices.
Revised: ∥ · ∥ denotes the ℓ 2 norm for vectors and spectral norm for matrices.

Original: ⟨ x, y ⟩ denotes the inner product of two vectors x and y .
Revised: ⟨ x, y ⟩ denotes the inner product of two vectors x and y .

Original: For vectors x and y , x r ( r > 0) denotes the element-wise power operation, x/y denotes the element-wise division and max( x, y ) denotes the element-wise maximum.
Revised: For vectors x and y , x r ( r > 0) denotes the element-wise power operation, x/y denotes the element-wise division and max( x, y ) denotes the element-wise maximum.

Original: I d denotes a d -dimensional identity matrix.
Revised: I d denotes a d -dimensional identity matrix.

Original: A ≻ 0 denotes that the matrix A is positive definite.
Revised: A ≻ 0 denotes that the matrix A is positive definite.

Original: Given function f ( x, y ) , f ( x, · ) denotes function w.r.t.
Revised: Given function f ( x, y ) , f ( x, · ) denotes function w.r.t.

Original: the second variable with fixing x , and f ( · , y ) denotes function w.r.t.
Revised: the second variable with fixing x , and f ( · , y ) denotes function w.r.t.

Original: the first variable with fixing y .
Revised: the first variable with fixing y .

Original: a = O ( b ) denotes that a ≤ Cb for some constant C > 0 .
Revised: a = O ( b ) denotes that a ≤ Cb for some constant C > 0 .

Original: The notation ˜ O ( · ) hides logarithmic terms.
Revised: The notation ˜ O ( · ) hides logarithmic terms.

Original: Given a convex closed set X , we define a projection operation to X asP X ( z ) =
Revised: Given a convex closed set X , we define a projection operation to X as

Original: arg min x ∈X 12 ∥ x − z ∥ 2 .
Revised: [Equation]

Original: 2. S OME M ILD A SSUMPTIONS
Revised: 2. S OME M ILD A SSUMPTIONS

Original: In this subsection, we give some mild assumptions on the problem (1).
Revised: In this subsection, we give some mild assumptions on the problem (1).

Original: Assumption 1 For any x and ζ , g ( x, y ; ζ ) is L g -smooth and µ -strongly convex function, i.e.,
Revised: Assumption 1. For any x and ζ , g ( x, y ; ζ ) is L g -smooth and µ -strongly convex function, i.e.,

Original: L g I p ⪰ ∇ 2 yy g ( x, y ; ζ ) ⪰ µI p .
Revised: L g I p ⪰ ∇ 2 yy g ( x, y ; ζ ) ⪰ µI p .

Original: Assumption 2 For functions f ( x, y ) and g ( x, y ) for all x ∈ X and y ∈ Y , we assume the following conditions hold: ∇ x f ( x, y ) and ∇ y f ( x, y ) are L f -Lipschitz continuous, ∇ y g
Revised: Assumption 2. For functions f ( x, y ) and g ( x, y ) for all x ∈ X and y ∈ Y , we assume the following conditions hold: ∇ x f ( x, y ) and ∇ y f ( x, y ) are L f -Lipschitz continuous, ∇ y g

Original: ( x, y ) is L g -Lipschitz continuous, ∇ 2 xy g
Revised: ( x, y ) is L g -Lipschitz continuous, ∇ 2 xy g

Original: ( x, y ) is L gxy -Lipschitz continuous, ∇ 2 yy g ( x, y ) is L gyy -Lipschitz continuous.
Revised: ( x, y ) is L gxy -Lipschitz continuous, ∇ 2 yy g ( x, y ) is L gyy -Lipschitz continuous.

Original: For example, for all x, x 1 , x 2 ∈ X and y, y 1 , y 2 ∈ Y , we have ∥∇ x f ( x 1 , y ) − ∇ x f ( x 2 , y )
Revised: For example, for all x, x 1 , x 2 ∈ X and y, y 1 , y 2 ∈ Y , we have ∥∇ x f ( x 1 , y ) − ∇ x f ( x 2 , y )

Original: ∥
Revised: ∥

Original: ≤ L f ∥ x 1 − x 2 ∥ , ∥∇ x f ( x, y 1 ) − ∇ x f ( x, y 2 ) ∥
Revised: ≤ L f ∥ x 1 − x 2 ∥ , ∥∇ x f ( x, y 1 ) − ∇ x f ( x, y 2 ) ∥

Original: ≤ L f ∥ y 1 − y 2 ∥ .
Revised: ≤ L f ∥ y 1 − y 2 ∥ .

Original: Assumption 3 For functions f ( x, y ; ξ ) and g ( x, y ; ζ ) for all x ∈ X , y ∈ Y , ξ and ζ , we assume the following conditions hold: ∇ x f ( x, y ; ξ ) and ∇ y f ( x, y ; ξ ) are L f -Lipschitz continuous, ∇ y g
Revised: Assumption 3. For functions f ( x, y ; ξ ) and g ( x, y ; ζ ) for all x ∈ X , y ∈ Y , ξ and ζ , we assume the following conditions hold: ∇ x f ( x, y ; ξ ) and ∇ y f ( x, y ; ξ ) are L f -Lipschitz continuous, ∇ y g

Original: ( x, y ; ζ ) is L g -Lipschitz continuous, ∇ 2 xy g
Revised: ( x, y ; ζ ) is L g -Lipschitz continuous, ∇ 2 xy g

Original: ( x, y ; ζ ) is L gxy -Lipschitz continuous, ∇ 2 yy g ( x, y ; ζ ) is L gyy -Lipschitz continuous.
Revised: ( x, y ; ζ ) is L gxy -Lipschitz continuous, ∇ 2 yy g ( x, y ; ζ ) is L gyy -Lipschitz continuous.

Original: For example, for all x, x 1 , x 2 ∈ X and y, y 1 , y 2 ∈ Y , we have ∥∇ x f ( x 1 , y ; ξ ) − ∇ x f ( x 2 , y ; ξ ) ∥
Revised: For example, for all x, x 1 , x 2 ∈ X and y, y 1 , y 2 ∈ Y , we have ∥∇ x f ( x 1 , y ; ξ ) − ∇ x f ( x 2 , y ; ξ ) ∥

Original: ≤ L f ∥ x 1 − x 2 ∥ , ∥∇ x f ( x, y 1 ; ξ ) − ∇ x f ( x, y 2 ; ξ ) ∥
Revised: ≤ L f ∥ x 1 − x 2 ∥ , ∥∇ x f ( x, y 1 ; ξ ) − ∇ x f ( x, y 2 ; ξ ) ∥

Original: ≤ L f ∥ y 1 − y 2 ∥ .
Revised: ≤ L f ∥ y 1 − y 2 ∥ .

Original: Assumption 4 The partial derivatives ∇ y f ( x, y ) and ∇ 2 xy g ( x, y ) are bounded, i.e., ∥∇ y f ( x, y ) ∥ 2 ≤ C 2 fy and ∥∇ 2 xy g ( x, y )
Revised: Assumption 4. The partial derivatives ∇ y f ( x, y ) and ∇ 2 xy g ( x, y ) are bounded, i.e., ∥∇ y f ( x, y ) ∥ 2 ≤ C 2 fy and ∥∇ 2 xy g ( x, y )

Original: ∥ 2 ≤ C 2 gxy .
Revised: ∥ 2 ≤ C 2 gxy .

Original: Assumption 5 Stochastic functions f ( x, y ; ξ ) and g ( x, y ; ζ ) have unbiased stochastic partial derivatives with bounded variance, e.g.,[Equation] [Equation]
Revised: Assumption 5. Stochastic functions f ( x, y ; ξ ) and g ( x, y ; ζ ) have unbiased stochastic partial derivatives with bounded variance, e.g.,

Original: [Equation]
Revised: [Equation]

Original: The same assumptions hold for ∇ y f ( x, y ; ξ ) , ∇ y g
Revised: The same assumptions hold for ∇ y f ( x, y ; ξ ) , ∇ y g

Original: ( x, y ; ζ ) , ∇ 2 xy g ( x, y ; ζ ) and ∇ 2 yy g ( x, y ; ζ ) .
Revised: ( x, y ; ζ ) , ∇ 2 xy g ( x, y ; ζ ) and ∇ 2 yy g ( x, y ; ζ ) .

Original: Assumptions 1-5 are commonly used in stochastic bilevel optimization problems Ghadimi & Wang (2018); Hong et al.
Revised: Assumptions 1-5 are commonly used in stochastic bilevel optimization problems Ghadimi & Wang (2018); Hong et al.

Original: ; Ji et al.
Revised: ; Ji et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: ; Khanduri et al.
Revised: ; Khanduri et al.

Original: Note that Assumption 3 is clearly stricter than Assumption 2.
Revised: Note that Assumption 3 is clearly stricter than Assumption 2.

Original: For example, given Assumption 3, we have ∥∇ x f ( x 1 , y ) − ∇ x f ( x 2 , y ) ∥ =
Revised: For example, given Assumption 3, we have ∥∇ x f ( x 1 , y ) − ∇ x f ( x 2 , y ) ∥ =

Original: ∥ E [ ∇ x f ( x 1 , y ; ξ ) − ∇ x f ( x 2 , y ; ξ )] ∥ ≤ E ∥∇ x f ( x 1 , y ; ξ ) − ∇ x f ( x 2 , y ; ξ ) ∥
Revised: ∥ E [ ∇ x f ( x 1 , y ; ξ ) − ∇ x f ( x 2 , y ; ξ )] ∥ ≤ E ∥∇ x f ( x 1 , y ; ξ ) − ∇ x f ( x 2 , y ; ξ ) ∥

Original: ≤ L f ∥ x 1 − x 2 ∥∥ for any x, y, ξ .
Revised: ≤ L f ∥ x 1 − x 2 ∥∥ for any x, y, ξ .

Original: At the same time, based on Assumptions 4-5, we also have ∥∇ y f
Revised: At the same time, based on Assumptions 4-5, we also have ∥∇ y f

Original: ( x, y ; ξ ) ∥ 2 = ∥∇ y f
Revised: ( x, y ; ξ ) ∥ 2 = ∥∇ y f

Original: ( x, y ; ξ ) − ∇ y f
Revised: ( x, y ; ξ ) − ∇ y f

Original: ( x, y ) − ∇ y f
Revised: ( x, y ) − ∇ y f

Original: ( x, y ) ∥ 2 ≤ 2 ∥∇ y f
Revised: ( x, y ) ∥ 2 ≤ 2 ∥∇ y f

Original: ( x, y ; ξ ) − ∇ y f
Revised: ( x, y ; ξ ) − ∇ y f

Original: ( x, y ) ∥ 2 +2 ∥∇ y f
Revised: ( x, y ) ∥ 2 +2 ∥∇ y f

Original: ( x, y ) ∥ 2 ≤ 2 σ 2 +2 C 2 fy and ∥∇ 2 xy g ( x, y ; ζ ) ∥ 2 ≤ 2 σ 2 +2 C 2 gxy .
Revised: ( x, y ) ∥ 2 ≤ 2 σ 2 +2 C 2 fy and ∥∇ 2 xy g ( x, y ; ζ ) ∥ 2 ≤ 2 σ 2 +2 C 2 gxy .

Original: Thus we argue that under Assumption 5, the bounded ∇ y f ( x, y ) and ∇ 2 xy g ( x, y ) are not milder than the bounded ∇ y f
Revised: Thus we argue that under Assumption 5, the bounded ∇ y f ( x, y ) and ∇ 2 xy g ( x, y ) are not milder than the bounded ∇ y f

Original: ( x, y ; ξ ) and ∇ 2 xy g ( x, y ; ζ ) for all ξ and ζ .
Revised: ( x, y ; ξ ) and ∇ 2 xy g ( x, y ; ζ ) for all ξ and ζ .

Original: 2. B ILEVEL O PTIMIZATION
Revised: 2. B ILEVEL O PTIMIZATION

Original: In this subsection, we review the basic first-order method for solving the problem (1).
Revised: In this subsection, we review the basic first-order method for solving the problem (1).

Original: Naturally, we give the following iteration to update the variables x, y : at the t -th step
Revised: Naturally, we give the following iteration to update the variables x, y : at the t -th step

Original: [Equation]
Revised: [Equation]

Original:  x t , y ∗ ( x t )) (cid:17) , where λ > 0 and γ > 0 denote the step sizes.
Revised: ∇ x f ( x t , y ∗ ( x t )) (cid:17) , where λ > 0 and γ > 0 denote the step sizes.

Original: Clearly, if there does not exist a closed form solution of the inner problem in the problem (1), i.e., y t +1 ̸
Revised: Clearly, if there does not exist a closed form solution of the inner problem in the problem (1), i.e., y t +1 ̸

Original: = y ∗ ( x t ) , we can not easily obtain the gradient ∇ F ( x t ) = ∇ f
Revised: = y ∗ ( x t ) , we can not easily obtain the gradient ∇ F ( x t ) = ∇ f

Original: ( x t , y ∗ ( x t )) .
Revised: ( x t , y ∗ ( x t )) .

Original: Thus, one of key points in solving the problem (1) is to estimate the gradient ∇ F ( x t ) .
Revised: Thus, one of key points in solving the problem (1) is to estimate the gradient ∇ F ( x t ) .

Original: Lemma 1 (Lemma 2.1 in Ghadimi & Wang (2018))
Revised: Lemma 1. (Lemma 2.1 in Ghadimi & Wang (2018))

Original: Under the above Assumption 2, we have, for any x ∈ X
Revised: Under the above Assumption 2, we have, for any x ∈ X

Original: [Equation]
Revised: [Equation]

Original: From the above Lemma 1, it is natural to use the following form to estimate ∇ F ( x ) , defined as,
Revised: From the above Lemma 1, it is natural to use the following form to estimate ∇ F ( x ) , defined as,

Original: [Equation]
Revised: [Equation]

Original: Note that although the inner problem of the problem (1) is a constrained optimization, we assume that the optimal condition of the inner problem still is ∇ y g
Revised: Note that although the inner problem of the problem (1) is a constrained optimization, we assume that the optimal condition of the inner problem still is ∇ y g

Original: ( x, y ∗ ( x ))
Revised: ( x, y ∗ ( x ))

Original: = 

Revised: =

Original: and y ∗ ( x ) ∈ Y .
Revised: and y ∗ ( x ) ∈ Y .

Original: Lemma 2 (Lemma 2.2 in Ghadimi & Wang (2018))
Revised: Lemma 2. (Lemma 2.2 in Ghadimi & Wang (2018))

Original: Under the above Assumptions (1, 2, 4), for all x, x 1 , x 2 ∈ X and y ∈ Y , we have
Revised: Under the above Assumptions (1, 2, 4), for all x, x 1 , x 2 ∈ X and y ∈ Y , we have

Original: [Equation]
Revised: [Equation]

Original: where L y =
Revised: where L y =

Original: L f + L f C gxy /µ + C fy (cid:0) L gxy /µ
Revised: L f + L f C gxy /µ + C fy (cid:0) L gxy /µ

Original: + L gyy C gxy /µ 2 (cid:1) , κ = C gxy /µ , and L =
Revised: + L gyy C gxy /µ 2 (cid:1) , κ = C gxy /µ , and L =

Original: L f +( L f +
Revised: L f +( L f +

Original: L y )
Revised: L y )

Original: C gxy /µ + C fy (cid:0) L gxy
Revised: C gxy /µ + C fy (cid:0) L gxy

Original: /µ
Revised: /µ

Original: + L gyy C gxy /µ 2 (cid:1) .
Revised: + L gyy C gxy /µ 2 (cid:1) .

Original: For the stochastic bilevel optimization, Yang et al.
Revised: For the stochastic bilevel optimization, Yang et al.

Original: ; Hong et al.
Revised: ; Hong et al.

Original: (2020) provided a stochastic estimator ∇ F ( x ) as follows:
Revised: (2020) provided a stochastic estimator ∇ F ( x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where ϑ > 0 and Q ≥ 1 .
Revised: where ϑ > 0 and Q ≥ 1 .

Original: Here S = (cid:8) ξ, ζ, ζ 1 , · · · ζ Q (cid:9) , where ξ is drawn from distribution D , and { ζ, ζ 1 , · · · ζ Q } are drawn from distribution M .
Revised: Here S = (cid:8) ξ, ζ, ζ 1 , · · · ζ Q (cid:9) , where ξ is drawn from distribution D , and { ζ, ζ 1 , · · · ζ Q } are drawn from distribution M .

Original:  A DAPTIVE B ILEVEL O PTIMIZATION M ETHODS
Revised:  A DAPTIVE B ILEVEL O PTIMIZATION M ETHODS

Original: In this section, we propose a class of fast single-loop adaptive bilevel optimization methods to solve the problem equation 1.
Revised: In this section, we propose a class of fast single-loop adaptive bilevel optimization methods to solve the problem equation 1.

Original: Specifically, our methods adopt the universal adaptive learning rates as in
Revised: Specifically, our methods adopt the universal adaptive learning rates as in

Original: Huang et al.
Revised: Huang et al.

Original: Moreover, our methods can be flexibly incorporate the momentum and variance reduced techniques.
Revised: Moreover, our methods can be flexibly incorporate the momentum and variance reduced techniques.

Original: 3. B I A DAM A LGORITHM
Revised: 3. B I A DAM A LGORITHM

Original: In this subsection, we propose a fast single-loop adaptive bilevel optimization method (BiAdam) based on the basic momentum technique.
Revised: In this subsection, we propose a fast single-loop adaptive bilevel optimization method (BiAdam) based on the basic momentum technique.

Original: Algorithm 1 shows the algorithmic framework of our
Revised: Algorithm 1 shows the algorithmic framework of our

Original: BiAdam algorithm.
Revised: BiAdam algorithm.

Original: At the line 4 of Algorithm 1, we generate the adaptive matrices A t and B t for updating variables x and y , respectively.
Revised: At the line 4 of Algorithm 1, we generate the adaptive matrices A t and B t for updating variables x and y , respectively.

Original: Specifically, we use the general adaptive matrix
Revised: Specifically, we use the general adaptive matrix

Original: A t ⪰ ρI
Revised: A t ⪰ ρI

Original: d ( ρ > 0) for variable x , and the global adaptive matrix B t = b t
Revised: d ( ρ > 0) for variable x , and the global adaptive matrix B t = b t

Original: I p ( b t > 0) .
Revised: I p ( b t > 0) .

Original: For example, we can generate the matrix A t as the Adam Kingma & Ba (2014), and generate the matrix B t as a novel version of AdaGrad-Norm
Revised: For example, we can generate the matrix A t as the Adam Kingma & Ba (2014), and generate the matrix B t as a novel version of AdaGrad-Norm

Original: Ward et al.
Revised: Ward et al.

Original: (2019), defined as
Revised: (2019), defined as

Original: [Equation]
Revised: [Equation]

Original: where α, β ∈ (0 , 1) and ρ > 0 , ε > 0 .
Revised: where α, β ∈ (0 , 1) and ρ > 0 , ε > 0 .

Original: At the lines 5-6 of Algorithm 1, we use the generalized projection gradient iteration with Bregman distance Censor & Zenios (1992); Huang et al.
Revised: At the lines 5-6 of Algorithm 1, we use the generalized projection gradient iteration with Bregman distance Censor & Zenios (1992); Huang et al.

Original: (2021) to update the variables x and y , respectively.
Revised: (2021) to update the variables x and y , respectively.

Original: When X = R d and Y = R p , i.e., unconstrained optimization problem (1), we have x t +

Revised: When X = R d and Y = R p , i.e., unconstrained optimization problem (1), we have x t +

Original: = x
Revised: = x

Original: t − γη t
Revised: t − γη t

Original: A − 1 t w t and y t
Revised: A − 1 t w t and y t

Original: +1 =
Revised: +1 =

Original: y t − λη t B − 1 t v t .
Revised: y t − λη t B − 1 t v t .

Original: At the line 7 of Algorithm 1, we draw K + 1 independent samples ¯ ξ = { ξ, ζ 0 , ζ 1 , · · · , ζ K − 1 } from distributions D and M , then we define a stochastic gradient estimator as in Khanduri et al.
Revised: At the line 7 of Algorithm 1, we draw K + 1 independent samples ¯ ξ = { ξ, ζ 0 , ζ 1 , · · · , ζ K − 1 } from distributions D and M , then we define a stochastic gradient estimator as in Khanduri et al.

Original: [Equation]
Revised: [Equation]

Original: where K ≥ 1 and k ∼ U{ 0 , 1 , · · · , K − 1 } is a uniform random variable independent on ¯ ξ .
Revised: where K ≥ 1 and k ∼ U{ 0 , 1 , · · · , K − 1 } is a uniform random variable independent on ¯ ξ .

Original: In fact, the estimator (6) is a specific case of the above estimator (3).
Revised: In fact, the estimator (6) is a specific case of the above estimator (3).

Original: In practice, thus, we can use a tuning parameter ϑ ∈ (0 , 1 L g ] instead of 1 L g in the estimator (6) as in Yang et al.
Revised: In practice, thus, we can use a tuning parameter ϑ ∈ (0 , 1 L g ] instead of 1 L g in the estimator (6) as in Yang et al.

Original: Here we use the term KL g (cid:81) ki
Revised: Here we use the term KL g (cid:81) ki

Original: =1 (cid:0) I p − 1 L g ∇ 

Revised: =1 (cid:0) I p − 1 L g ∇

Original: yy g ( x, y ; ζ i ) (cid:1) to approximate the Hessian inverse, i.e., (cid:0) ∇ 2 yy g ( x, y ; ζ ) (cid:1) − 1 .
Revised: yy g ( x, y ; ζ i ) (cid:1) to approximate the Hessian inverse, i.e., (cid:0) ∇ 2 yy g ( x, y ; ζ ) (cid:1) − 1 .

Original: Clearly, the above ¯ ∇ f ( x, y, ¯ ξ ) is a biased estimator in estimating 
Revised: Clearly, the above ¯ ∇ f ( x, y, ¯ ξ ) is a biased estimator in estimating ¯ ∇ f ( x, y ) , i.e. E ¯

Original: [Equation]
Revised: [Equation]

Original: ̸ = ¯ ∇ f 
Revised: ¯ ∇ f ( x, y

Original:  ( x, y ) .
Revised: [Equation] ¯ ξ ) (cid:3) ̸ = ¯ ∇ f ( x, y ) .

Original: In the following, we give Lemma 3, which shows that the bias R ( x, y ) =
Revised: In the following, we give Lemma 3, which shows that the bias R ( x, y ) =

Original: ¯ ∇ f ( x, y ) −
Revised: ¯ ∇ f ( x, y ) −

Original: E ¯ ξ (cid:2) ¯ ∇ f ( x, y ; ¯ ξ ) (cid:3) in the gradient estimator (6) decays exponentially fast with number K .
Revised: E ¯ ξ (cid:2) ¯ ∇ f ( x, y ; ¯ ξ ) (cid:3) in the gradient estimator (6) decays exponentially fast with number K .

Original: Lemma 3 (Lemma 2.1 in Khanduri et al.
Revised: Lemma 3. (Lemma 2.1 in Khanduri et al.

Original: (2021) and Lemma 11 in Hong et al.
Revised: (2021) and Lemma 11 in Hong et al.

Original: Under the about Assumptions (1, 4), for any K ≥ 1 , the gradient estimator in equation 6 satisfies
Revised: Under the about Assumptions (1, 4), for any K ≥ 1 , the gradient estimator in equation 6 satisfies

Original: [Equation]
Revised: [Equation]

Original: where R ( x, y ) =¯ ∇ f ( x, y )
Revised: where R ( x, y [Equation] =

Original: − E ¯ ξ (cid:2) ¯ ∇ f ( x, y ; ¯ ξ ) (cid:3) .
Revised: [Equation]

Original: From Lemma 3, choose K
Revised: From Lemma 3, choose K

Original: = L g µ log( C gxy C fy T/µ ) in Algorithm 1, we have ∥ R ( x, y )
Revised: = L g µ log( C gxy C fy T/µ ) in Algorithm 1, we have ∥ R ( x, y )

Original: ∥ ≤ 1 T for all t ≥ 1 .
Revised: ∥ ≤ 1 T for all t ≥ 1 .

Original: Thus, this result guarantees convergence of our algorithms only requiring a small mini-batch samples.
Revised: Thus, this result guarantees convergence of our algorithms only requiring a small mini-batch samples.

Original: For notational simplicity, let R t
Revised: For notational simplicity, let R t

Original: = R ( x t , y t ) for all t ≥ 1 .
Revised: = R ( x t , y t ) for all t ≥ 1 .

Original: Lemma 4 (Lemma B.2 in Khanduri et al.
Revised: Lemma 4. (Lemma 3.1 in Khanduri et al.

Original: Under the above Assumptions (1, 3, 4), stochastic gradient estimate ¯ ∇ f ( x, y ; ¯ ξ ) is Lipschitz continuous, such that for x, x 1 , x 2 ∈ X and y, y 1 , y 2 ∈ Y ,
Revised: Under the above Assumptions (1, 3, 4), stochastic gradient estimate ¯ ∇ f ( x, y ; ¯ ξ ) is Lipschitz continuous, such that for x, x 1 , x 2 ∈ X and y, y 1 , y 2 ∈ Y ,

Original: [Equation]
Revised: [Equation]

Original: ¯ ξ
Revised: where L 2 K = 2 L 2 f + 6 C 2 gxy L 2 f

Original: [Equation]
Revised: [Equation]

Original: where L 2 K= 2 L 2 f + 6 C 2 gxy L
Revised: K

Original: [Equation]
Revised: [Equation]

Original: 10: end for 11: Output: Chosen uniformly random from { x t , y t } Tt =1 .
Revised: K [Equation] [Table] [Equation]

Original: 3. VR-B I A DAM A LGORITHM
Revised: 3. VR-B I A DAM A LGORITHM

Original: In this subsection, we propose an accelerated version of BiAdam method (VR-BiAdam) by using the momentum-based variance reduced technique.
Revised: In this subsection, we propose an accelerated version of BiAdam method (VR-BiAdam) by using the momentum-based variance reduced technique.

Original: Algorithm 2 demonstrates the algorithmic framework of our VR-BiAdam algorithm.
Revised: Algorithm 2 demonstrates the algorithmic framework of our VR-BiAdam algorithm.

Revised: 10: end for 11: Output: Chosen uniformly random from { x t , y t } Tt =1 .

Original: At the lines 8-9 of Algorithm 2, we use the momentum-based variance reduced technique to estimate the stochastic partial derivatives v t and w t .
Revised: At the lines 8-9 of Algorithm 2, we use the momentum-based variance reduced technique to estimate the stochastic partial derivatives v t and w t .

Original: For example, the estimator of partial derivative ¯ ∇ f ( x t +1 , y t +1 ) is defined as
Revised: For example, the estimator of partial derivative ¯ ∇ f ( x t +1 , y t +1 ) is defined as

Original: [Equation]
Revised: [Equation]

Original: Compared with the estimator w t +1 in Algorithm 1, w t +1 in Algorithm 2 adds the term ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 )
Revised: Compared with the estimator w t +1 in Algorithm 1, w t +1 in Algorithm 2 adds the term ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 )

Original: − ¯ ∇ f ( x t , y t ; ¯ ξ t +1 ) to control the variances of estimator.
Revised: − ¯ ∇ f ( x t , y t ; ¯ ξ t +1 ) to control the variances of estimator.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: In this section, we study the convergence properties of our algorithms (BiAdam and VR-BiAdam) under some mild conditions.
Revised: In this section, we study the convergence properties of our algorithms (BiAdam and VR-BiAdam) under some mild conditions.

Original: All proofs are provided in the Appendix A.
Revised: All proofs are provided in the Appendix A.

Original: 4. A DDITIONAL M ILD A SSUMPTIONS
Revised: 4. A DDITIONAL M ILD A SSUMPTIONS

Original: Assumption 6 The estimated stochastic partial derivative ¯ ∇ f ( x, y ; ¯ ξ ) satisfies [Equation] x, y ) , E ¯ ξ [Equation]
Revised: Assumption 6.

Original: f [Equation] x, y ; ¯ ξ ) − ¯ ∇ f ( x, y  ) [Equation]
Revised: The estimated stochastic partial derivative ¯ ∇ f ( x, y ; ¯ ξ ) satisfies

Original: R

Original: [Equation]

Original: x, y ) ∥ 2 ≤ σ

Original: [Equation]
Revised: [Equation]

Original: The stochastic partial derivative ∇ y g
Revised: The stochastic partial derivative ∇ y g

Original: ( x, y ; ζ ) satisfies
Revised: ( x, y ; ζ ) satisfies

Original: [Equation]
Revised: [Equation]

Original:  Assumption 7 In our algorithms, the adaptive matrices A t and B t for all t ≥ 1 satisfy A t ⪰ ρI d ( ρ > 0) and B t = bI p ( b u ≥ b ≥ b l > 0) , respectively, where ρ , b u and b l are appropriate positive numbers.
Revised: [Equation] Assumption 7. In our algorithms, the adaptive matrices A t and B t for all t ≥ 1 satisfy A t ⪰ ρI d ( ρ > 0) and B t = bI p ( b u ≥ b ≥ b l > 0) , respectively, where ρ , b u and b l are appropriate positive numbers.

Original: Assumption 6 is commonly used in the stochastic bilevel optimization methods Ji et al.
Revised: Assumption 6 is commonly used in the stochastic bilevel optimization methods Ji et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: ; Khanduri et al.
Revised: ; Khanduri et al.

Original: In the paper, we consider the general adaptive learning rates (including the coordinate-wise and global learning rates) for variable x and the global learning rate for variable y .
Revised: In the paper, we consider the general adaptive learning rates (including the coordinate-wise and global learning rates) for variable x and the global learning rate for variable y .

Original: Assumption 7 ensures that the adaptive matrices A t for all t ≥ 1 are positive definite as in Huang et al.
Revised: Assumption 7 ensures that the adaptive matrices A t for all t ≥ 1 are positive definite as in Huang et al.

Original: Assumption 7 also guarantees the global adaptive matrices B t for all t ≥ 1 are positive definite and bounded.
Revised: Assumption 7 also guarantees the global adaptive matrices B t for all t ≥ 1 are positive definite and bounded.

Original: In fact, Assumption 7 is mild.
Revised: In fact, Assumption 7 is mild.

Original: For example, in the problem min x ∈ R p E [ f ( x ; ξ )] , Ward et al.
Revised: For example, in the problem min x ∈ R p E [ f ( x ; ξ )] , Ward et al.

Original: (2019) apply a global adaptive learning rate to the update form x t
Revised: (2019) apply a global adaptive learning rate to the update form x t

Original: = x t − 1 − η ∇ f ( x t − 1 ; ξ t − 1 ) b t , b 2 t = b 2 t − 1 + ∥∇ f ( x t − 1 ; ξ t − 1 ) ∥ 2 , b 0 > 0 , η > 0 for all t ≥ 1 , which is equivalent to the form x t
Revised: = x t − 1 − η ∇ f ( x t − 1 ; ξ t − 1 ) b t , b 2 t = b 2 t − 1 + ∥∇ f ( x t − 1 ; ξ t − 1 ) ∥ 2 , b 0 > 0 , η > 0 for all t ≥ 1 , which is equivalent to the form x t

Original: = x t − 1 − ηB − 1 t ∇ f ( x t − 1 ; ξ t − 1 ) with B t =
Revised: = x t − 1 − ηB − 1 t ∇ f ( x t − 1 ; ξ t − 1 ) with B t =

Original: b t I p and b t ≥ · · · ≥ b 0 > 0 .
Revised: b t I p and b t ≥ · · · ≥ b 0 > 0 .

Original: Li & Orabona (2019); Cutkosky & Orabona (2019) use a global adaptive learning rate to the update form x t +1 = x
Revised: Li & Orabona (2019); Cutkosky & Orabona (2019) use a global adaptive learning rate to the update form x t +1 = x

Original: t − ηg t /b t , where g t is stochastic gradient and b t = (cid:0) ω
Revised: t − ηg t /b t , where g t is stochastic gradient and b t = (cid:0) ω

Original: + (cid:80) ti =1 ∥∇ f
Revised: + (cid:80) ti =1 ∥∇ f

Original: ( x i ; ξ i ) ∥ 2 (cid:1)
Revised: ( x i ; ξ i ) ∥ 2 (cid:1)

Original: α /k , k > 0 , ω > 0 and α ∈ (0 , 1) , which is equivalent to x t +1 = x
Revised: α /k , k > 0 , ω > 0 and α ∈ (0 , 1) , which is equivalent to x t +1 = x

Original: t − ηB − 1 t g t with B t =
Revised: t − ηB − 1 t g t with B t =

Original: b t I p and b t ≥ · · · ≥ b 0 = ω
Revised: b t I p and b t ≥ · · · ≥ b 0 = ω

Original: α k > 0 .
Revised: α k > 0 .

Original: At the same time, the problem min x ∈ R p
Revised: At the same time, the problem min x ∈ R p

Original: f
Revised: f

Original: ( x ) =
Revised: ( x ) =

Original: E [ f ( x ; ξ )] approaches the stationary points, i.e., ∇ f ( x ) = 0 or even ∇ f ( x ; ξ ) = 0 for all ξ .
Revised: E [ f ( x ; ξ )] approaches the stationary points, i.e., ∇ f ( x ) = 0 or even ∇ f ( x ; ξ ) = 0 for all ξ .

Original: Thus, these global adaptive learning rates are generally bounded, i.e., b u ≥ b t ≥ b l > 0 for all t ≥ 1 .
Revised: Thus, these global adaptive learning rates are generally bounded, i.e., b u ≥ b t ≥ b l > 0 for all t ≥ 1 .

Original: 4. U SEFUL C ONVERGENCE M ETRIC
Revised: 4. U SEFUL C ONVERGENCE M ETRIC

Original: In the subsection, we define a useful convergence metric for our algorithms and some useful lemmas.
Revised: In the subsection, we define a useful convergence metric for our algorithms and some useful lemmas.

Original: Lemma [-
-]
Revised: Lemma [-
-]5.

Original: Given gradient estimator w t is generated from Algorithms 1 or 2, for all t ≥ 1 , we have
Revised: Given gradient estimator w t is generated from Algorithms 1 or 2, for all t ≥ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: where 
Revised: where L 20 =

Original: [Equation]
Revised: [Equation]

Original: For our Algorithms 1 and 2, based on Lemma 5, we provide a convergence metric E [ G t ] , defined as
Revised: For our Algorithms 1 and 2, based on Lemma 5, we provide a convergence metric E [ G t ] , defined as

Original: [Equation]
Revised: [Equation]

Original: where the first two terms of G t measure the convergence of the iteration solutions { x t } Tt =1 , and the last term measures the convergence of the iteration solutions { y t } Tt =1 .
Revised: where the first two terms of G t measure the convergence of the iteration solutions { x t } Tt =1 , and the last term measures the convergence of the iteration solutions { y t } Tt =1 .

Original: Let ϕ t ( x ) =
Revised: Let ϕ t ( x ) =

Original: 12 x T A t x , we define a prox-function (a.k.a., Bregman distance) Censor & Lent (1981);
Revised: 12 x T A t x , we define a prox-function (a.k.a., Bregman distance) Censor & Lent (1981);

Original: Censor & Zenios (1992); Ghadimi et al.
Revised: Censor & Zenios (1992); Ghadimi et al.

Original: (2016) associated with ϕ t ( x ) as follows:
Revised: (2016) associated with ϕ t ( x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: The line 5 of Algorithm 1 or 2 is equivalent to the following generalized projection problem:
Revised: The line 5 of Algorithm 1 or 2 is equivalent to the following generalized projection problem:

Original: [Equation]
Revised: [Equation]

Original: As in Ghadimi et al.
Revised: As in Ghadimi et al.

Original: (2016), we define a generalized projected gradient G X ( x t , w t , γ ) =
Revised: (2016), we define a generalized projected gradient G X ( x t , w t , γ ) =

Original: 1 γ ( x t − ˜ x t +1 ) .
Revised: 1 γ ( x t − ˜ x t +1 ) .

Original: At the same time, we define a gradient mapping G X ( x t , ∇ F ( x t ) , γ ) =
Revised: At the same time, we define a gradient mapping G X ( x t , ∇ F ( x t ) , γ ) =

Original: 1 γ ( x t − x + t +1 ) withx + t
Revised: 1 γ ( x t − x + t +1 ) with

Original: [Equation]
Revised: [Equation]

Original: 1 = arg min x ∈X (cid:110) ⟨∇

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ) (cid:111) .

Original: According to the Proposition 1 of Ghadimi et al.
Revised: According to the Proposition 1 of Ghadimi et al.

Original: , we have ||G X ( x t , w t , γ ) −
Revised: , we have ||G X ( x t , w t , γ ) −

Original: G X ( x t , ∇ F ( x t ) , γ ) || ≤ 1 ρ ||∇ F ( x t ) − w t || .
Revised: G X ( x t , ∇ F ( x t ) , γ ) || ≤ 1 ρ ||∇ F ( x t ) − w t || .

Original: Since ||G X ( x t , ∇ F ( x t ) , γ ) || ≤ ||G X
Revised: Since ||G X ( x t , ∇ F ( x t ) , γ ) || ≤ ||G X

Original: ( x t , w t , γ ) ||
Revised: ( x t , w t , γ ) ||

Original: + ||G X ( x t , w t , γ ) − G X
Revised: + ||G X ( x t , w t , γ ) − G X

Original: ( x t , ∇ F ( x t ) , γ ) || , we have||G X ( x t , ∇ F ( x t ) , γ )
Revised: ( x t , ∇ F ( x t ) , γ ) || , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds by the above Lemma 5.
Revised: where the last inequality holds by the above Lemma 5.

Original: Thus, our new convergence measure E [ G t ] is tighter than the standard gradient mapping E ||G X
Revised: Thus, our new convergence measure E [ G t ] is tighter than the standard gradient mapping E ||G X

Original: ( x t , ∇ F ( x t ) , γ ) || used in Hong et al.
Revised: ( x t , ∇ F ( x t ) , γ ) || used in Hong et al.

Original: When
Revised: When

Original: G t → 0 , we have ∥G X ( x t , ∇ F ( x t ) , γ ) ∥
Revised: G t → 0 , we have ∥G X ( x t , ∇ F ( x t ) , γ ) ∥

Original: → 0 , where x t is a stationary point or local minimum of the bilevel problem equation 1 Ghadimi et al.
Revised: → 0 , where x t is a stationary point or local minimum of the bilevel problem equation 1 Ghadimi et al.

Original: ; Hong et al.
Revised: ; Hong et al.

Original: 4. C ONVERGENCE A NALYSIS OF B I A DAM A LGORITHM
Revised: 4. C ONVERGENCE A NALYSIS OF B I A DAM A LGORITHM

Original: In this subsection, we study the convergence properties of our BiAdam algorithm.
Revised: In this subsection, we study the convergence properties of our BiAdam algorithm.

Original: The detailed proofs are provided in the Appendix A.5.
Revised: The detailed proofs are provided in the Appendix A.5.

Original: Theorem 1 Under the above Assumptions (1, 2, 4, 6, 7), in the Algorithm 1, given X ⊂ R d , η t = k
Revised: Theorem 1. Under the above Assumptions (1, 2, 4, 6, 7), in the Algorithm 1, given X ⊂ R d , η t = k

Original: ( m + t ) 1 / 2 for all t ≥ 0 , α t +1 =
Revised: ( m + t ) 1 / 2 for all t ≥ 0 , α t +1 =

Original: c 1 η t , β t +

Revised: c 1 η t , β t +

Original: =
Revised: =

Original: c
Revised: c

Original: 2 η t , m ≥ max (cid:0) k 2 , ( c 1 k ) 2 , ( c 2 k ) 2 (cid:1) , k > 0 , 125 L 20 6 µ 
Revised: 2 η t , m ≥ max (cid:0) k 2 , ( c 1 k ) 2 , ( c 2 k ) 2 (cid:1) , k > 0 , 125 L 20 6 µ [Equation] min

Original: [Equation]
Revised: [Equation]

Original: (cid:1) and K =
Revised: (cid:1) and K =

Original: L g µ log( C gxy C fy T/µ ) , we have
Revised: L g µ log( C gxy C fy T/µ ) , we have

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: (12) where G = F ( x 1 ) − F ∗ ρkγ 
Revised: (12) where G = F ( x 1 ) − F ∗ ρkγ +

Revised: [Equation]

Revised: ln( m + T )+ 4( m + T )

Original: [Equation]
Revised: [Equation]

Revised: 2 kT 2 + 8 k ρ 2 T and ∆ 0 = ∥

Revised: y 1 − y ∗ ( x 1 ) ∥ 2 ,

Revised: L

Revised: [Equation]

Revised: = 12 L 2 g µ 2 125 L 20 + 2 L 20 3 .

Original: Remark 1 Without loss of generality, let k
Revised: Remark 1. Without loss of generality, let k

Original: = O (1) and m =
Revised: = O (1) and m =

Original: O (1) , we have G
Revised: O (1) , we have G

Original: = O (ln( m +
Revised: = O (ln( m +

Original: T ))
Revised: T ))

Original: =
Revised: =

Original: ˜ O (1) .
Revised: ˜ O (1) .

Original: Thus our BiAdam algorithm has a convergence rate of ˜ O
Revised: Thus our BiAdam algorithm has a convergence rate of ˜ O

Original: ( 1 T 1 / 4 ) .
Revised: ( 1 T 1 / 4 ) .

Original: Let E [ G ζ ] = 1 T (cid:80) T t =1 E [ G t ] =
Revised: Let E [ G ζ ] = 1 T (cid:80) Tt =1 E [ G t ] =

Original: ˜ O ( 1 T 1 / 4 ) ≤ ϵ , we have T =
Revised: ˜ O ( 1 T 1 / 4 ) ≤ ϵ , we have T =

Original: ˜ O
Revised: ˜ O

Original: ( ϵ − 4 ) .
Revised: ( ϵ − 4 ) .

Original: Since our BiAdam algorithm only requires
Revised: Since our BiAdam algorithm only requires

Original: K + 2 =
Revised: K + 2 =

Original: L g µ log( C gxy C fy T/µ ) + 2 = ˜ O
Revised: L g µ log( C gxy C fy T/µ ) + 2 = ˜ O

Original: (1) samples to estimate stochastic partial derivatives in each iteration, and needs T iterations.
Revised: (1) samples to estimate stochastic partial derivatives in each iteration, and needs T iterations.

Original: Thus our BiAdam algorithm requires sample complexity of ( K + 2) T =
Revised: Thus our BiAdam algorithm requires sample complexity of ( K + 2) T =

Original: ˜ O
Revised: ˜ O

Original: ( ϵ − 4 ) for finding an ϵ -stationary point of the problem (1).
Revised: ( ϵ − 4 ) for finding an ϵ -stationary point of the problem (1).

Original: Note that the convergence properties of our BiAdam algorithm for unconstrained bilevel optimization are provided in the Appendix A.3.
Revised: Note that the convergence properties of our BiAdam algorithm for unconstrained bilevel optimization are provided in the Appendix A.3.

Original: 4. C ONVERGENCE A NALYSIS OF VR-B I A DAM A LGORITHM
Revised: 4. C ONVERGENCE A NALYSIS OF VR-B I A DAM A LGORITHM

Original: In this subsection, we study convergence properties of our VR-BiAdam algorithm.
Revised: In this subsection, we study convergence properties of our VR-BiAdam algorithm.

Original: The detailed proofs are provided in the Appendix A.6.
Revised: The detailed proofs are provided in the Appendix A.6.

Original: Theorem 2 Under the above Assumptions (1, 3, 4, 6, 7), in the Algorithm 2, given X ⊂ R d , η t = k
Revised: Theorem 2. Under the above Assumptions (1, 3, 4, 6, 7), in the Algorithm 2, given X ⊂ R d , η t = k

Original: ( m + t ) 1 / 3 for all t ≥ 0 , α t +1 =
Revised: ( m + t ) 1 / 3 for all t ≥ 0 , α t +1 =

Original: c 1 η 2 t , β t +

Revised: c 1 η 2 t , β t +

Original: = c 2 η 2 t , m ≥ max (cid:0) 2 , k 3 , ( c 1 k ) 3 , ( c 2 k ) 3 (cid:1) , k > 0 , c 1 ≥ 23 k 3 + 125 L 20 6 µ 2 , c 2 ≥ 23 k 3 + 92 , 0 < λ ≤ min (cid:0) 15 b l L 20 16 L 22 µ , b l 6 L g (cid:1) , 0 < γ ≤ min (cid:0)[-√ 6 λµρ 2 √ 24 L 22 λ 2 µ 2 +125 b 2 u L 20 κ 2 , m 1 / 
 ρ 4 Lk (cid:1) and K = L g µ log( C gxy C fy T/µ ) , we have-]
Revised: = c 2 η 2 t , m ≥ max (cid:0) 2 , k 3 , ( c 1 k ) 3 , ( c 2 k ) 3 (cid:1) , k > 0 , c 1 ≥ 23 k 3 + 125 L 20 6 µ 2 , c 2 ≥ 23 k 3 + 92 , 0 < λ ≤ min (cid:0) 15 b l L 20 16 L 22 µ , b l 6 L g (cid:1) , 0 < γ ≤ min (cid:0)[-√ 6 λµρ 2 √ 24 L 22 λ 2 µ 2 +125 b 2 u L 20 κ 2 , m 1 / 
 ρ 4 Lk (cid:1) and K = L g µ log( C gxy C fy T/µ ) , we have-]

Original: [Equation]
Revised: [Table]

Original: T (cid:88) [Equation] (13) where M = F ( x 1 ) − F ∗ ρkγ + 5 b 1 L 20 ∆ 

Revised: κ 2 , m 1 /

Original: ρ 2 kλµ + 2 m 1 / 3 σ [Equation] + 2 k
Revised: ρ 4 Lk (cid:1) and K =

Original: [Equation] c 21 + c 22 ) σ 2 ln
Revised: L g µ log( C gxy C fy T/µ ) , we have

Original: [Equation]
Revised: [Equation]

Original: m + T[Equation] + 6 k
Revised: T

Original: [Equation]
Revised: [Equation]

Original: m + T ) [Equation] 3 ρ
Revised: where M = F ( x 1 ) − F ∗ ρkγ

Original: [Equation]
Revised: [Equation]

Original: , ∆ 0 = ∥ y 1 − y ∗ ( x 1 ) ∥ 2 and L 22 =
Revised: y ∗ ( x 1 ) ∥ 2 and L 22 =

Original: L 2 g + L 2 K .
Revised: L 2 g + L 2 K .

Original: Remark 2 Without loss of generality, let k
Revised: Remark 2. Without loss of generality, let k

Original: = O (1) and m =
Revised: = O (1) and m =

Original: O (1) , we have M =
Revised: O (1) , we have M =

Original: O
Revised: O

Original: (ln( m +
Revised: (ln( m +

Original: T ))
Revised: T ))

Original: =
Revised: =

Original: ˜ O (1) .
Revised: ˜ O (1) .

Original: Thus our VR-BiAdam algorithm has a convergence rate of ˜ O
Revised: Thus our VR-BiAdam algorithm has a convergence rate of ˜ O

Original: ( 1 T 1 / 3 ) .
Revised: ( 1 T 1 / 3 ) .

Original: Let E [ G ζ ] = 1 T (cid:80) Tt =1 E
Revised: Let E [ G ζ ] = 1 T (cid:80) Tt =1 E

Original: [ G t ] =
Revised: [ G t ] =

Original: ˜ O ( 1 T 1 / 3 ) ≤ ϵ , we have T =
Revised: ˜ O ( 1 T 1 / 3 ) ≤ ϵ , we have T =

Original: ˜ O
Revised: ˜ O

Original: ( ϵ − 3 ) .
Revised: ( ϵ − 3 ) .

Original: Since our VR-BiAdam algorithm requires
Revised: Since our VR-BiAdam algorithm requires

Original: K + 2 =
Revised: K + 2 =

Original: L g µ log( C gxy C fy T/µ ) + 2 = ˜ O
Revised: L g µ log( C gxy C fy T/µ ) + 2 = ˜ O

Original: (1) samples to estimate stochastic partial derivatives in each iteration, and needs T iterations.
Revised: (1) samples to estimate stochastic partial derivatives in each iteration, and needs T iterations.

Original: Thus our VR-BiAdam algorithm requires sample complexity of ( K + 2) T =
Revised: Thus our VR-BiAdam algorithm requires sample complexity of ( K + 2) T =

Original: ˜ O
Revised: ˜ O

Original: ( ϵ − 3 ) for finding an ϵ -stationary point of the problem (1).
Revised: ( ϵ − 3 ) for finding an ϵ -stationary point of the problem (1).

Original: Note that the convergence properties of our VR-BiAdam algorithm for unconstrained bilevel optimization are provided in the Appendix A.3.
Revised: Note that the convergence properties of our VR-BiAdam algorithm for unconstrained bilevel optimization are provided in the Appendix A.3.

Original:  N UMERICAL E XPERIMENTS
Revised:  N UMERICAL E XPERIMENTS

Original: In this section, we perform two hyper-parameter optimization tasks to demonstrate the efficiency of our algorithms: 1) data hyper-cleaning task over the MNIST dataset; 2) hyper-representation learning task over the Omniglot dataset.
Revised: In this section, we perform two hyper-parameter optimization tasks to demonstrate the efficiency of our algorithms: 1) data hyper-cleaning task over the MNIST dataset; 2) hyper-representation learning task over the Omniglot dataset.

Original: In all experiments, we use a server with AMD EPYC64-Core CPU and 1 NVIDIA RTX A5000.
Revised: In all experiments, we use a server with AMD EPYC64-Core CPU and 1 NVIDIA RTX A5000.

Original: 5. D ATA H YPER -C LEANING
Revised: 5. D ATA H YPER -C LEANING

Original: In the hyper-cleaning task, we clean a noisy dataset through a bilevel formulation.
Revised: In the hyper-cleaning task, we clean a noisy dataset through a bilevel formulation.

Original: The precise formulation of the problem is included in the Appendix A.1.1.
Revised: The precise formulation of the problem is included in the Appendix A.1.1.

Original: In particular, we use a training set and a validation set, where each contains 5000 images in our experiments.
Revised: In particular, we use a training set and a validation set, where each contains 5000 images in our experiments.

Original: A portion of the training data are corrupted by randomly changing their labels, and we denote the portion of corrupted images as ρ .
Revised: A portion of the training data are corrupted by randomly changing their labels, and we denote the portion of corrupted images as ρ .

Original: We compare our algorithms ( i.e. , BiAdam and VR-BiAdam) with various baselines.
Revised: We compare our algorithms ( i.e. , BiAdam and VR-BiAdam) with various baselines.

Original: See the
Revised: See the

Original: Appendix A.1 for a brief introduction of the baselines.
Revised: Appendix A.1 for a brief introduction of the baselines.

Original: For all methods, we perform grid search over hyper-parameters and choose the best setting.
Revised: For all methods, we perform grid search over hyper-parameters and choose the best setting.

Original: The detailed experimental setup is described in the Appendix A.1.1.
Revised: The detailed experimental setup is described in the Appendix A.1.1.

Original: The experimental results are summarized in Figure 1.
Revised: The experimental results are summarized in Figure 1.

Original: As shown by the figure, our BiAdam algorithm outperforms its non-adaptive counterparts such as stocBiO, MRBO and
Revised: As shown by the figure, our BiAdam algorithm outperforms its non-adaptive counterparts such as stocBiO, MRBO and

Original: SUSTAIN, furthermore, our VR-BiAdam gets the best performance, where it outperforms VRBO, which requires using large batch-sizes every a few iterations.
Revised: SUSTAIN, furthermore, our VR-BiAdam gets the best performance, where it outperforms VRBO, which requires using large batch-sizes every a few iterations.

Original: 5. H YPER - REPRESENTATION L EARNING
Revised: 5. H YPER - REPRESENTATION L EARNING

Original: In the hyper-representation learning task, we learn a hyper-representation of the data such that a linear classifier can be learned quickly with a small number of data samples.
Revised: In the hyper-representation learning task, we learn a hyper-representation of the data such that a linear classifier can be learned quickly with a small number of data samples.

Original: The precise formulation of the problem is included in Appendix A.1.2.
Revised: The precise formulation of the problem is included in Appendix A.1.2.

Original: We compare our algorithms ( i.e. , BiAdam and VRBiAdam) with various baselines.
Revised: We compare our algorithms ( i.e. , BiAdam and VRBiAdam) with various baselines.

Original: See Appendix A.1 for a brief introduction of the baselines.
Revised: See Appendix A.1 for a brief introduction of the baselines.

Original: For all methods, we perform grid search over hyper-parameters and choose the best setting.
Revised: For all methods, we perform grid search over hyper-parameters and choose the best setting.

Original: The detailed experimental setup is described in the Appendix A.1.2.
Revised: The detailed experimental setup is described in the Appendix A.1.2.

Original: The experimental results are summarized in
Revised: The experimental results are summarized in

Original: Figure 2.
Revised: Figure 2.

Original: As shown by the figure, both our BiAdam and VR-BiAdam algorithms outperform other baselines.
Revised: As shown by the figure, both our BiAdam and VR-BiAdam algorithms outperform other baselines.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we proposed a class of novel adaptive bilevel optimization methods for nonconvexstrongly-convex bilevel optimization problems.
Revised: In this paper, we proposed a class of novel adaptive bilevel optimization methods for nonconvexstrongly-convex bilevel optimization problems.

Original: Our methods use unified adaptive matrices including many types of adaptive learning rates, and can flexibly use the momentum and variance reduced techniques.
Revised: Our methods use unified adaptive matrices including many types of adaptive learning rates, and can flexibly use the momentum and variance reduced techniques.

Original: Moreover, we provided a useful convergence analysis framework for both the constrained and unconstrained bilevel optimization.
Revised: Moreover, we provided a useful convergence analysis framework for both the constrained and unconstrained bilevel optimization.

Original: Our VR-BiAdam algorithm reaches the best known sample complexity of ˜ O
Revised: Our VR-BiAdam algorithm reaches the best known sample complexity of ˜ O

Original: ( ϵ − 3 ) for finding an ϵ -stationary point.
Revised: ( ϵ − 3 ) for finding an ϵ -stationary point

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In this section, we provide the additional experiment results, related works and additional theoretical results.
Revised: In this section, we provide the additional experiment results, related works and additional theoretical results.

Original: We also provide the detailed convergence analysis.
Revised: We also provide the detailed convergence analysis.

Original: A. A DDITIONAL E XPERIMENTAL D ETAILS AND R ESULTS
Revised: A. A DDITIONAL E XPERIMENTAL D ETAILS AND R ESULTS

Original: In this subsection, we introduce more details of our experiments.
Revised: In this subsection, we introduce more details of our experiments.

Original: We compare our BiAdam and
Revised: We compare our BiAdam and

Original: VR-BiAdam algorithms with the following bilevel optimization algorithms: reverse Franceschi et al.
Revised: VR-BiAdam algorithms with the following bilevel optimization algorithms: reverse Franceschi et al.

Original: (2018), AID-CG Grazzi et al.
Revised: (2018), AID-CG Grazzi et al.

Original: (2020), AID-FP Grazzi et al.
Revised: (2020), AID-FP Grazzi et al.

Original: (2020), stocBio Ji et al.
Revised: (2020), stocBio Ji et al.

Original: (2021)), MRBO Ji & Liang (2021), VRBO Ji & Liang (2021), FSLA Li et al.
Revised: (2021)), MRBO Ji & Liang (2021), VRBO Ji & Liang (2021), FSLA Li et al.

Original: MSTSA/SUSTAIN Khanduri et al.
Revised: MSTSA/SUSTAIN Khanduri et al.

Original: (2021), SMB Guo et al.
Revised: (2021), SMB Guo et al.

Original: (2021b), SVRB Guo & Yang (2021).
Revised: (2021b), SVRB Guo & Yang (2021).

Original: A.1. D ATA H YPER - CLEANING
Revised: A.1. D ATA H YPER - CLEANING

Original: In this task, we perform data hyper-cleaning over the MNIST dataset LeCun et al.
Revised: In this task, we perform data hyper-cleaning over the MNIST dataset LeCun et al.

Original: The formulation of this problem is as follows:
Revised: The formulation of this problem is as follows:

Original: [Equation]
Revised: [Equation]

Original: where f ( · ) denotes the cross entropy loss, D T and D V are training and validation dataset, respectively.
Revised: where f ( · ) denotes the cross entropy loss, D T and D V are training and validation dataset, respectively.

Original: Here τ = { τ i } i ∈D T are hyper-parameters and C ≥ 0 is a tuning parameter, σ ( · ) denotes the sigmoid function.
Revised: Here τ = { τ i } i ∈D T are hyper-parameters and C ≥ 0 is a tuning parameter, σ ( · ) denotes the sigmoid function.

Original: In the experiment, we set C = 0 .
Revised: In the experiment, we set C = 0 .

Original: For training/validation batch-size, we use batch-size of 32, while for VRBO, we choose larger batchsize 5000 and sampling interval is set as 3.
Revised: For training/validation batch-size, we use batch-size of 32, while for VRBO, we choose larger batchsize 5000 and sampling interval is set as 3.

Original: For AID-FP, AID-CG and reverse, we use the warm-start trick as our algorithms, i.e. the inner variable starts from the state of last iteration.
Revised: For AID-FP, AID-CG and reverse, we use the warm-start trick as our algorithms, i.e. the inner variable starts from the state of last iteration.

Original: We fine tune the number of inner-loop iterations and set it to be 50 for these algorithms.
Revised: We fine tune the number of inner-loop iterations and set it to be 50 for these algorithms.

Original: For MRBO, VRBO,
Revised: For MRBO, VRBO,

Original: SUSTAIN and our BiAdam/VR-BiAdam, we set K = 3 to evaluate the hyper-gradient.
Revised: SUSTAIN and our BiAdam/VR-BiAdam, we set K = 3 to evaluate the hyper-gradient.

Original: For FSLA,
Revised: For FSLA,

Original: K = 1 as the hyper-gradient is evaluated recursively.
Revised: K = 1 as the hyper-gradient is evaluated recursively.

Original: As for learning rates, we set 1000 as the outer learning rate for all algorithms except our algorithms which use 0.5 as we change the learning rate adaptively.
Revised: As for learning rates, we set 1000 as the outer learning rate for all algorithms except our algorithms which use 0.5 as we change the learning rate adaptively.

Original: As for the inner learning rates, we set the stepsize as 0.05 for reverse, AID-CG, stocBiO/AID-FP, MRBO/SUSTAIN, FSLA; we set the stepsize as 0.2 for VRBO; we set the stepsize as 1 for SUSTAIN; we set the stepsize as 0.00025 for BiAdam and 0.0005 for VR-BiAdam.
Revised: As for the inner learning rates, we set the stepsize as 0.05 for reverse, AID-CG, stocBiO/AID-FP, MRBO/SUSTAIN, FSLA; we set the stepsize as 0.2 for VRBO; we set the stepsize as 1 for SUSTAIN; we set the stepsize as 0.00025 for BiAdam and 0.0005 for VR-BiAdam.

Original: A.1. H YPER - REPRESENTATION L EARNING
Revised: A.1. H YPER - REPRESENTATION L EARNING

Original: In this task, we perform the hyper-representation learning task over the Omniglot dataset Lake et al.
Revised: In this task, we perform the hyper-representation learning task over the Omniglot dataset Lake et al.

Original: The formulation of this problem is as follows:
Revised: The formulation of this problem is as follows:

Original: [Equation]
Revised: [Equation]

Original: where f ( · ) denotes the cross entropy loss, D T ,ξ and D V ,ξ are training and validation dataset for a randomly sampled meta task.
Revised: where f ( · ) denotes the cross entropy loss, D T ,ξ and D V ,ξ are training and validation dataset for a randomly sampled meta task.

Original: Here τ = { τ i } i ∈D T are hyper-representations and C ≥ 0 is a tuning parameter to gaurantee the inner problem to be strongly convex.
Revised: Here τ = { τ i } i ∈D T are hyper-representations and C ≥ 0 is a tuning parameter to gaurantee the inner problem to be strongly convex.

Original: In experiment, we set C = 0 .
Revised: In experiment, we set C = 0 .

Original: In every hyper-iteration, we choose 4 meta tasks, while for VRBO, we choose larger batch-sizeand sampling interval is set as 3.
Revised: In every hyper-iteration, we choose 4 meta tasks, while for VRBO, we choose larger batch-sizeand sampling interval is set as 3.

Original: For stocBiO/AID-FP, AID-CG and reverse, we use the warmstart trick as our algorithms, i.e. the inner variable starts from the state of last iteration.
Revised: For stocBiO/AID-FP, AID-CG and reverse, we use the warmstart trick as our algorithms, i.e. the inner variable starts from the state of last iteration.

Original: We fine tune the number of inner-loop iterations and set it to be 16 for these algorithms.
Revised: We fine tune the number of inner-loop iterations and set it to be 16 for these algorithms.

Original: For MRBO, VRBO,
Revised: For MRBO, VRBO,

Original: SUSTAIN and our algorithms, we set K = 5 to evaluate the hyper-gradient.
Revised: SUSTAIN and our algorithms, we set K = 5 to evaluate the hyper-gradient.

Original: For FSLA, K = 1 as the hyper-gradient is evaluated recursively.
Revised: For FSLA, K = 1 as the hyper-gradient is evaluated recursively.

Original: As for learning rates, we set 1000 as the outer learning rate for all algorithms except our algorithms which use 0.001 as we change the learning rate adaptively.
Revised: As for learning rates, we set 1000 as the outer learning rate for all algorithms except our algorithms which use 0.001 as we change the learning rate adaptively.

Original: As for the inner learning rates, we set the stepsize as 0.4 for all algorithms.
Revised: As for the inner learning rates, we set the stepsize as 0.4 for all algorithms.

Original: A. R ELATED W ORKS
Revised: A. R ELATED W ORKS

Original: In this subsection, we overview the existing bilevel optimization methods and adaptive methods for single-level optimization, respectively.
Revised: In this subsection, we overview the existing bilevel optimization methods and adaptive methods for single-level optimization, respectively.

Original: A.2. B ILEVEL O PTIMIZATION M ETHODS
Revised: A.2. B ILEVEL O PTIMIZATION M ETHODS

Original: Bilevel optimization has shown successes in many machine learning problems with hierarchical structures such as policy optimization Hong et al.
Revised: Bilevel optimization has shown successes in many machine learning problems with hierarchical structures such as policy optimization Hong et al.

Original: , model-agnostic meta-learning Liu et al.
Revised: , model-agnostic meta-learning Liu et al.

Original: (2021a)
Revised: (2021a)

Original: ; Ji et al.
Revised: ; Ji et al.

Original: ; Lu et al.
Revised: ; Lu et al.

Original: and adversarial training Zhang et al.
Revised: and adversarial training Zhang et al.

Original: Thus, its researches have become active in the machine learning community, and some bilevel optimization methods recently have been proposed.
Revised: Thus, its researches have become active in the machine learning community, and some bilevel optimization methods recently have been proposed.

Original: For example, one class of successful methods Colson et al.
Revised: For example, one class of successful methods Colson et al.

Original: (2007); Kunapuli et al.
Revised: (2007); Kunapuli et al.

Original: (2008) are to reformulate the bilevel problem as a single-level problem by replacing the inner problem by its optimality conditions.
Revised: (2008) are to reformulate the bilevel problem as a single-level problem by replacing the inner problem by its optimality conditions.

Original: Another class of successful methods Ghadimi & Wang (2018); Hong et al.
Revised: Another class of successful methods Ghadimi & Wang (2018); Hong et al.

Original: ; Ji et al.
Revised: ; Ji et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: ; Khanduri et al.
Revised: ; Khanduri et al.

Original: ; Guo & Yang (2021);
Revised: ; Guo & Yang (2021);

Original: Liu et al.
Revised: Liu et al.

Original: (2021b; 2022);
Revised: (2021b; 2022);

Original: Li et al.
Revised: Li et al.

Original: (2021) for bilevel optimization are to iteratively approximate the (stochastic) gradient of the outer problem either in forward or backward.
Revised: (2021) for bilevel optimization are to iteratively approximate the (stochastic) gradient of the outer problem either in forward or backward.

Original: Specifically, Liu et al.
Revised: Specifically, Liu et al.

Original: (2022) proposed a general gradient-based descent aggregation framework for bilevel optimization.
Revised: (2022) proposed a general gradient-based descent aggregation framework for bilevel optimization.

Original: Moreover, the non-asymptotic analysis of these gradient-based bilevel optimization methods has been recently studied.
Revised: Moreover, the non-asymptotic analysis of these gradient-based bilevel optimization methods has been recently studied.

Original: For example, Ghadimi & Wang (2018) first studied the sample complexity of O ( ϵ − 6 ) of the proposed double-loop algorithm for the bilevel problem equation(Please see Table 1).
Revised: For example, Ghadimi & Wang (2018) first studied the sample complexity of O ( ϵ − 6 ) of the proposed double-loop algorithm for the bilevel problem (1) (Please see Table 1).

Original: Subsequently, Ji et al.
Revised: Subsequently, Ji et al.

Original: (2021) proposed an accelerated double-loop algorithm that reaches the sample complexity of O ( ϵ − 4 ) relying on large batches.
Revised: (2021) proposed an accelerated double-loop algorithm that reaches the sample complexity of O ( ϵ − 4 ) relying on large batches.

Original: At the same time, Hong et al.
Revised: At the same time, Hong et al.

Original: (2020) studied a single-loop algorithm that reaches the sample complexity of O ( ϵ − 5 ) without relying on large batches.
Revised: (2020) studied a single-loop algorithm that reaches the sample complexity of O ( ϵ − 5 ) without relying on large batches.

Original: Moreover, Khanduri et al.
Revised: Moreover, Khanduri et al.

Original: ; Guo & Yang (2021); Yang et al.
Revised: ; Guo & Yang (2021); Yang et al.

Original: (2021) proposed a class of accelerated single-loop methods for the bilevel problem equation 1 by using momentum-based variance reduced technique, which achieve the best known sample complexity of O ( ϵ − 3 ) .
Revised: (2021) proposed a class of accelerated single-loop methods for the bilevel problem (1) by using momentum-based variance reduced technique, which achieve the best known sample complexity of O ( ϵ − 3 ) .

Original: More recently, Lu et al.
Revised: More recently, Lu et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: ; Tarzanagh et al.
Revised: ; Tarzanagh et al.

Original: (2022) studied the distributed bilevel optimization.
Revised: (2022) studied the distributed bilevel optimization.

Original: A.2. A DAPTIVE G RADIENT M ETHODS
Revised: A.2. A DAPTIVE G RADIENT M ETHODS

Original: Adaptive gradient methods recently have been shown great successes in current machine learning problems such as training Deep Neural Networks (DNNs).
Revised: Adaptive gradient methods recently have been shown great successes in current machine learning problems such as training Deep Neural Networks (DNNs).

Original: Recently, thus many adaptive gradient methods Duchi et al.
Revised: Recently, thus many adaptive gradient methods Duchi et al.

Original: ; Kingma & Ba (2014); Loshchilov & Hutter (2018); Zhuang et al.
Revised: ; Kingma & Ba (2014); Loshchilov & Hutter (2018); Zhuang et al.

Original: (2020) have been developed and studied.
Revised: (2020) have been developed and studied.

Original: For example, Adagrad Duchi et al.
Revised: For example, Adagrad Duchi et al.

Original: (2011) is the first adaptive gradient method that shows good performances under the sparse gradient setting.
Revised: (2011) is the first adaptive gradient method that shows good performances under the sparse gradient setting.

Original: One variant of
Revised: One variant of

Original: Adagrad, i.e., Adam Kingma & Ba (2014) is a very popular adaptive gradient method and basically is a default method of choice for training DNNs.
Revised: Adagrad, i.e., Adam Kingma & Ba (2014) is a very popular adaptive gradient method and basically is a default method of choice for training DNNs.

Original: Subsequently, some variants of Adam algorithm
Revised: Subsequently, some variants of Adam algorithm

Original: Reddi et al.
Revised: Reddi et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2019) have been developed and studied, and especially they have convergence guarantee under the nonconvex setting.
Revised: (2019) have been developed and studied, and especially they have convergence guarantee under the nonconvex setting.

Original: At the same time, some adaptive gradient methods Loshchilov & Hutter (2018); Chen et al.
Revised: At the same time, some adaptive gradient methods Loshchilov & Hutter (2018); Chen et al.

Original: ; Zhuang et al.
Revised: ; Zhuang et al.

Original: (2020) have been presented to improve the generalization performance of Adam algorithm.
Revised: (2020) have been presented to improve the generalization performance of Adam algorithm.

Original: The norm version of AdaGrad (i.e.,
Revised: The norm version of AdaGrad (i.e.,

Original: AdaGrad-Norm)
Revised: AdaGrad-Norm)

Original: Ward et al.
Revised: Ward et al.

Original: (2019) has been presented to accelerate AdaGrad without sacrificing generalization.
Revised: (2019) has been presented to accelerate AdaGrad without sacrificing generalization.

Original: Moreover, some accelerated adaptive gradient methods such as STORM Cutkosky & Orabona (2019) and SUPER-ADAM Huang et al.
Revised: Moreover, some accelerated adaptive gradient methods such as STORM Cutkosky & Orabona (2019) and SUPER-ADAM Huang et al.

Original: (2021) have been proposed by using variancereduced technique.
Revised: (2021) have been proposed by using variancereduced technique.

Original: Meanwhile, Huang et al.
Revised: Meanwhile, Huang et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: (2021a) studied the convergence analysis framework for adaptive gradient methods.
Revised: (2021a) studied the convergence analysis framework for adaptive gradient methods.

Original: A. A DDITIONAL T HEORETICAL R ESULTS
Revised: A. A DDITIONAL T HEORETICAL R ESULTS

Original: In this subsection, we further give the convergence properties of our BiAdam algorithm for unconstrained bilevel optimization.
Revised: In this subsection, we further give the convergence properties of our BiAdam algorithm for unconstrained bilevel optimization.

Original: Theorem 3 Under the above Assumptions (1, 2, 4, 6, 7), in the Algorithm 1, given X = R d , η t = k
Revised: Theorem 3. Under the above Assumptions (1, 2, 4, 6, 7), in the Algorithm 1, given X = R d , η t = k

Original: ( m + t ) 1 / 2 for all t ≥ 0 , α t +1 =
Revised: ( m + t ) 1 / 2 for all t ≥ 0 , α t +1 =

Original: c 1 η t , β t +

Revised: c 1 η t , β t +

Original: =
Revised: =

Original: c
Revised: c

Original: 2 η t , m ≥ max (cid:0) k 2 , ( c 1 k ) 2 , ( c 2 k ) 2 (cid:1) , k > 0 , 125 L {+
+}
Revised: 2 η t , m ≥ max (cid:0) k 2 , ( c 1 k ) 2 , ( c 2 k ) 2 (cid:1) , k > 0 , 125 L {+
+}

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  Lk (cid:1) and K =
Revised: 4 Lk (cid:1) and K =

Original: L g µ log( C gxy C fy T/µ ) , we have
Revised: L g µ log( C gxy C fy T/µ ) , we have

Original: [Equation]
Revised: [Equation]

Original: whereG
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Remark 3 Under the same conditions in Theorem 1, based on the metric E ∥∇ f ( x ) ∥ , our BiAdam algorithm still has a gradient complexity of ˜ O
Revised: Remark 3. Under the same conditions in Theorem 1, based on the metric E ∥∇ f ( x ) ∥ , our BiAdam algorithm still has a gradient complexity of ˜ O

Original: ( ϵ − 4 ) without relying on the large mini-batches.
Revised: ( ϵ − 4 ) without relying on the large mini-batches.

Original: Interestingly, the right hand side of the above inequality (14) includes a term
Revised: Interestingly, the right hand side of the above inequality (14) includes a term

Original: [Equation]
Revised: [Equation]

Original: that can be seen as an upper bound of the expected condition number of adaptive matrices { A t } Tt =1 .
Revised: that can be seen as an upper bound of the expected condition number of adaptive matrices { A t } Tt =1 .

Original: When
Revised: When

Original: A t given in the above (4), we have
Revised: A t given in the above (4), we have

Original: [Equation]
Revised: [Equation]

Original: as in the existing adaptive gradient methods assuming the standard bounded stochastic gradient ∥∇ f ( x ; ξ ) ∥ ≤ G 1 .
Revised: as in the existing adaptive gradient methods assuming the standard bounded stochastic gradient ∥∇ f ( x ; ξ ) ∥ ≤ G 1 .

Original: Next, we further give the convergence properties of our VR-BiAdam algorithm for unconstrained bilevel optimization.
Revised: Next, we further give the convergence properties of our VR-BiAdam algorithm for unconstrained bilevel optimization.

Original: Theorem 4 Under the above Assumptions (1, 3, 4, 6, 7), in the Algorithm 2, given X = R d , η t = k
Revised: Theorem 4. Under the above Assumptions (1, 3, 4, 6, 7), in the Algorithm 2, given X = R d , η t = k

Original: ( m + t ) 1 / 3 for all t ≥ 0 , α t +1 =
Revised: ( m + t ) 1 / 3 for all t ≥ 0 , α t +1 =

Original: c 1 η 2 t , β t +

Revised: c 1 η 2 t , β t +

Original: = c 2 η 2 t , m ≥ max (cid:0) 2 , k 3 , ( c 1 k ) 3 , ( c 2 k ) 3 (cid:1) , k > 0 , c 1 ≥ 23 k 3 + 125 L 20 6 µ 2 , c 2 ≥ 23 k 3 + 92 , 0 < λ ≤ min (cid:0) 15 b l L 20 16 L 22 µ , b l 6 L g (cid:1) , 0 < γ ≤ min (cid:0) 
Revised: = c 2 η 2 t , m ≥ max (cid:0) 2 , k 3 , ( c 1 k ) 3 , ( c 2 k ) 3 (cid:1) , k > 0 , c 1 ≥ 23 k 3 + 125 L 20 6 µ 2 , c 2 ≥ 23 k 3 + 92 , 0 < λ ≤ min (cid:0) 15 b l L 20 16 L 22 µ , b l 6 L g (cid:1) , 0 < γ ≤ min (cid:0) [Equation] λµρ

Original: [Equation]
Revised: [Equation]

Revised: L

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original:  , m 
Revised: 125 b 2 u [Equation] [Equation] 2 , m 1 /

Original: [Equation]
Revised: [Equation]

Original:  (cid:1) and K =
Revised: Lk (cid:1) and K =

Original: L g µ log( C gxy C fy T/µ ) , we have
Revised: L g µ log( C gxy C fy T/µ ) , we have

Original: [Equation]
Revised: [Equation]

Original:  where M ′ =
Revised: (15) where M ′ =

Original: ρ ( F ( x 1 ) − F ∗ ) kγ
Revised: ρ ( F ( x 1 ) − F ∗ ) kγ

Original: [Equation]
Revised: [Equation]

Original: Remark 4 Under the same conditions in Theorem 2, based on the metric E ∥∇ f ( x ) ∥ , our VRBiAdam algorithm still has a gradient complexity of ˜ O
Revised: Remark 4. Under the same conditions in Theorem 2, based on the metric E ∥∇ f ( x ) ∥ , our VRBiAdam algorithm still has a gradient complexity of ˜ O

Original: ( ϵ − 3 ) without relying on the large minibatches.
Revised: ( ϵ − 3 ) without relying on the large minibatches.

Original: A. D ETAILED C ONVERGENCE A NALYSIS
Revised: A. D ETAILED C ONVERGENCE A NALYSIS

Original: In this subsection, we provide the detailed convergence analysis of our algorithms.
Revised: In this subsection, we provide the detailed convergence analysis of our algorithms.

Original: We first review and provide some useful lemmas.
Revised: We first review and provide some useful lemmas.

Original: Given a ρ -strongly convex function ϕ ( x ) , we define a prox-function (Bregman distance) Censor &
Revised: Given a ρ -strongly convex function ϕ ( x ) , we define a prox-function (Bregman distance) Censor &

Original: Lent (1981); Censor & Zenios (1992) associated with ϕ ( x ) as follows:
Revised: Lent (1981); Censor & Zenios (1992) associated with ϕ ( x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Then we define a generalized projection problem as in Ghadimi et al.
Revised: Then we define a generalized projection problem as in Ghadimi et al.

Original: [Equation]
Revised: [Equation]

Original: where X ⊆ R d , w ∈ R d and γ > 0 .
Revised: where X ⊆ R d , w ∈ R d and γ > 0 .

Original: Here h ( x ) is convex and possibly nonsmooth function.
Revised: Here h ( x ) is convex and possibly nonsmooth function.

Original: At the same time, we define a generalized gradient as follows:
Revised: At the same time, we define a generalized gradient as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Lemma 6 (Lemma 1 in Ghadimi et al.
Revised: Lemma 6. (Lemma 1 in Ghadimi et al.

Original: Let x ∗ be given in equation 17.
Revised: Let x ∗ be given in (17).

Original: Then, for any x ∈ X , w ∈ R d and γ > 0 , we have
Revised: Then, for any x ∈ X , w ∈ R d and γ > 0 , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ρ > 0 depends on ρ -strongly convex function ϕ ( x ) .
Revised: where ρ > 0 depends on ρ -strongly convex function ϕ ( x ) .

Original: When h ( x ) = 0 , in the above lemma 6, we have
Revised: When h ( x ) = 0 , in the above lemma 6, we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Lemma 7 (Restatement of Lemma 5) When the gradient estimator w t generated from Algorithmor 2, for all t ≥ 1 , we have
Revised: Lemma 7. (Restatement of Lemma 5) When the gradient estimator w t generated from Algorithmor 2, for all t ≥ 1 , we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Revised: Proof.

Original: We first consider the term ∥∇ F ( x t ) − ¯ ∇ f ( x t , y t )
Revised: We first consider the term ∥∇ F ( x t ) − ¯ ∇ f ( x t , y t )

Original: ∥ 2 .
Revised: ∥ 2 .

Original: Since ∇ f ( x t , y ∗ ( x t ))
Revised: Since ∇ f ( x t , y ∗ ( x t ))

Original: = ∇ F ( x t ) , we have ∥∇ f ( x t , y ∗ ( x t )) − ¯ ∇ f ( x t , y t ) ∥= ∥∇ x f ( x t , y ∗ ( x t ))
Revised: = ∇ F ( x t ) , we have [Equation]

Original: − ∇ 2 xyg
Revised: + ∇ 2 xy

Original: x t , y ∗ ( x t )) (cid:0) ∇ 2 yy g ( x t , y ∗ ( x t )) (cid:1) − 1 ∇ y f
Revised: [Equation]

Original: ( x t, y ∗ ( x ))
Revised: ( x t

Original: [Equation]
Revised: [Equation]

Original: ∇ x f ( x t , y t ) + ∇ 2 xy g ( x t , y t ) (cid:0)
Revised: y t ) (cid:0)

Original: [Equation]
Revised: [Equation]

Original: 2 yyg
Revised: 2 yy

Revised: [Equation]

Revised: [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Revised: y

Revised: [Equation]

Revised: [Equation]

Original: x t 
Revised: t )) (cid:1) [Equation] [Equation] y

Original: [Equation]
Revised: [Equation]

Original: y t
Revised: ( x t )) [Equation] xy g ( x t

Original: [Equation]
Revised: [Equation]

Original: (cid:1)
Revised: y t ) (cid:0) [Equation] yy g ( [Equation]

Revised: t

Original: [Equation]
Revised: [Equation]

Revised: y

Original: [Equation]
Revised: [Equation]

Original: y f ( x t , y t ) ∥=
Revised: x t )) (cid:1)

Original: ∥∇ x f ( x t , y ∗ ( x t ))
Revised: [Equation]

Original: − ∇ x f ( x t , y t ) − ∇ 2 xy g ( x t , y ∗ ( x t )) (cid:0) ∇ 2 yy g
Revised: [Equation]

Original: ( x t , y ∗ ( x t ))
Revised: y f ( x t [Equation]

Original: (cid:1) − 1 ∇ y f
Revised: y [Equation]

Original: ( x t , y ∗ ( x t ))
Revised: [Equation] t [Equation]

Original: + ∇
Revised: y t

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: = 4¯

Original: [Equation]
Revised: [Equation]

Original:  where the second last inequality is due to Assumptions 1, 2 and 4; the last equality holds by ¯ L 2 =
Revised: (22) where the second last inequality is due to Assumptions 1, 2 and 4; the last equality holds by ¯ L 2 =

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Lemma 8 Under the Assumptions 1, 2, 4, we have
Revised: Lemma 8. Under the Assumptions 1, 2, 4, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where L 2 0 = 8 (cid:0) L2 f + L 2 gxy C 2 fy µ
Revised: where L 2 0 = 8 (cid:0) L

Original: [Equation]
Revised: [Equation]

Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original:  where the first inequality holds by the Assumptions 1 and
Revised: (25) where the first inequality holds by the Assumptions 1 and

Original: [Equation]
Revised: [Equation]

Original: and the second inequality holds by the
Revised: and the second inequality holds by the

Original: Assumption
Revised: Assumption

Original: [Equation]
Revised: [Equation]

Original: Lemma 9 Suppose that the sequence { x t , y t } Tt =1 be generated from Algorithm 1 or 2.
Revised: Lemma 9. Suppose that the sequence { x t , y t } Tt =1 be generated from Algorithm 1 or 2.

Original: Let 0 < η t ≤ 1 and 0 < γ ≤ ρ 2 Lη t , then we have
Revised: Let 0 < η t ≤ 1 and 0 < γ ≤ ρ 2 Lη t , then we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: Proof.

Original: According to Lemma 2, the function F ( x ) is L -smooth.
Revised: According to Lemma 2, the function F ( x ) is L -smooth.

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: where the second equality is due to x t
Revised: where the second equality is due to x t

Original: +1 = x t + η t
Revised: +1 = x t + η t

Original: ˜ x t +1 − x t ) .
Revised: ˜ x t +1 − x t ) .

Original: According to Assumption 7, i.e., A t ≻ ρI d for any t ≥ 1 , the function ϕ t ( x ) = 12 x T
Revised: According to Assumption 7, i.e., A t ≻ ρI d for any t ≥ 1 , the function ϕ t ( x ) = 12 x T

Original: A t x is ρstrongly convex, then we define a prox-function (a.k.a.
Revised: A t x is ρstrongly convex, then we define a prox-function (a.k.a.

Original: Bregman distance) associated with ϕ t ( x ) as in Censor & Zenios (1992); Ghadimi et al.
Revised: Bregman distance) associated with ϕ t ( x ) as in Censor & Zenios (1992); Ghadimi et al.

Original: [Equation]
Revised: [Equation]

Original: By using the above Lemma 6 to the problem ˜ x t +1 = arg
Revised: By using the above Lemma 6 to the problem ˜ x t +1 = arg

Original: [Equation]
Revised: [Equation]

Original: x ∈X
Revised: x ∈X

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x ⟩ + 12 γ ( x − x t ) T 
Revised: ) T [Equation]

Original: A t ( x − x t ) (cid:9) at the line 5 of Algorithm 1 or 2, we have
Revised: t ( x − x t ) (cid:9) at the line 5 of Algorithm 1 or 2, we have

Original: [Equation]
Revised: [Equation]

Original: Then we obtain
Revised: Then we obtain

Original: [Equation]
Revised: [Equation]

Original: Next, consider the bound of the term T 2 , we have
Revised: Next [Equation] consider the bound of the term T [Equation]

Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to the Cauchy-Schwarz inequality and the last is due to Young’s inequality.
Revised: where the first inequality is due to the Cauchy-Schwarz inequality and the last is due to Young’s inequality.

Original: By combining the above inequalities equation
Revised: By combining the above inequalities (27)

Original: [Equation]
Revised: [Equation]

Original: equation 30 with equation
Revised: (30) with (31)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to 0 < γ ≤
Revised: where the last inequality is due to 0 < γ ≤

Original: [Equation]
Revised: [Equation]

Original: Lemma 10 Suppose the sequence { x t , y t } Tt =1 be generated from Algorithm 1 or 2.
Revised: Lemma 10. Suppose the sequence { x t , y t } Tt =1 be generated from Algorithm 1 or 2.

Original: Under the above assumptions, given 0 < η t ≤ 1 , B t =
Revised: Under the above assumptions, given 0 < η t ≤ 1 , B t =

Original: b t
Revised: b t

Original: I p ( b u ≥ b t ≥ b l > 0) for all t ≥ 1 , and 0 < λ ≤ b l 6 L g , we have
Revised: I p ( b u ≥ b t ≥ b l > 0) for all t ≥ 1 , and 0 < λ ≤ b l 6 L g , we have

Original: [Equation]
Revised: [Equation]

Original: where κ =
Revised: where κ =

Original: L g /µ .
Revised: L g /µ .

Original: According to Assumption 1, i

Original: [Equation]
Revised: Proof.

Original: e., the function g ( x, y ) is µ -strongly convex w.r.t y , we have[Equation] [Equation] According to the Assumption 2, i
Revised: According to Assumption 1, i.e., the function g ( x, y ) is µ -strongly convex w.r.t y , we have

Original: [Equation]
Revised: [Equation]

Original: e., the function g ( x, y ) is L g -smooth, we have
Revised: According to the Assumption 2, i.e., the function g ( x, y ) is L g -smooth, we have

Original: [Equation]
Revised: [Equation]

Original: Combining the about inequalities equation 34 with equation 35, we have
Revised: Combining the about inequalities (34) with (35), we have

Original: [Equation]
Revised: [Equation]

Original: By the optimality condition of the problem ˜ y
Revised: By the optimality condition of the problem ˜ y

Original: [Equation]
Revised: [Equation]

Original: −

Original: [Equation]
Revised: [Equation]

Original: at the line 6 of Algorithm 1 or 2, given B t = b t I p , we have
Revised: at the line 6 of Algorithm 1 or 2, given B t = b t I p , we have

Original: [Equation]
Revised: [Equation]

Original: Then we obtain
Revised: Then we obtain

Original: [Equation]
Revised: [Equation]

Original: By pugging the inequalities equation 38 into equation 36, we have
Revised: By pugging the inequalities (38) into (36), we have

Original: [Equation]
Revised: [Equation]

Original: Let y =
Revised: Let y =

Original: y ∗ ( x t )
Revised: y ∗ ( x t )

Original: [Equation]
Revised: [Equation]

Original: then we have
Revised: then we have

Original: [Equation]
Revised: [Equation]

Original: Due to the strongly-convexity of g ( ·, y ) and y ∗ ( x t )
Revised: Due to the strongly-convexity of g ( ·

Original: [Equation]
Revised: [Equation]

Original: y  ∈Y g ( x t , y )
Revised: y ) and y ∗ ( x t ) = arg min y ∈Y g ( x t [Equation] y )

Original: [Equation]
Revised: [Equation]

Original: we have g ( x t , y ∗ ( x t )) ≤ g ( x t , ˜ y t +1 ) .
Revised: we have g ( x t , y ∗ ( x t )) ≤ g ( x t , ˜ y t +1 ) .

Original: Thus, we obtain
Revised: Thus, we obtain

Original: [Equation]
Revised: [Equation]

Original: y t

Original: [Equation]

Original: By y t +1 =
Revised: By y t +1 =

Original: y t + η t
Revised: y t + η t

Original: ˜ y t
Revised: ˜ y t

Original: +1 − y t ) , we have
Revised: +1 − y t ) , we have

Original: [Equation]
Revised: [Equation]

Original: Then we obtain
Revised: Then we obtain

Original: [Equation]
Revised: [Equation]

Original: Consider the upper bound of the term ⟨∇ y g ( x t 
Revised: Consider the upper bound of the term ⟨∇ y g ( x t , y t ) − v t , y ∗ ( x t ) − ˜ y t +1 ⟩

Original: [Equation]
Revised: [Equation]

Original: y t ) − v t
Revised: we have ⟨∇

Original: [Equation]
Revised: [Equation]

Original: y ∗ ( x t) − ˜ y t +1 ⟩
Revised: t

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: t ,

Original: [Equation]
Revised: [Equation]

Original: By plugging the inequalities equation 43 and equation 44 into equation
Revised: By plugging the inequalities (43) and (44) into (41)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds by L g ≥ µ and 0 < η t ≤ [Equation] and the last inequality is due to 0{+
+} < λ ≤ b l 6 L g ≤ b t 6 L g .
Revised: where the second inequality holds by L g ≥ µ and 0 < η t ≤ 1 , and the last inequality is due to {+
+} < λ ≤ b l 6 L g ≤ b t 6 L g .

Original: It implies that
Revised: It implies that

Original: [Equation]
Revised: [Equation]

Original: Next, we decompose the term ∥
Revised: Next, we decompose the term ∥

Original: y t +1 − y ∗ ( x t +1 )
Revised: y t +1 − y ∗ ( x t +1 )

Original: ∥ 2 as follows:
Revised: ∥ 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: where the first inequality holds by Cauchy-Schwarz inequality and Young’s inequality [Equation] and the second inequality is due to Lemma
Revised: where the first inequality holds by Cauchy-Schwarz inequality and Young’s inequality, and the second inequality is due to Lemma

Original: [Equation]
Revised: [Equation]

Original: and the last equality holds by x t
Revised: and the last equality holds by x t

Original: +1 = x t + η t
Revised: +1 = x t + η t

Original: ˜ x t +1 − x t ) .
Revised: ˜ x t +1 − x t ) .

Original: By combining the above inequalities equation 46 and equation
Revised: By combining the above inequalities (46) and (47)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Revised: η

Revised: [Equation]

Original:  Since 0 < η t ≤ 1 , 0 < λ ≤ b l 6 L g ≤ b t 6 L g and L g ≥ µ , we have λ ≤ b t 6 L g ≤ b t 6 µ and η t ≤ 1 ≤ b t 6 µλ .
Revised: − [Equation] [Equation] Since 0 < η t ≤ 1 , 0 < λ ≤ b l 6 L g ≤ b t 6 L g and L g ≥ µ , we have λ ≤ b t 6 L g ≤ b t 6 µ and η t ≤ 1 ≤ b t 6 µλ .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where the second last inequality is due to η t µλb t ≤ 16 and the last inequality holds by b t 6 µλη t ≥ 1 .
Revised: where the second last inequality is due to η t µλb t ≤ 16 and the last inequality holds by b t 6 µλη t ≥ 1 .

Original: By using x t +1 = x t + η t (˜ x t +1 − x t ) , then we have
Revised: By using x t +1 = x t + η t (˜ x t +1 − x t ) , then we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A. C ONVERGENCE A NALYSIS OF B I A DAM A LGORITHM
Revised: A. C ONVERGENCE A NALYSIS OF B I A DAM A LGORITHM

Original: In the subsection, we provide the detail convergence analysis of BiAdam algorithm.
Revised: In the subsection, we provide the detail convergence analysis of BiAdam algorithm.

Original: For notational simplicity, let R t
Revised: For notational simplicity, let R t

Original: = R ( x t , y t ) for all t ≥1 .
Revised: = R ( x t , y t ) for all t ≥

Original:  Lemma 11 Assume that the stochastic partial derivatives v t +1 , and w t +1 be generated from Algorithm 1, we have
Revised: [Equation] Lemma 11. Assume that the stochastic partial derivatives v t +

Original: [Equation]
Revised: [Equation]

Original: E
Revised: and w t +1 be generated from Algorithm 1, we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: where L 20 = 8 (cid:0) L 2 f + L 2 gxy C 2 fy µ+ L 2 gyy C 2 gxy C
Revised: where L 20 = 8 (cid:0) [Equation] for all t ≥ 1 . Proof.

Revised: Without loss of generality, we only prove the term E ∥ w

Original: [Equation]
Revised: [Equation]

Original: + L 2 f C 2 gxy µ(cid:1) and R t =
Revised: 1 − ¯ ∇ f [Equation]

Original: ¯ ∇ f ( x t , y t ) − E ¯ ξ [ ¯ ∇ f ( x t , y t ; ¯ ξ )] for all t ≥ 1 .
Revised: x [Equation] , y [Equation]

Original: Without loss of generality, we only prove the term E ∥ w t +1 − ¯ ∇ f ( x t +1 , y t +1 ) − R t +1 ∥ 2 .
Revised: ) − R [Equation] 1 ∥ [Equation]

Original: The other term is similar for this term.
Revised: The other term is similar for this term.

Original: Since w t +1 = β t +1 ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 )
Revised: Since w t +1 = β t +1 ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +

Original:  + (1 − β t+1 ) w t , we have
Revised: [Equation] + [Equation] 1 − β t

Original: E
Revised: [Equation] w t , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: ≤

Original: [Equation]
Revised: [Equation]

Original:  where the third equality is due to E ¯ ξ t +1 [ ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 )]
Revised: 52) where the third equality is due to E ¯ ξ t +1 [ ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 )]

Original: = ¯ ∇ f ( x t +1 , y t +1 )
Revised: = ¯ ∇ f ( x t +1 , y t +1 )

Original: + R t +1 ; the second last inequality holds by 0 ≤ β t +1 ≤ 1 such that (1 − β t +1 ) 

Revised: + R t +1 ; the second last inequality holds by 0 ≤ β t +1 ≤ 1 such that (1 − β t +1 )

Original: (1+ β t +1 )
Revised: (1+ β t +1 )

Original: = 1 − β t +1 − β 2 t
Revised: = 1 − β t +1 − β 2 t

Original: +1 + β 3 t +1 ≤ 1 − β t
Revised: +1 + β 3 t +1 ≤ 1 − β t

Original: +

Revised: +

Original: and (1 − β t +1 ) 

Revised: and (1 − β t +1 )

Original: (1 + 1 β t +1 ) ≤ (1 − β t +1 )
Revised: (1 + 1 β t +1 ) ≤ (1 − β t +1 )

Original: (1 + 1 β t +1 )
Revised: (1 + 1 β t +1 )

Original: = − β t +1 + 1 β t +1 ≤ 1 β t +1 , and the last inequality holds by the above Lemma 8 and x t +

Revised: = − β t +1 + 1 β t +1 ≤ 1 β t +1 , and the last inequality holds by the above Lemma 8 and x t +

Original: = x t + η t
Revised: = x t + η t

Original: ˜ x t +1 − x t ) , y t +1 =
Revised: ˜ x t +1 − x t ) , y t +1 =

Original: y t + η t
Revised: y t + η t

Original: ˜ y t
Revised: ˜ y t

Original: +1 − y t ) .
Revised: +1 − y t ) .

Original: Theorem 5 (Restatement of Theorem 1) Under the above Assumptions (1, 2, 4, 6, 7), in the
Revised: Theorem 5. (Restatement of Theorem 1) Under the above Assumptions (1, 2, 4, 6, 7), in the

Original: Algorithm 1, given X ⊂ R d , η t = k
Revised: Algorithm 1, given X ⊂ R d , η t = k

Original: ( m + t ) 1 / 2 for all t ≥ 0 , α t +1 =
Revised: ( m + t ) 1 / 2 for all t ≥ 0 , α t +1 =

Original: c 1 η t , β t +

Revised: c 1 η t , β t +

Original: =
Revised: =

Original: c
Revised: c

Original: 2 η t , m ≥ max (cid:0) k 2 , ( c 1 k ) 2 , ( c 2 k ) 2 (cid:1) , k > 0 , 125 L 20 6 µ≤ c 1 ≤ m 1 / 2 k , 92 ≤ c 2 ≤ m 1 / 2 k , 0 < λ ≤ min (cid:0) 15 b l L 20 4 L 21 µ , b l 6 L g (cid:1) , 0 < γ ≤ min (cid:0) √ 6 λµρ √ 6 L 21 λ 2 µ 2 +125 b 2 u L 20 κ 2 , m 1 / 2 ρ 4 Lk (cid:1) and K
Revised: 2 η t , m ≥ max (cid:0) k 2 , ( c 1 k ) 2 , ( c 2 k ) 2 (cid:1) , k > 0 , 125 L 20 6 µ≤ c 1 ≤ m 1 / 2 k , 92 ≤ c 2 ≤ m 1 / 2 k , 0 < λ ≤ min (cid:0) [Equation] [Equation] [Equation]

Original: = L g µ log( C gxy C fy T/µ ) , we have
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where G = F ( x 1 ) − F ∗ ρkγ
Revised: g (cid:1) , 0 < γ ≤ min (cid:0)

Original: [Equation]
Revised: [Equation]

Original: + [Equation] +
Revised: and K =

Original: [Equation] mσ [Equation] k ln( m + T )+ 4( m + T ) 
Revised: L g µ log( C gxy C fy T/µ ) , we have

Original: [Equation]
Revised: [Equation]

Original: kT [Equation] +
Revised: where G = F ( x 1 ) − F

Original: [Equation]
Revised: [Equation]

Original: ρ [Equation] T , ∆ 0 = ∥ y 1 − y ∗ ( x 1 ) ∥and L  

Revised: and L 21 =

Original: = 12 L 2 g µ 2 125 L 20 + 2 L 20 3 .
Revised: L 2 g µ 2 125 L 20 + 2 L 20 3 .

Revised: Proof.

Original: Since η t = k ( m + t ) 1 / 2 on t is decreasing and m ≥ k 2 , we have η t ≤ η 0 =
Revised: Since η t = k ( m + t ) 1 / 2 on t is decreasing and m ≥ k 2 , we have η t ≤ η 0 =

Original: km 1 / 2 ≤ 1 and γ ≤ m 1 / 2 ρ 4 Lk ≤ ρ 2 Lη 0 ≤ ρ 2 Lη t for any t ≥ 0 .
Revised: km 1 / 2 ≤ 1 and γ ≤ m 1 / 2 ρ 4 Lk ≤ ρ 2 Lη 0 ≤ ρ 2 Lη t for any t ≥ 0 .

Original: Due to 0 < η t ≤ 1 and m ≥ ( c 1 k ) 2 , we have α t +

Revised: Due to 0 < η t ≤ 1 and m ≥ ( c 1 k ) 2 , we have α t +

Original: =
Revised: =

Original: c 1 η t ≤ c 1 k m 1 / 2 ≤ 1 .
Revised: c 1 η t ≤ c 1 k m 1 / 2 ≤ 1 .

Original: Similarly, due to m ≥ ( c 2 k ) 2 , we have β t +1 ≤ 1 .
Revised: Similarly, due to m ≥ ( c 2 k ) 2 , we have β t +1 ≤ 1 .

Original: At the same time, we have c 1 , c 2 ≤ m 1 / 2 k .
Revised: At the same time, we have c 1 , c 2 ≤ m 1 / 2 k .

Original: According to Lemma 11, we have
Revised: According to Lemma 11, we have

Original: [Equation]
Revised: [Equation]

Original: where the above equality holds by α t
Revised: where the above equality holds by α t

Original: +1 =
Revised: +1 =

Original: c 1 η t , and the last inequality is due to 125 L 20 6 µ 2 ≤ c 1 ≤ m 1 / 2 k .
Revised: c 1 η t , and the last inequality is due to 125 L 20 6 µ 2 ≤ c 1 ≤ m 1 / 2 k .

Original: Similarly, we have
Revised: Similarly, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds by β t
Revised: where the last inequality holds by β t

Original: +1 =
Revised: +1 =

Original: c 2 η t and 92 ≤ c 2 ≤ m 1 / 2 k .
Revised: c 2 η t and 92 ≤ c 2 ≤ m 1 / 2 k .

Original: According to Lemmas 7 and9, we have
Revised: According to Lemmas 7 and

Original: [Equation]
Revised: [Equation]

Revised: we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 10, we have
Revised: According to Lemma [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Next, we define a Lyapunov function (i
Revised: Next [Equation] we define a Lyapunov function (i

Original: [Equation]
Revised: [Equation]

Original: e., potential function), for any t ≥ 1 ,
Revised: e., potential function), for any t ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Revised: ∥

Revised: [Equation]

Revised: + ∥

Revised: w

Revised: [Equation]

Revised: − ¯ ∇ f

Revised: [Equation]

Revised: x t

Revised: [Equation]

Revised: y t ) − R t ∥

Revised: [Equation]

Original: For notational simplicity, let L 21 = 12 L 2 g µ 2 125 L 20 + 2 L 20 3 .
Revised: For notational simplicity, let L 21 = 12 L 2 g [Equation] 125 L 20 + 2 L 20 3 .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original:  where the first inequality holds by the above inequalities equation 54, equation 55, equation 56 and equation 57; the last inequality is due to 0 < γ ≤
Revised: 58) where the first inequality holds by the above inequalities (54), (55), (56) and (57); the last inequality is due to 0 < γ ≤

Original: [Equation]
Revised: [Equation]

Original:  all t ≥ 1 .
Revised: 6 L 21 λ 2 µ 2 +125 b 2 t L 20 κ 2 , 0 < λ ≤ 15 b [Table] L 20 4 L 21 µ ≤ 15 b t L 20 4 L 21 µ for all t ≥ 1 .

Original: Let Φ t =
Revised: Let Φ t =

Original: L [Equation] y
Revised: L 20 ∥

Original: [Equation] − y [Equation] + 2 ∥w
Revised: y t − y ∗ ( x t ) ∥ 2 + 2 ∥

Original: [Equation] − ¯ ∇ f ( x t , y t ) − R t ∥ 2 , we have 
Revised: w t − ¯ ∇ f ( x t , y t ) − R t ∥ 2 , we have γη t

Original: [Equation]
Revised: [Equation]

Original: Taking average over t = 1 , 2 , · · · , T on both sides of equation 59, we have
Revised: Taking average over t = 1 , 2 , · · · , T on both sides of (59), we have

Original: [Equation]
Revised: [Equation]

Original: Given x 1 ∈ X and y 1 ∈ Y , let ∆ 

Revised: Given x 1 ∈ X and y 1 ∈ Y , let ∆

Original: = ∥
Revised: = ∥

Original: y 1 − y ∗ ( x 1 ) ∥ 2 , we have Γ
Revised: y 1 − y ∗ ( x 1 ) ∥ 2 , we have Γ

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds by Assumption 2.
Revised: E

Original: Since η t is decreasing on t  , i
Revised: [Equation] F ( x t ) + 5 b 1 L 20 γ λµρ ∥ y 1 − y ∗ ( x 1 ) ∥ 2 + γ ρ (cid:0) ∥ v 1 − ∇ y g x 1 , y 1 ) ∥ 2 + ∥ w 1 − ¯ ∇ f ( x

Original: [Equation]
Revised: [Equation]

Original: e., η − 1 T ≥ η −  1 t for any 0 ≤ t ≤ T , we have
Revised: y 1 ) − R 1 ∥ 2 (cid:1)(cid:3) ≤ F ( x 1 ) +

Original: [Equation]
Revised: [Equation]

Original:  ≤ 
Revised: where the last inequality holds by Assumption 2. Since η t is decreasing on t , i.e., η − 1 T ≥ η − 1 t for any 0 ≤ t ≤ T , we have

Original: [Equation]
Revised: [Equation]

Original: ≤ ρ Tγη T
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: Tγη
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: Tη
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: λµ
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: Tη
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: mσ
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: Tη
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:90)
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds by the above inequality equation 60 and ∥ R t ∥ ≤ 1 T for all t ≥by choosing K
Revised: where the second inequality holds by the above inequality (60) and ∥ R t ∥ ≤ 1 T for all t ≥ 1 by choosing K

Original: = L g µ log( C gxy C fy T/µ ) in Algorithm 1.
Revised: = L g µ log( C gxy C fy T/µ ) in Algorithm 1.

Original: Let G = F ( x 1 ) − F ∗ ρkγ
Revised: Let G = F ( x 1 ) − F ∗ ρkγ

Original: [Equation]
Revised: [Equation]

Original:  m 
Revised: ln [Equation] m +

Original: [Equation]
Revised: [Equation]

Revised: +

Original: 2 kT 2 + 8 k ρ 2 T , we have
Revised: [Equation] + [Equation] , we have

Original: [Equation]
Revised: [Equation]

Original: According to the Jensen’s inequality
Revised: According to the Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to ( a + b ) 1 / 4 ≤ a 1 / 4 + b 1 / 4 for all a [Equation] b > 0 .
Revised: where the last inequality is due to ( a + b ) 1 / 4 ≤ a 1 / 4 + b 1 / 4 for all a, b > 0 .

Original: Thus we have
Revised: Thus we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since ∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ ≥ ∥ w t − ¯ ∇ f ( x t , y t )
Revised: Since ∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ ≥ ∥ w t − ¯ ∇ f ( x t , y t )

Original: ∥ − ∥ R t ∥ , by the above inequality equation 63, we can obtain 
Revised: ∥ − ∥ R t ∥ , by the above inequality (63), we can obtain [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to ∥ R t ∥ ≤ 1 T for all t ≥ 1 by choosing K
Revised: where the last inequality is due to ∥ R t ∥ ≤ 1 T for all t ≥ 1 by choosing K

Original: = L g µ log( C gxy C fy T/µ ) in Algorithm 1.
Revised: = L g µ log( C gxy C fy T/µ ) in Algorithm 1.

Original: According to the above inequality equation [Equation] we have
Revised: According to the above inequality (11), we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 6 (Restatement of Theorem 3) Under the above Assumptions (1, 2, 4, 6, 7), in the
Revised: Theorem 6. (Restatement of Theorem 3) Under the above Assumptions (1, 2, 4, 6, 7), in the

Original: Algorithm 1, given X = R d , η t = k
Revised: Algorithm 1, given X = R d , η t = k

Original: ( m + t ) 1 / 2 for all t ≥ 0 , α t +1 =
Revised: ( m + t ) 1 / 2 for all t ≥ 0 , α t +1 =

Original: c 1 η t , β t +

Revised: c 1 η t , β t +

Original: =
Revised: =

Original: c
Revised: c

Original: 2 η t , m ≥ max (cid:0) k 2 , ( c 1 k ) 2 , ( c 2 k ) 2 (cid:1) , k > 0 , 125 L 20 6 µ≤ c [Equation] ≤ m [Equation]
Revised: 2 η t , m ≥ max (cid:0) k 2 , ( c 1 k ) 2 , ( c 2 k ) 2 (cid:1) , k > 0 , 125 L 20 6 µ≤ c 1 ≤ m 1 / 2 k , 92 ≤ c 2 ≤ m 1 / 2 k , 0 < λ ≤ min (cid:0) 15 b l L 20 4 L 21 µ , b l 6 L g (cid:1) , 0 < γ ≤ min (cid:0) √ 6 λµρ √ 6 L 21 λ 2 µ 2 +125 b 2 u L 20 κ 2 , m 1 / 2 ρ 4 Lk (cid:1) and K

Original: 92 ≤ [Equation] ≤ [Equation]
Revised: =

Original: < λ ≤ min (cid:0) [Equation] [Equation] 21 µ 
Revised: L g µ log( C gxy C fy T/µ ) , we have

Original: [Equation]
Revised: [Equation]

Original: g [Equation]
Revised: ) where G ′ =

Original: [Equation] γ [Equation] min
Revised: ρ ( F ( x 1 ) − F ∗ ) kγ

Original: [Equation] K = [Equation] g
Revised: + 5 b u L 20 ∆ 0 kλµ + 2 σ 2 k

Original: [Equation]
Revised: + 2 mσ

Original: log [Equation] C gxy [Equation]
Revised: 2 k ln( m + T )

Original: , we have [Equation] where G ′ = ρ ( F ( x
Revised: + 4( m + T ) 9 kT 2 + 8 kT .

Original: [Equation]
Revised: Proof.

Original: F
Revised: According to Lemma 7, we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 7, we have
Revised: ∇

Original: [Equation]
Revised: [Equation]

Original: where the equality ( i ) holds by ˜ x t
Revised: where the equality ( i ) holds by ˜ x t

Original: +1 = x t − γA − 1 t w t that can be easily obtained from the stepof Algorithm 1 when X = R d , and the inequality ( ii ) holds by ∥
Revised: +1 = x t − γA − 1 t w t that can be easily obtained from the stepof Algorithm 1 when X = R d , and the inequality ( ii ) holds by ∥

Original: A t ∥ ≥ ρ for all t ≥ 1 due to
Revised: A t ∥ ≥ ρ for all t ≥ 1 due to

Original: Assumption 7.
Revised: Assumption 7.

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: According to Cauchy-Schwarz inequality [Equation] we have
Revised: According to Cauchy-Schwarz inequality, we have

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where the third inequality holds by the above inequality equation
Revised: where the third inequality holds by the above inequality (61)

Original: [Equation]
Revised: [Equation]

Original: and the last inequality holds by ∥ R t ∥ ≤ 1 T for all t ≥ 1 by choosing K
Revised: and the last inequality holds by ∥ R t ∥ ≤ 1 T for all t ≥ 1 by choosing K

Original: = L g µ log( C gxy C fy T/µ ) .
Revised: = L g µ log( C gxy C fy T/µ ) .

Original: By combining the inequalities equation 69 and equation [Equation] we have
Revised: By combining the inequalities (69) and (70), we have

Original: [Equation]
Revised: [Equation]

Original: where G ′ =
Revised: where G ′ =

Original: ρ 2 G .
Revised: ρ 2 G .

Original: A C ONVERGENCE A NALYSIS OF VR-B I A DAM A LGORITHM
Revised: A. C ONVERGENCE A NALYSIS OF VR-B I A DAM A LGORITHM

Original: In the subsection, we detail convergence analysis of VR-BiAdam algorithm 
Revised: In the subsection [Equation] we detail convergence analysis of VR-BiAdam algorithm [Equation]

Original: Lemma 12 Under the above Assumptions (1, 3, 4), assume the stochastic gradient estimators v t and w t be generated from Algorithm 2, we have
Revised: Lemma 12. Under the above Assumptions (1, 3, 4), assume the stochastic gradient estimators v t and w t be generated from Algorithm 2, we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where L 2 K = 2 L 2 f + 6 C 2 gxy L 2 f
Revised: where L 2 K = 2 L 2 f + 6 C 2 gxy L 2 f

Original: [Equation]
Revised: [Equation]

Original:  Without loss of generality, we only prove the term E ∥ w t +1 − ¯ ∇ f ( x t +1 , y t +1 ) − R t +1 ∥ 2 .
Revised: Proof. Without loss of generality, we only prove the term E ∥ w t +1 − ¯ ∇ f ( x t +1 , y t +1 ) − R t +1 ∥ 2 .

Original: The other term is similar for this term.
Revised: The other term is similar for this term.

Original: Since w t +1 = ¯ ∇ f ( x t +1 , y t +1 ; ¯ [Equation] + [Equation]
Revised: Since w t +1 = ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 )

Original: 1 −

Original: [Equation]

Original: (cid:0) w t − ¯ ∇ f ( x t , y t ; ¯ ξ t +1 ) (cid:1) , we have

Original: [Equation]

Original: ( x

Original: [Equation]

Original: ¯

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ∥

Original: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: w

Original: [Equation]

Original: ¯

Original: [Equation]

Original: y

Original: [Equation]

Original: ) − R t )

Original: [Equation]

Original: t

Original: [Equation]

Original: 1 (cid:0) ¯

Original: [Equation]

Original: x t

Original: [Equation]

Original: [Equation]

Original: y

Original: [Equation]

Original: [Equation]

Original: ¯

Original: [Equation]

Original: [Equation]

Original: ¯

Original: [Equation]

Original: t

Original: [Equation]

Original: [Equation]

Original: y t

Original: [Equation]

Original: [Equation]

Original: R t

Original: [Equation]

Original: 1 (cid:1)

Original: [Equation]

Original: [Equation]

Original:  t [Equation] 1 ) (cid:0) [Equation]
Revised: + (1 − β t +1 ) (cid:0) w t −

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ∥

Original: [Equation]

Original: = (

Original: [Equation]

Original:  t +1 ) [Equation] ∥ w t [Equation]
Revised: ¯ ∇ f ( x t , y t ; ¯ ξ t +1 ) (cid:1) , we have

Original: ¯

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]

Original: y t ) [Equation] t ∥ [Equation]
Revised: where the third equality holds by E ¯ ξ

Original: ∥

Original: [Equation]

Original: t

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:0) ¯ ∇

Original: [Equation]

Original: ( x t +

Original: [Equation]

Original: y t +

Original: [Equation]

Original: ¯ξ t +1 ) − ¯ ∇ f[Equation] x t [Equation]
Revised: ¯ ∇ f

Original: [Equation]

Original: y t

Original: [Equation]
Revised: [Equation]

Original: 1 ) − R t

Original: [Equation]

Original: 1 (cid:1)

Original: [Equation]

Original: [Equation] t +1 ) (cid:0) ¯ ∇ f ( xt + [Equation] y t +
Revised: x

Original: [Equation]
Revised: [Equation]

Original: ¯ ξ t [Equation] [Equation] ¯
Revised: +

Original: [Equation]
Revised: [Equation]

Original: f ( x t +

Original: [Equation]

Original:  yt
Revised: , y

Original: [Equation]
Revised: [Equation]

Original: [Equation] R t +1 − ( ¯ [Equation] f ( x t
Revised: +

Original: [Equation]
Revised: [Equation]

Original: y t
Revised: ; ¯

Original: [Equation]
Revised: [Equation]

Original:  ¯ ξ t ) [Equation] ¯
Revised: = ¯ ∇ f

Original: [Equation]
Revised: [Equation]

Original: x t
Revised: x t

Original: [Equation]
Revised: [Equation]

Original:  y t) − R t ) (cid:1) ∥ [Equation] t
Revised: , y t

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation] t
Revised: + R t

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [-1 ) − R t +1 ∥+ 2(1 − β t +1 ) 
 ∥ ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 ) − ¯ ∇ f ( x t +1 , y t +1 )-]
Revised: [-1 ) − R t +1 ∥+ 2(1 − β t +1 ) 
 ∥ ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 ) − ¯ ∇ f ( x t +1 , y t +1 )-]and

Original: − R t +1 − ( ¯ ∇ f ( x t , y t ; ¯ ξ t ))

Original: − ¯ ∇ f ( x t , y t ) − R t ) ∥≤ (1 − β t +1 ) 2 E[-∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ 2 + 2 β 2 t +1 σ 2 + 2(1 − β t +1 ) 
 ∥ ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 )) − ¯ ∇ f ( x t , y t ; ¯ ξ t ) ∥≤ (1 − β t +1 ) 2 E ∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ 2 + 2 β 2 t +1 σ 2 + 4(1 − β t +1 ) 2 L 2 K (cid:0) ∥-]
Revised: E[-∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ 2 + 2 β 2 t +1 σ 2 + 2(1 − β t +1 ) 
 ∥ ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 )) − ¯ ∇ f ( x t , y t ; ¯ ξ t ) ∥≤ (1 − β t +1 ) 2 E ∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ 2 + 2 β 2 t +1 σ 2 + 4(1 − β t +1 ) 2 L 2 K (cid:0) ∥-]

Original: x t +1 − x t ∥ 2 + ∥ y t +1 − y t ∥ 2 (cid:1) ≤ (1 − β t +1 ) E ∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ 2 + 2 β 2 t +1 σ 2 + 4 L 2 K η 2 t (cid:0) ∥ ˜ x t +1 − x t ∥ 2 + ∥ ˜
Revised: [Equation]

Original: y t +1 − y t ∥ 2 (cid:1) , where the third equality holds by E ¯ ξ (cid:2) ¯ ∇ f ( x t +1 , y t +1 ; ¯ ξ t +1 ) (cid:3) =[-¯ ∇ f ( x t +1 , y t +1 ) + R t +
-]
Revised: t )) (cid:3) =[-¯ ∇ f ( x t +1 , y t +1 ) + R t +
-]

Original: and

Original: E ¯ ξ (cid:2) ¯ ∇ f ( x t , y t ; ¯ ξ t )) (cid:3) =

Original: ¯ ∇ f ( x t , y t )
Revised: ¯ ∇ f ( x t , y t )

Original: + R t ; the third last inequality holds by the inequality E ∥ ζ −
Revised: + R t ; the third last inequality holds by the inequality E ∥ ζ −

Original: E [ ζ ] ∥ 2 ≤ E ∥ ζ ∥ 2 ; the second last inequality is due to Lemma 4; the last inequality holds by 0 < β t +1 ≤ 1 and x t +1 = x t + η t
Revised: E [ ζ ] ∥ 2 ≤ E ∥ ζ ∥ 2 ; the second last inequality is due to Lemma 4; the last inequality holds by 0 < β t +1 ≤ 1 and x t +1 = x t + η t

Original: (˜ x t +1 − x t ) , y t +1 =
Revised: (˜ x t +1 − x t ) , y t +1 =

Original: y t + η t
Revised: y t + η t

Original: ˜ y t
Revised: ˜ y t

Original: +1 − y t ) .
Revised: +1 − y t ) .

Original: Theorem 7 (Restatement of Theorem 2) Under the above Assumptions (1, 3, 4, 6, 7), in the Algorithm 2, given X ⊂ R d , η t = k
Revised: Theorem 7. (Restatement of Theorem 2) Under the above Assumptions (1, 3, 4, 6, 7), in the Algorithm 2, given X ⊂ R d , η t = k

Original: ( m + t ) 1 / 3 for all t ≥ 0 , α t +1 =
Revised: ( m + t ) 1 / 3 for all t ≥ 0 , α t +1 =

Original: c 1 η 2 t , β t +

Revised: c 1 η 2 t , β t +

Original: = c 2 η 2 t , m ≥ max (cid:0) 2 , k 3 , ( c 1 k ) 3 , ( c 2 k ) 3 (cid:1) , k > 0 , c 1 ≥ 23 k 3 + 125 L 20 6 µ 2 , c 2 ≥ 23 k 3 + 92 , 0 < λ ≤ min (cid:0) 15 b l L 20 16 L 22 µ , b l 6 L g (cid:1) , 0 < γ ≤ min (cid:0)
Revised: = c 2 η 2 t , m ≥ max (cid:0) 2 , k 3 , ( c 1 k ) 3 , ( c 2 k ) 3 (cid:1) , k > 0 , c 1 ≥ 23 k 3 + 125 L 20 6 µ 2 , c 2 ≥ 23 k 3 + 92 , 0 < λ ≤ min (cid:0) 15 b l L 20 16 L 22 µ , b l 6 L g (cid:1) , 0 < γ ≤ min (cid:0)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original:  2 , m 1 / 

Revised: 20 κ 2 , m 1 /

Original: ρ 4 Lk (cid:1) and K =
Revised: ρ 4 Lk (cid:1) and K =

Original: L g µ log( C gxy C fy T/µ ) , we haveT
Revised: L g µ log( C gxy C fy T/µ ) , we have

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation] T [Equation] (75) where M = F ( x 1 ) − F ∗ ρkγ+
Revised: where M = F ( x 1 ) − F ∗ ρkγ

Original: [Equation]
Revised: [Equation]

Original: kλµ +

Original: [Equation][Equation]
Revised: [Equation]

Original: +

Original: [Equation]

Original: [Equation]

Original: 21 +

Original: [Equation]

Original: [Equation]

Original: [Equation] ( m + T ) [Equation] + 6 k ( m + T ) 1 / 3 ρ 2 T , ∆ 

Revised: ) σ 2 ln( m + T ) ρ+ 6 k ( m + T ) 1 / 3 ρ 2 T , ∆

Original: = ∥
Revised: = ∥

Original: y 1 − y ∗ ( x 1 ) ∥ 2 and L 22 =
Revised: y 1 − y ∗ ( x 1 ) ∥ 2 and L 22 =

Original: L 2 g + L 2 K .
Revised: L 2 g + L 2 K .

Revised: Proof.

Original: Since η t = k ( m + t ) 1 / 3 on t is decreasing and m ≥ k 3 , we have η t ≤ η 0 =
Revised: Since η t = k ( m + t ) 1 / 3 on t is decreasing and m ≥ k 3 , we have η t ≤ η 0 =

Original: km 1 / 3 ≤ 1 and γ ≤ m 1 / 3 ρ 4 Lk ≤ ρ 2 Lη 0 ≤ ρ 2 Lη t for any t ≥ 0 .
Revised: km 1 / 3 ≤ 1 and γ ≤ m 1 / 3 ρ 4 Lk ≤ ρ 2 Lη 0 ≤ ρ 2 Lη t for any t ≥ 0 .

Original: Due to 0 < η t ≤ 1 and m ≥ ( c 1 k ) 3 , we have α t +

Revised: Due to 0 < η t ≤ 1 and m ≥ ( c 1 k ) 3 , we have α t +

Original: =
Revised: =

Original: c 1 η 2 t ≤ c 1 η t ≤ c 1 k m 1 / 3 ≤ 1 .
Revised: c 1 η 2 t ≤ c 1 η t ≤ c 1 k m 1 / 3 ≤ 1 .

Original: Similarly, due to m ≥ ( c 2 k ) 3 , we have β t +1 ≤ 1 .
Revised: Similarly, due to m ≥ ( c 2 k ) 3 , we have β t +1 ≤ 1 .

Original: According to Lemma 12, we have
Revised: According to Lemma 12, we have

Original: [Equation]
Revised: [Equation]

Original: where the second equality is due to α t
Revised: where the second equality is due to α t

Original: +1 =
Revised: +1 =

Original: c 1 η 2 t .
Revised: c 1 η 2 t .

Original: Similarly [Equation] we have
Revised: Similarly, we have

Original: [Equation]
Revised: [Equation]

Original: By η t = k
Revised: By η t = k

Original: ( m + t ) 1 / 3 , we have
Revised: ( m + t ) 1 / 3 , we have

Original: [Equation]
Revised: [Equation]

Original:  where the first inequality holds by the concavity of function f ( x ) = x 1 / 3 , i.e. , ( x + y ) 1 / 3 ≤ x 1 / 3 + y 3 x 2 / 3 ; the second inequality is due to m ≥ 2 , and the last inequality is due to 0 < η t ≤ 1 .
Revised: 78) where the first inequality holds by the concavity of function f ( x ) = x 1 / 3 , i.e. , ( x + y ) 1 / 3 ≤ x 1 / 3 + y 3 x 2 / 3 ; the second inequality is due to m ≥ 2 , and the last inequality is due to 0 < η t ≤ 1 .

Original: Let c 1 ≥ 23 k 3 + 125 L 20 6 µ 2 , we have
Revised: Let c 1 ≥ 23 k 3 + 125 L 20 6 µ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Let c 2 ≥ 23 k 3 + 92 , we have
Revised: Let c 2 ≥ 23 k 3 + 92 , we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemmas 7 and [Equation] we have
Revised: According to Lemmas 7 and 9, we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemma [Equation] we have
Revised: According to Lemma 10, we have

Original: [Equation]
Revised: [Equation]

Original: Next [Equation] we define a Lyapunov function, for any t ≥
Revised: Next, we define a Lyapunov function, for any t ≥Θ t

Original: [Equation]
Revised: = E (cid:2) F ( x t ) + 5 b t L 20 γ λµρ ∥ y t − y ∗ ( x t ) ∥ 2 + γ ρη t −(cid:0) ∥

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: t − ∇ y g

Original: [Equation]
Revised: ( x t , y t )

Original: [Table]
Revised: ∥ 2 + ∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ 2 (cid:1)(cid:3) .

Original: [Equation] [Table] [Equation] [Table]
Revised: For

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the first inequality holds by the above inequalities equation 79, equation 80, equation 81 and equation 82; the last inequality is due to 0 < γ ≤
Revised: where the first inequality holds by the above inequalities (79), (80), (81) and (82); the last inequality is due to 0 < γ ≤

Original: [Equation]
Revised: [Equation]

Original: for
Revised: all t ≥ 1 . Let Φ

Original: [Equation]
Revised: [Equation]

Original: Let Φ t =L 20 ∥
Revised: =

Original: y t − y ∗ ( x t ) ∥ 2 + 2 ∥
Revised: [Equation] ( x t ) ∥ 2 + 2 ∥

Original: w t − ¯ ∇ f ( x t , y t ) − R t ∥ 2 , then we have γηt
Revised: w t − ¯ ∇ f ( x t , y t ) − R t ∥ 2 , then we have γη

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Taking average over t = 1 , 2 , · · · , T on both sides of equation 84, we have
Revised: Taking average over t = 1 , 2 , · · · , T on both sides of (84), we have

Original: [Equation]
Revised: [Equation]

Original: Given x 1 ∈ X and y 1 ∈ Y , let ∆ 

Revised: Given x 1 ∈ X and y 1 ∈ Y , let ∆

Original: = ∥
Revised: = ∥

Original: y 1 − y ∗ ( x 1 ) ∥ 2 , we haveΘ 1 = [Equation] (cid:2) F ( x [Equation]
Revised: y 1 − y ∗ ( x 1 ) ∥ 2 , we have

Original: + 5 b

Original: [Equation]
Revised: [Equation]

Original: L 20 γ λµρ ∥ y

Original: [Equation]

Original: − y

Original: [Equation]

Original: ( x

Original: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]

Original: + γ ρη

Original: [Equation]

Original: 1 ∥

Original: [Equation]

Original: (cid:1)(cid:3) ≤

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds by Assumption 2.
Revised: where the last inequality holds by Assumption 2.

Original: Since η t is decreasing, i.e., η − 1 T ≥ η − 1 t for any 0 ≤ t ≤ T , we have
Revised: Since η t is decreasing, i.e., η − 1 T ≥ η − 1 t for any 0 ≤ t ≤ T , we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds by the above inequality equation 85.
Revised: where the second inequality holds by the above inequality (85).

Original: Let M = F ( x 1 ) − F ∗ ρkγ+
Revised: Let M = F ( x 1 ) − F ∗ ρkγ

Original: [Equation]
Revised: [Equation]

Revised: 2 m 1 / 3 σ

Revised: [Equation]

Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: According to Jensen’s inequality
Revised: According to Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to ( a + b ) 1 / 6 ≤ a 1 / 6 + b 1 / 6 for all a, b > 0 .
Revised: where the last inequality is due to ( a + b ) 1 / 6 ≤ a 1 / 6 + b 1 / 6 for all a, b > 0 .

Original: Thus we have
Revised: Thus we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since ∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ ≥ ∥ w t − ¯ ∇ f ( x t , y t )
Revised: Since ∥ w t − ¯ ∇ f ( x t , y t ) − R t ∥ ≥ ∥ w t − ¯ ∇ f ( x t , y t )

Original: ∥ − ∥ R t ∥
Revised: ∥ − ∥ R t ∥

Original: [Equation]
Revised: [Equation]

Original: by the above inequality equation
Revised: by the above inequality (89)

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to ∥ R t ∥ ≤ 1 T for all t ≥ 1 by choosing K
Revised: where the last inequality is due to ∥ R t ∥ ≤ 1 T for all t ≥ 1 by choosing K

Original: = L g µ log( C gxy C fy T/µ ) in Algorithm 2.
Revised: = L g µ log( C gxy C fy T/µ ) in Algorithm 2.

Original: According to the above inequality equation 11, we have
Revised: According to the above inequality (11), we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 8 (Restatement of Theorem 4) Under the above Assumptions (1, 3, 4, 6, 7), in the Algorithm 2, given X = R d , η t =[Equation]
Revised: Theorem 8. (Restatement of Theorem 4) Under the above Assumptions (1, 3, 4, 6, 7), in the Algorithm 2, given X = R d , η t =

Original: 3 for all t

Original: [Equation]
Revised: [Equation]

Original: , α

Original: [Equation]
Revised: [Equation]

Original: β [Equation] = c
Revised: 3 for all

Original: [Equation]
Revised: [Equation]

Original: η
Revised: max

Original: [Equation]
Revised: [Equation]

Original: t
Revised: k

Original: [Equation]
Revised: [Equation]

Original: m ≥ max (cid:0) 2 , k 3 ,
Revised: min (cid:16)

Original: [Equation]
Revised: [Equation]

Original: k ) [Equation] (cid:0)
Revised: Lk (cid:17) and K

Original: [Equation] (cid:1) , [Equation] (cid:17) and K =
Revised: =

Original: L g µ log( C gxy C fy T/µ ) , we have
Revised: L g µ log( C gxy C fy T/µ ) , we have

Original: [Equation]
Revised: [Equation]

Original: where M′ = ρ (
Revised: where M

Original: [Equation]
Revised: [Equation]

Revised: Proof.

Original: According to Lemma 7, we have
Revised: According to Lemma 7, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the equality ( i ) holds by ˜ x t
Revised: where the equality ( i ) holds by ˜ x t

Original: +1 = x t − γA − 1 t w t that can be easily obtained from the stepof Algorithm 2 when X = R d
Revised: +1 = x t − γA − 1 t w t that can be easily obtained from the stepof Algorithm 2 when X = R d

Original: [Equation]
Revised: [Equation]

Original: and the inequality ( ii ) holds by ∥
Revised: and the inequality ( ii ) holds by ∥

Original: A t ∥ ≥ ρ for all t ≥ 1 due to
Revised: A t ∥ ≥ ρ for all t ≥ 1 due to

Original: Assumption 7.
Revised: Assumption 7.

Original: Then we have
Revised: Then we have

Original: [Equation]

Original: F

Original: [Equation]
Revised: [Equation]

Original: According to Cauchy-Schwarz inequality
Revised: According to Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Revised: where the third inequality holds by the above inequality (87)

Original: [Equation]
Revised: [Equation]

Original: where the third inequality holds by the above inequality equation 87, and the last inequality holds by ∥ R t ∥ ≤ 1 T for all t ≥ 1 by choosing K
Revised: and the last inequality holds by ∥ R t ∥ ≤ 1 T for all t ≥ 1 by choosing K

Original: = L g µ log( C gxy C fy T/µ ) .
Revised: = L g µ log( C gxy C fy T/µ ) .

Original: By combining the above inequalities equation 95 and equation 96, we have
Revised: By combining the above inequalities (95) and (96) [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: where M ′ =
Revised: where M ′ =

Original: ρ 2 M .
Revised: ρ 2 M .