Original: ES-B ASED J ACOBIAN E NABLES F ASTER B ILEVEL O PTIMIZATION
Revised: ES-B ASED J ACOBIAN E NABLES F ASTER B ILEVEL O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Bilevel optimization (BO) has arisen as a powerful tool for solving many modern machine learning problems.
Revised: Bilevel optimization (BO) has arisen as a powerful tool for solving many modern machine learning problems.

Original: However, due to the nested structure of BO, existing gradient-based methods require second-order derivative approximations via
Revised: However, due to the nested structure of BO, existing gradient-based methods require second-order derivative approximations via

Original: Jacobian- or/and Hessian-vector computations, which can be very costly in practice, especially with large neural network models.
Revised: Jacobian- or/and Hessian-vector computations, which can be very costly in practice, especially with large neural network models.

Original: In this work, we propose a novel
Revised: In this work, we propose a novel

Original: BO algorithm, which adopts Evolution Strategies (ES) based method to approximate the response Jacobian matrix in the hypergradient of BO, and hence fully eliminates all second-order computations.
Revised: BO algorithm, which adopts Evolution Strategies (ES) based method to approximate the response Jacobian matrix in the hypergradient of BO, and hence fully eliminates all second-order computations.

Original: We call our algorithm as ESJ (which stands for the ES-based Jacobian method) and further extend it to the stochastic setting as ESJ-S. Theoretically, we characterize the convergence guarantee and computational complexity for our algorithms.
Revised: We call our algorithm as ESJ (which stands for the ES-based Jacobian method) and further extend it to the stochastic setting as ESJ-S. Theoretically, we characterize the convergence guarantee and computational complexity for our algorithms.

Original: Experimentally, we demonstrate the superiority of our proposed algorithms compared to the state of the art methods on various bilevel problems.
Revised: Experimentally, we demonstrate the superiority of our proposed algorithms compared to the state of the art methods on various bilevel problems.

Original: Particularly, in our experiment in the few-shot metalearning problem, we meta-learn the twelve millions parameters of a ResNetnetwork over the miniImageNet dataset, which evidently demonstrates the scalability of our ES-based bilevel approach and its feasibility in the large-scale setting.
Revised: Particularly, in our experiment in the few-shot metalearning problem, we meta-learn the twelve millions parameters of a ResNetnetwork over the miniImageNet dataset, which evidently demonstrates the scalability of our ES-based bilevel approach and its feasibility in the large-scale setting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Bilevel optimization has recently arisen as a powerful tool to capture various modern machine learning problems, including meta-learning (Bertinetto et al., 2018; Franceschi et al., 2018; Rajeswaran et al., 2019; Ji et al., 2020a; Liu et al., 2021a), hyperparamater optimization (Franceschi et al., 2018;
Revised: Bilevel optimization has recently arisen as a powerful tool to capture various modern machine learning problems, including meta-learning (Bertinetto et al., 2018; Franceschi et al., 2018; Rajeswaran et al., 2019; Ji et al., 2020a; Liu et al., 2021a), hyperparamater optimization (Franceschi et al., 2018;

Original: Shaban et al., 2019), neural architecture search (Liu et al., 2018a; Zhang et al., 2021), etc.
Revised: Shaban et al., 2019), neural architecture search (Liu et al., 2018a; Zhang et al., 2021), etc.

Original: Bilevel optimization generally takes the following mathematical form:
Revised: Bilevel optimization generally takes the following mathematical form:

Original: [Equation]
Revised: [Equation]

Original: where the outer and inner objectives f : R p × R d
Revised: where the outer and inner objectives f : R p × R d

Original: →
Revised: →

Original: R
Revised: R

Original: and g :
Revised: and g :

Original: R p × R
Revised: R p × R

Original: d
Revised: d

Original: → R are both continuously differentiable with respect to (w.r.t.)
Revised: → R are both continuously differentiable with respect to (w.r.t.)

Original: the inner and outer variables x ∈ R p and y ∈ R d .
Revised: the inner and outer variables x ∈ R p and y ∈ R d .

Original: In many machine learning problems, the loss functions f and g take a ﬁnite-sum form over a set of given data
Revised: In many machine learning problems, the loss functions f and g take a ﬁnite-sum form over a set of given data

Original: D n,m = { ξ i , ζ j , i = 1 , ..., n, j = 1 , ..., m } as follows:
Revised: D n,m = { ξ i , ζ j , i = 1 , ..., n, j = 1 , ..., m } as follows:

Original: [Equation]
Revised: [Equation]

Original: where the sample sizes n and m are typically very large.
Revised: where the sample sizes n and m are typically very large.

Original: In this work, we consider a popular setting where the total objective function Φ( x ) in eq.
Revised: In this work, we consider a popular setting where the total objective function Φ( x ) in eq.

Original: (1) is possibly nonconvex w.r.t.
Revised: (1) is possibly nonconvex w.r.t.

Original: x and the inner function g is strongly convex w.r.t.
Revised: x and the inner function g is strongly convex w.r.t.

Original: y .
Revised: y .

Original: Gradient-based methods have served as a popular tool for solving bilevel optimization problems.
Revised: Gradient-based methods have served as a popular tool for solving bilevel optimization problems.

Original: Two types of approaches have been widely used: the iterative differentiation (ITD) method (Domke, 2012; Maclaurin et al., 2015; Finn et al., 2017; Franceschi et al., 2017; Shaban et al., 2019; Rajeswaran et al., 2019; Liu et al., 2020a) and the approximate iterative differentiation (AID) method (Domke, 2012; Pedregosa, 2016; Gould et al., 2016;
Revised: Two types of approaches have been widely used: the iterative differentiation (ITD) method (Domke, 2012; Maclaurin et al., 2015; Finn et al., 2017; Franceschi et al., 2017; Shaban et al., 2019; Rajeswaran et al., 2019; Liu et al., 2020a) and the approximate iterative differentiation (AID) method (Domke, 2012; Pedregosa, 2016; Gould et al., 2016;

Original: Liao et al., 2018; Lorraine et al., 2020).
Revised: Liao et al., 2018; Lorraine et al., 2020).

Original: In the ITD method, the objective value f ( x, ˆ y ( x )) is ﬁrst computed using an approximation point ˆ y ( x ) close to y ∗ ( x ) , and the hypergradient ∇ Φ( x ) =
Revised: In the ITD method, the objective value f ( x, ˆ y ( x )) is ﬁrst computed using an approximation point ˆ y ( x ) close to y ∗ ( x ) , and the hypergradient ∇ Φ( x ) =

Original: ∂f ( x,y ∗ ( x )) ∂x is then approximated by ∂f ( x, ˆ y ( x ))
Revised: ∂f ( x,y ∗ ( x )) ∂x is then approximated by ∂f ( x, ˆ y ( x ))

Original: ∂x using either reverse or forward mode automatic differentiation.
Revised: ∂x using either reverse or forward mode automatic differentiation.

Original: Alternatively, the AID method leverages the Implicit Function Theorem (IFT) to establish an implicit expression for the hypergradient.
Revised: Alternatively, the AID method leverages the Implicit Function Theorem (IFT) to establish an implicit expression for the hypergradient.

Original: Due to the bilevel structure of the problem, these gradient-based ITD and AID approaches typically involve second-order matrix computations.
Revised: Due to the bilevel structure of the problem, these gradient-based ITD and AID approaches typically involve second-order matrix computations.

Original: Existing efﬁcient implementations of these methods often adopt Jacobian- or/and Hessian-vector computations, which can still be very costly in practice for high-dimensional problems with deep neural networks.
Revised: Existing efﬁcient implementations of these methods often adopt Jacobian- or/and Hessian-vector computations, which can still be very costly in practice for high-dimensional problems with deep neural networks.

Original: To overcome the computational challenge of current gradient-based methods, Evolution Strategies (ES) can be a good candidate to eliminate the second-order computations, which uses function values for estimating its gradient in blackbox optimization (Nesterov & Spokoiny, 2017).
Revised: To overcome the computational challenge of current gradient-based methods, Evolution Strategies (ES) can be a good candidate to eliminate the second-order computations, which uses function values for estimating its gradient in blackbox optimization (Nesterov & Spokoiny, 2017).

Original: An ES-based bilevel optimizer has recently been proposed in Gu et al.
Revised: An ES-based bilevel optimizer has recently been proposed in Gu et al.

Original: (2021) for hyperparameter optimization, which uses the ES method to approximate the hypergradient of Φ( x ) , treating Φ( x ) fully as a blackbox objective.
Revised: (2021) for hyperparameter optimization, which uses the ES method to approximate the hypergradient of Φ( x ) , treating Φ( x ) fully as a blackbox objective.

Original: However, the hypergradient in bilevel optimization has a much more involved structure (see eq.
Revised: However, the hypergradient in bilevel optimization has a much more involved structure (see eq.

Original: (3)) than the gradient in standard minimization problems, and hence direct use of the
Revised: (3)) than the gradient in standard minimization problems, and hence direct use of the

Original: ES method without exploiting the structure of the hypergradient can encounter a large estimation bias and result in poor performance in practice, especially in deep learning as demonstrated in our experiments in Section 4.
Revised: ES method without exploiting the structure of the hypergradient can encounter a large estimation bias and result in poor performance in practice, especially in deep learning as demonstrated in our experiments in Section 4.

Original: This hence motivates the following intriguing question:
Revised: This hence motivates the following intriguing question:

Original: Can we design a better hypergradient estimator based on the ES method by leveraging the analytical structure of the hypergradient, and then use such an estimator to construct more efﬁcient ES-based bilevel optimizers, which are scalable for high-dimensional problems?
Revised: Can we design a better hypergradient estimator based on the ES method by leveraging the analytical structure of the hypergradient, and then use such an estimator to construct more efﬁcient ES-based bilevel optimizers, which are scalable for high-dimensional problems?

Original: Furthermore, since the hypergradient in bilevel optimization is much more challenging to estimate than the gradient in standard minimization problems, it is unclear whether the ES-based estimator can yield a provably convergent algorithm for bilevel optimization.
Revised: Furthermore, since the hypergradient in bilevel optimization is much more challenging to estimate than the gradient in standard minimization problems, it is unclear whether the ES-based estimator can yield a provably convergent algorithm for bilevel optimization.

Original: So far the ES-based bilevel optimizer in Gu et al.
Revised: So far the ES-based bilevel optimizer in Gu et al.

Original: (2021) did not come with a polynomial-time complexity guarantee.
Revised: (2021) did not come with a polynomial-time complexity guarantee.

Original: Our empirical experiments in Section 4 show that such an algorithm fail to converge even with shallow neural networks.
Revised: Our empirical experiments in Section 4 show that such an algorithm fail to converge even with shallow neural networks.

Original: This thus motivates the second question we ask:
Revised: This thus motivates the second question we ask:

Original: Can the ES method lead to bilevel optimizers with theoretical ﬁnite-time convergence guarantee?
Revised: Can the ES method lead to bilevel optimizers with theoretical ﬁnite-time convergence guarantee?

Original: This paper provides the afﬁrmative answers to both questions.
Revised: This paper provides the afﬁrmative answers to both questions.

Original: 1. M AIN C ONTRIBUTIONS
Revised: 1. M AIN C ONTRIBUTIONS

Original: Novel ES-based Jacobian bilevel optimizer.
Revised: Novel ES-based Jacobian bilevel optimizer.

Original: We propose a novel bilevel optimizer with the ESbased J acobian estimation, which we call as ESJ.
Revised: We propose a novel bilevel optimizer with the ESbased J acobian estimation, which we call as ESJ.

Original: In contrast to the existing ES-based optimizer in
Revised: In contrast to the existing ES-based optimizer in

Original: Gu et al.
Revised: Gu et al.

Original: (2021), which estimates the hypergradient by treating the outer-level objective Φ( x ) fully as a blackbox, ESJ estimates only the respone Jacobian matrix (i.e., gradient of y ∗ ( x ) with respect to x ), which is the major computational bottleneck, and then leverages the analytical structure of the hypergradient to construct a much more accurate estimator.
Revised: (2021), which estimates the hypergradient by treating the outer-level objective Φ( x ) fully as a blackbox, ESJ estimates only the respone Jacobian matrix (i.e., gradient of y ∗ ( x ) with respect to x ), which is the major computational bottleneck, and then leverages the analytical structure of the hypergradient to construct a much more accurate estimator.

Original: Further, ESJ has only gradient computations, and is computationally much more efﬁcient than existing AID and ITD bilevel optimizers that require Hessian- and/or Jacobian-vector product computations.
Revised: Further, ESJ has only gradient computations, and is computationally much more efﬁcient than existing AID and ITD bilevel optimizers that require Hessian- and/or Jacobian-vector product computations.

Original: We further propose a stochastic bilevel optimizer ESJ-S to allow the algorithm scalable over large dataset.
Revised: We further propose a stochastic bilevel optimizer ESJ-S to allow the algorithm scalable over large dataset.

Original: Convergence guarantee.
Revised: Convergence guarantee.

Original: Theoretically, we characterize the convergence rate and their computational complexity for both ESJ and ESJ-S to achieve an (cid:15) -accurate solution.
Revised: Theoretically, we characterize the convergence rate and their computational complexity for both ESJ and ESJ-S to achieve an (cid:15) -accurate solution.

Original: Technically, in contrast to the standard analysis of ES-based methods on the smoothed blackbox function values, we develop tools to analyze the ES estimator on the output of the execution of an inner optimizer.
Revised: Technically, in contrast to the standard analysis of ES-based methods on the smoothed blackbox function values, we develop tools to analyze the ES estimator on the output of the execution of an inner optimizer.

Original: Such an analysis can be of independent interest for ES-based bilevel optimization.
Revised: Such an analysis can be of independent interest for ES-based bilevel optimization.

Original: Superior performance.
Revised: Superior performance.

Original: Experimentally, our algorithms ESJ and ESJ-S achieve superior performance compared to the current state of the art bilevel optimizers.
Revised: Experimentally, our algorithms ESJ and ESJ-S achieve superior performance compared to the current state of the art bilevel optimizers.

Original: Our approach is not only efﬁcient and scalable, but also stable and robust across various bilevel problems.
Revised: Our approach is not only efﬁcient and scalable, but also stable and robust across various bilevel problems.

Original: In particular, on the few-shot meta-learning problem, we meta-learn the twelve millions weights of a ResNet-12 network.
Revised: In particular, on the few-shot meta-learning problem, we meta-learn the twelve millions weights of a ResNet-12 network.

Original: Previous applications of other bilevel algorithms to meta-learning were limited only to settings where the backbone network has only up to few hundreds of thousands of parameters.
Revised: Previous applications of other bilevel algorithms to meta-learning were limited only to settings where the backbone network has only up to few hundreds of thousands of parameters.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: Bilevel optimization Bilevel optimization has been studied for decades since Bracken & McGill (1973).
Revised: Bilevel optimization. Bilevel optimization has been studied for decades since Bracken & McGill (1973).

Original: A variety of bilevel optimization algorithms were then developed, including constraintbased approaches (Hansen et al., 1992; Shi et al., 2005; Moore, 2010), approximate implicit differ- entiation (AID) approaches (Domke, 2012; Pedregosa, 2016; Gould et al., 2016;
Revised: A variety of bilevel optimization algorithms were then developed, including constraintbased approaches (Hansen et al., 1992; Shi et al., 2005; Moore, 2010), approximate implicit differ- entiation (AID) approaches (Domke, 2012; Pedregosa, 2016; Gould et al., 2016;

Original: Liao et al., 2018;
Revised: Liao et al., 2018;

Original: Lorraine et al., 2020) and iterative differentiation (ITD) approaches (Domke, 2012; Maclaurin et al., 2015; Franceschi et al., 2017;
Revised: Lorraine et al., 2020) and iterative differentiation (ITD) approaches (Domke, 2012; Maclaurin et al., 2015; Franceschi et al., 2017;

Original: Finn et al., 2017; Shaban et al., 2019; Rajeswaran et al., 2019; Liu et al., 2020a).
Revised: Finn et al., 2017; Shaban et al., 2019; Rajeswaran et al., 2019; Liu et al., 2020a).

Original: These methods often suffer from expensive computations of second-order information (e.g., Hessian-vector products).
Revised: These methods often suffer from expensive computations of second-order information (e.g., Hessian-vector products).

Original: Hessian-free algorithms were recently proposed based on interiorpoint method (Liu et al., 2021b) and Evolution Strategies (ES) (Gu et al., 2021).
Revised: Hessian-free algorithms were recently proposed based on interiorpoint method (Liu et al., 2021b) and Evolution Strategies (ES) (Gu et al., 2021).

Original: This paper proposes a more efﬁcient ES-based approach via exploiting the benign structure of the hypergradient.
Revised: This paper proposes a more efﬁcient ES-based approach via exploiting the benign structure of the hypergradient.

Original: Recently, the convergence rate has been established for gradient-based (Hessian involved) bilevel algorithms (Grazzi et al., 2020; Ji et al., 2021; Rajeswaran et al., 2019; Ji & Liang, 2021).
Revised: Recently, the convergence rate has been established for gradient-based (Hessian involved) bilevel algorithms (Grazzi et al., 2020; Ji et al., 2021; Rajeswaran et al., 2019; Ji & Liang, 2021).

Original: This paper provides the convergence analysis for the proposed Hessian-free ES-based approach.
Revised: This paper provides the convergence analysis for the proposed Hessian-free ES-based approach.

Original: Stochastic bilevel optimization.
Revised: Stochastic bilevel optimization.

Original: Ghadimi & Wang (2018); Ji et al. (2021); Hong et al.
Revised: Ghadimi & Wang (2018); Ji et al. (2021); Hong et al.

Original: (2020) proposed stochastic gradient descent (SGD) type bilevel optimization algorithms by employing Neumann series for Hessian-inverse-vector approximation.
Revised: (2020) proposed stochastic gradient descent (SGD) type bilevel optimization algorithms by employing Neumann series for Hessian-inverse-vector approximation.

Original: Recent works (Khanduri et al., 2021a;b;
Revised: Recent works (Khanduri et al., 2021a;b;

Original: Chen et al., 2021; Guo et al., 2021; Guo & Yang, 2021;
Revised: Chen et al., 2021; Guo et al., 2021; Guo & Yang, 2021;

Original: Yang et al., 2021) then leveraged momentumbased variance reduction to further reduce the computational complexity of existing SGD-type bilevel optimizers.
Revised: Yang et al., 2021) then leveraged momentumbased variance reduction to further reduce the computational complexity of existing SGD-type bilevel optimizers.

Original: In this paper, we propose a stochastic ES-based method, which eliminate the computation of second-order information required by all aforementioned stochastic methods.
Revised: In this paper, we propose a stochastic ES-based method, which eliminate the computation of second-order information required by all aforementioned stochastic methods.

Original: Please see Appendix B for more works about applications of bilevel optimization and ES methods.
Revised: Please see Appendix B for more works about applications of bilevel optimization and ES methods.

Original:  P ROPOSED A LGORITHMS
Revised:  P ROPOSED A LGORITHMS

Original: We describe our proposed deterministic and stochastic algorithms for bilevel optimization.
Revised: We describe our proposed deterministic and stochastic algorithms for bilevel optimization.

Original: 2. H YPERGRADIENTS
Revised: 2. H YPERGRADIENTS

Original: The key step in popular gradient-based bilevel optimizers is the estimation of the hypergradient (i.e., the gradient of the objective with respect to the outer variable x ), which takes the following form:
Revised: The key step in popular gradient-based bilevel optimizers is the estimation of the hypergradient (i.e., the gradient of the objective with respect to the outer variable x ), which takes the following form:

Original: [Equation]
Revised: [Equation]

Original: where the Jacobian matrix J ∗ ( x ) =
Revised: where the Jacobian matrix J ∗ ( x ) =

Original: ∂y ∗ ( x ) ∂x ∈ R d × p .
Revised: ∂y ∗ ( x ) ∂x ∈ R d × p .

Original: Following Lorraine et al.
Revised: Following Lorraine et al.

Original: (2020), it can be seen that ∇ Φ( x ) contains two components: the direct gradient ∇ x f ( x, y ∗ ( x )) and the indirect gradient J ∗ ( x ) (cid:62) ∇ y f
Revised: (2020), it can be seen that ∇ Φ( x ) contains two components: the direct gradient ∇ x f ( x, y ∗ ( x )) and the indirect gradient J ∗ ( x ) (cid:62) ∇ y f

Original: ( x, y ∗ ( x )) .
Revised: ( x, y ∗ ( x )) .

Original: The direct component can be efﬁciently computed using the existing automatic differentiation techniques.
Revised: The direct component can be efﬁciently computed using the existing automatic differentiation techniques.

Original: The indirect component, however, is computationally much more complex, because J ∗ ( x ) takes the form of J ∗ ( x ) = − (cid:2) ∇ 2 y g ( x, y ∗ ( x )) (cid:3) − 1 ∇ x ∇ y g
Revised: The indirect component, however, is computationally much more complex, because J ∗ ( x ) takes the form of J ∗ ( x ) = − (cid:2) ∇ 2 y g ( x, y ∗ ( x )) (cid:3) − 1 ∇ x ∇ y g

Original: ( x, y ∗ ( x ))
Revised: ( x, y ∗ ( x ))

Original: (if ∇ 2 y g ( x, y ∗ ( x )) is invertible), which contains the Hessian inverse and the second-order mixed derivative.
Revised: (if ∇ 2 y g ( x, y ∗ ( x )) is invertible), which contains the Hessian inverse and the second-order mixed derivative.

Original: Some approaches mitigate the issue by designing Jacobian-vector and Hessian-vector products (Pedregosa, 2016; Franceschi et al., 2017; Grazzi et al., 2020) to replace second-order computations.
Revised: Some approaches mitigate the issue by designing Jacobian-vector and Hessian-vector products (Pedregosa, 2016; Franceschi et al., 2017; Grazzi et al., 2020) to replace second-order computations.

Original: But these algorithms still have poor scalability to modern large-scale bilevel problems with high-dimensional variables such as neural network parameters.
Revised: But these algorithms still have poor scalability to modern large-scale bilevel problems with high-dimensional variables such as neural network parameters.

Original: We next introduce the ES approach which is at the core of our idea for designing scalable Hessian-free bilevel optimizers.
Revised: We next introduce the ES approach which is at the core of our idea for designing scalable Hessian-free bilevel optimizers.

Original: 2. E VOLUTION S TRATEGIES (ES) M ETHOD
Revised: 2. E VOLUTION S TRATEGIES (ES) M ETHOD

Original: Evolution Strategies (ES) is a powerful technique to estimate the gradient of a function based on function values, when it is not feasible (such as in black-box problems) or computationally costly to evaluate the gradient.
Revised: Evolution Strategies (ES) is a powerful technique to estimate the gradient of a function based on function values, when it is not feasible (such as in black-box problems) or computationally costly to evaluate the gradient.

Original: The idea of the ES method in Nesterov & Spokoiny (2017) is to approximate the gradient of a general black-box function h : R n → R using the following oracle based only on the function values (i.e., the zeroth-order information)
Revised: The idea of the ES method in Nesterov & Spokoiny (2017) is to approximate the gradient of a general black-box function h : R n → R using the following oracle based only on the function values (i.e., the zeroth-order information)

Original: [Equation]
Revised: [Equation]

Original: where u ∈ R n is a Gaussian random vector and µ > 0 is the smoothing parameter.
Revised: where u ∈ R n is a Gaussian random vector and µ > 0 is the smoothing parameter.

Original: Such an oracle can be shown to be an unbiased estimator of the gradient of the smoothed function E u
Revised: Such an oracle can be shown to be an unbiased estimator of the gradient of the smoothed function E u

Original: [ h ( x + µu )] .
Revised: [ h ( x + µu )] .

Original: 2. ES- BASED J ACOBIAN AND ITS E NABLED B ILEVEL O PTIMIZERS
Revised: 2. ES- BASED J ACOBIAN AND ITS E NABLED B ILEVEL O PTIMIZERS

Original: Drawback of an existing ES-based bilevel optimizer.
Revised: Drawback of an existing ES-based bilevel optimizer.

Original: In Gu et al.
Revised: In Gu et al.

Original: (2021), the ES method was directly applied to estimate the hypergradient of Φ( x ) , treating Φ( x ) as a blackbox objective.
Revised: (2021), the ES method was directly applied to estimate the hypergradient of Φ( x ) , treating Φ( x ) as a blackbox objective.

Original: Since α j

Original: − 1 k j − α ∇ y g x k + µu k j y t − 1 k j

Original: Compute δ j µ 13:

Original: Compute (cid:98) ∇ Φ( x k ) =

Original: ∇ x f ( x k y Nk ) + 1 Q (cid:80)

Original: Qj =1 (cid:10)

Original: δ j

Original: ∇ y f

Original: ( x k y Nk ) (cid:11)

Original: u k j 15:

Original: Update x 1 = x − β ) 16: end for the hypergradient has a complex form as in eq.
Revised: Since the hypergradient has a complex form as in eq.

Original: (3) and can be very sensitive to both inner and outer functions, such an estimation will likely have a large bias error.
Revised: (3) and can be very sensitive to both inner and outer functions, such an estimation will likely have a large bias error.

Original: As our experiments in Sectiondemonstrate, such a method is not robust and even fails to converge in complex bilevel optimization problems (such as with neural network parameters).
Revised: As our experiments in Sectiondemonstrate, such a method is not robust and even fails to converge in complex bilevel optimization problems (such as with neural network parameters).

Original: Our key idea is to exploit the analytical structure of the hypergradient in eq.
Revised: Our key idea is to exploit the analytical structure of the hypergradient in eq.

Original: (3), where the derivatives ∇ x f ( x, y ∗ ( x )) and ∇ y f ( x, y ∗ ( x )) can be computed efﬁciently and accurately, and then use the ES method only to estimate the Jacobian J ∗ ( x ) , which is the major term posing computational difﬁculty.
Revised: (3), where the derivatives ∇ x f ( x, y ∗ ( x )) and ∇ y f ( x, y ∗ ( x )) can be computed efﬁciently and accurately, and then use the ES method only to estimate the Jacobian J ∗ ( x ) , which is the major term posing computational difﬁculty.

Original: In this way, our estimation of the hypergradient can be much more accurate and reliable.
Revised: In this way, our estimation of the hypergradient can be much more accurate and reliable.

Original: We propose the following novel ES-based Jacobian estimator, which contains two ingredients: (i) for a given x , apply an algorithm to solve the inner optimization problem and use the output as an approximation of y ∗ ( x ) ; for example, the output y N ( x ) of N gradient descent steps of the inner problem can serve as an estimate for y ∗ ( x ) .
Revised: We propose the following novel ES-based Jacobian estimator, which contains two ingredients: (i) for a given x , apply an algorithm to solve the inner optimization problem and use the output as an approximation of y ∗ ( x ) ; for example, the output y N ( x ) of N gradient descent steps of the inner problem can serve as an estimate for y ∗ ( x ) .

Original: Then J N ( x ) =
Revised: Then J N ( x ) =

Original: ∂y N ( x ) ∂x serves as an estimate of J ∗ ( x ) ; and (ii) construct an ES-based Jacobian estimator ˆ J N ( x ; u
Revised: ∂y N ( x ) ∂x serves as an estimate of J ∗ ( x ) ; and (ii) construct an ES-based Jacobian estimator ˆ J N ( x ; u

Original: ) ∈ R
Revised: ) ∈ R

Original: d × p for J N ( x ) as
Revised: d × p for J N ( x ) as

Original: [Equation]
Revised: [Equation]

Original: where u ∈ R p is a Gaussian vector with independent and identically distributed (i.i.d.) entries.
Revised: where u ∈ R p is a Gaussian vector with independent and identically distributed (i.i.d.) entries.

Original: Then for any vector v ∈ R d , the Jacobian-vector product can be efﬁciently computed using only vector-vector dot product ˆ J N ( x ; u ) (cid:62) v = (cid:104) δ ( x ; u ) , v (cid:105) u , where δ ( x ; u ) =
Revised: Then for any vector v ∈ R d , the Jacobian-vector product can be efﬁciently computed using only vector-vector dot product ˆ J N ( x ; u ) (cid:62) v = (cid:104) δ ( x ; u ) , v (cid:105) u , where δ ( x ; u ) =

Original: y N ( x + µu ) − y N ( x ) µ ∈ R d .
Revised: y N ( x + µu ) − y N ( x ) µ ∈ R d .

Original: ESJ:
Revised: ESJ:

Original: A novel bilevel optimizer with ES Jacobian oracles.
Revised: A novel bilevel optimizer with ES Jacobian oracles.

Original: We design a bilevel optimizer (see
Revised: We design a bilevel optimizer (see

Original: Algorithm 1) using the ES-based Jacobian oracle in eq.
Revised: Algorithm 1) using the ES-based Jacobian oracle in eq.

Original: (5), which we call as the ESJ algorithm.
Revised: (5), which we call as the ESJ algorithm.

Original: At each step k of the algorithm, ESJ runs an N -step full GD to approximate y Nk ( x k ) .
Revised: At each step k of the algorithm, ESJ runs an N -step full GD to approximate y Nk ( x k ) .

Original: ESJ then samples
Revised: ESJ then samples

Original: Q Gaussian vectors { u k,j ∈ N (0 , I ) , j = 1 , ..., Q } , and for each sample u k,j , runs an N -step full GD to approximate y Nk ( x k + µu k,j ) , and then computes the Jacobian estimator ˆ J N ( x ; u k,j ) as in eq.
Revised: Q Gaussian vectors { u k,j ∈ N (0 , I ) , j = 1 , ..., Q } , and for each sample u k,j , runs an N -step full GD to approximate y Nk ( x k + µu k,j ) , and then computes the Jacobian estimator ˆ J N ( x ; u k,j ) as in eq.

Original: Then the sample average over the Q estimators is used for constructing the following hypergradient estimator for updating the outer variable x . 
Revised: Then the sample average over the Q estimators is used for constructing the following hypergradient estimator for updating the outer variable x . [Equation]

Original: [Equation]
Revised: [Equation]

Original: Computationally, in contrast to the existing AID and ITD based bilevel optimizers (Pedregosa (2016), Franceschi et al.
Revised: Computationally, in contrast to the existing AID and ITD based bilevel optimizers (Pedregosa (2016), Franceschi et al.

Original: (2018), Grazzi et al.
Revised: (2018), Grazzi et al.

Original: that contains the complex Hessian- and/or
Revised: that contains the complex Hessian- and/or

Original: Jacobian-vector product computations, ESJ has only gradient computations, and hence is computationally much more efﬁcient as demonstrated in our experiments.
Revised: Jacobian-vector product computations, ESJ has only gradient computations, and hence is computationally much more efﬁcient as demonstrated in our experiments.

Original: ESJ-S:
Revised: ESJ-S:

Original: A stochastic bilevel optimizer with ES Jacobian oracles.
Revised: A stochastic bilevel optimizer with ES Jacobian oracles.

Original: For the ﬁnite-sum problem with the objective functions given in eq.
Revised: For the ﬁnite-sum problem with the objective functions given in eq.

Original: (2), we design a stochastic bilevel optimizer (see Algorithmin Appendix A) based on ES Jacobian oracles, which we call as ESJ-S .
Revised: (2), we design a stochastic bilevel optimizer (see Algorithmin Appendix A) based on ES Jacobian oracles, which we call as ESJ-S .

Original: Differently from Algorithm 1, which applies GD updates to ﬁnd y N ( x k ) , ESJ-S adopts N stochastic gradient descent (SGD) steps to ﬁnd { Y Nk , Y Nk, 1 , ..., Y Nk,Q } to the inner problem, each with the outer variable set to be x k + µu k,j .
Revised: Differently from Algorithm 1, which applies GD updates to ﬁnd y N ( x k ) , ESJ-S adopts N stochastic gradient descent (SGD) steps to ﬁnd { Y Nk , Y Nk, 1 , ..., Y Nk,Q } to the inner problem, each with the outer variable set to be x k + µu k,j .

Original: Note that all SGD runs follow the same batch sampling path {S 0 , ..., S N − 1 } .
Revised: Note that all SGD runs follow the same batch sampling path {S 0 , ..., S N − 1 } .

Original: The Jacobian estimator ˆ J N ( x k ; u k,j ) can then be computed as in eq.
Revised: The Jacobian estimator ˆ J N ( x k ; u k,j ) can then be computed as in eq.

Original: At the outer level, ESJ-S samples a new batch D F independently from the inner batches {S 0 , ..., S N − 1 } to evaluate the stochastic gradients ∇ x F ( x k , Y Nk ; D F ) and ∇ y F ( x k , Y Nk ; D F ) .
Revised: At the outer level, ESJ-S samples a new batch D F independently from the inner batches {S 0 , ..., S N − 1 } to evaluate the stochastic gradients ∇ x F ( x k , Y Nk ; D F ) and ∇ y F ( x k , Y Nk ; D F ) .

Original: The hypergradient is then estimated as follows.
Revised: The hypergradient is then estimated as follows.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  C ONVERGENCE A NALYSIS 3. T ECHNICAL A SSUMPTIONS
Revised:  C ONVERGENCE A NALYSIS 3. T ECHNICAL A SSUMPTIONS

Original: In this work, we focus on the following types of loss functions.
Revised: In this work, we focus on the following types of loss functions.

Original: Assumption 1.
Revised: Assumption 1.

Original: The inner function g ( x, y ) is µ g -strongly convex with respect to the variable y .
Revised: The inner function g ( x, y ) is µ g -strongly convex with respect to the variable y .

Original: For the ﬁnite-sum case, the same assumption holds for ∇ G ( x, y ; ζ ) . 
Revised: For the ﬁnite-sum case, the same assumption holds for ∇ G ( x, y ; ζ ) . [Table] [Table] [Table]

Revised: [Table]

Original: We take the following assumptions on the inner and outer loss functions g ( x, y ) and f ( x, y ) , as also adopted in Ghadimi & Wang (2018); Ji et al.
Revised: [Table] [Table] [Table] [Table]

Revised: +

Revised: [Table]

Revised: (cid:80) Qj =

Revised: [Table]

Revised: δ j

Revised: [Table]

Revised: f

Revised: [Table]

Revised: k y

Revised: [Table]

Revised: We take the following assumptions on the inner and outer loss functions g ( x, y ) and f ( x, y ) , as also adopted in Ghadimi & Wang (2018); Ji et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: Assumption 2.
Revised: Assumption 2.

Original: Let w
Revised: Let w

Original: =
Revised: =

Original: ( x, y ) ∈ R d + p .
Revised: ( x, y ) ∈ R d + p .

Original: The gradient ∇ g ( w ) is
Revised: The gradient ∇ g ( w ) is

Original: L g -Lipschitz continuous, i.e., for any w 1 , w 2 ∈ R d + p , (cid:13)(cid:13) ∇ g ( w 1 ) − ∇ g ( w 2 ) (cid:13)(cid:13) ≤
Revised: L g -Lipschitz continuous, i.e., for any w 1 , w 2 ∈ R d + p , (cid:13)(cid:13) ∇ g ( w 1 ) − ∇ g ( w 2 ) (cid:13)(cid:13) ≤

Original: L g (cid:13)(cid:13) w 1 − w 2 (cid:13)(cid:13) ; further, the derivatives ∇ 2 y g ( w ) and ∇ x ∇ y g ( w ) are ρ - and τ -Lipschitz continuous, i.e, (cid:13)(cid:13) ∇ 2 y g ( w 1 ) − ∇ 2 y g ( w 2 ) (cid:13)(cid:13) F ≤ ρ (cid:13)(cid:13) w 1 − w 2 (cid:13)(cid:13) and (cid:13)(cid:13) ∇ x ∇ y g ( w 1 ) − ∇ x ∇ y g ( w 2 ) (cid:13)(cid:13) F ≤ τ (cid:13)(cid:13) w 1 − w 2 (cid:13)(cid:13) .
Revised: L g (cid:13)(cid:13) w 1 − w 2 (cid:13)(cid:13) ; further, the derivatives ∇ 2 y g ( w ) and ∇ x ∇ y g ( w ) are ρ - and τ -Lipschitz continuous, i.e, (cid:13)(cid:13) ∇ 2 y g ( w 1 ) − ∇ 2 y g ( w 2 ) (cid:13)(cid:13) F ≤ ρ (cid:13)(cid:13) w 1 − w 2 (cid:13)(cid:13) and (cid:13)(cid:13) ∇ x ∇ y g ( w 1 ) − ∇ x ∇ y g ( w 2 ) (cid:13)(cid:13) F ≤ τ (cid:13)(cid:13) w 1 − w 2 (cid:13)(cid:13) .

Original: For the ﬁnite-sum case, the same assumptions hold for G ( w ; ζ ) .
Revised: For the ﬁnite-sum case, the same assumptions hold for G ( w ; ζ ) .

Original: Assumption 3.
Revised: Assumption 3.

Original: Let w
Revised: Let w

Original: =
Revised: =

Original: ( x, y ) ∈ R d + p .
Revised: ( x, y ) ∈ R d + p .

Original: The objective f ( w ) and its gradient ∇ f ( w ) are M - and
Revised: The objective f ( w ) and its gradient ∇ f ( w ) are M - and

Original: L f -Lipschitz continuous, i.e., for any w 1 , w 2 ∈ R d + p , 
Revised: L f -Lipschitz continuous, i.e., for any w 1 , w 2 ∈ R d + p , [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the ﬁnite-sum case, the same assumptions hold for F ( w ; ξ ) .
Revised: For the ﬁnite-sum case, the same assumptions hold for F ( w ; ξ ) .

Original: For the ﬁnite-sum setting, we take the standard bounded-variance assumption on ∇ G ( w ; ζ ) .
Revised: For the ﬁnite-sum setting, we take the standard bounded-variance assumption on ∇ G ( w ; ζ ) .

Original: Assumption 4.
Revised: Assumption 4.

Original: Gradient ∇ G ( w ; ζ ) has a bounded variance, i.e., E ζ (cid:107)∇ G ( w ; ζ ) − ∇ g ( w ) (cid:107) 2 ≤ σfor some constant σ ≥ 0 .
Revised: Gradient ∇ G ( w ; ζ ) has a bounded variance, i.e., E ζ (cid:107)∇ G ( w ; ζ ) − ∇ g ( w ) (cid:107) 2 ≤ σfor some constant σ ≥ 0 .

Original: 3. C ONVERGENCE A NALYSIS FOR ESJ
Revised: 3. C ONVERGENCE A NALYSIS FOR ESJ

Original: Differently from the standard zeroth-order analysis for a blackbox function, here we develop new techniques to analyze the ES-based estimator that depend on the entire inner optimization trajectory, which is speciﬁc to bilevel optimization.
Revised: Different from the standard zeroth-order analysis for a blackbox function, here we develop new techniques to analyze the ES-based estimator that depend on the entire inner optimization trajectory, which is speciﬁc to bilevel optimization.

Original: We ﬁrst establish the following essential proposition which characterizes the Lipshitzness property of the approximate Jacobian matrix J N ( x ) =
Revised: We ﬁrst establish the following essential proposition which characterizes the Lipshitzness property of the approximate Jacobian matrix J N ( x ) =

Original: ∂y N ( x ) ∂x .
Revised: ∂y N ( x ) ∂x .

Original: Proposition 1.
Revised: Proposition 1.

Original: Suppose that Assumptions 1 and 2 hold.
Revised: Suppose that Assumptions 1 and 2 hold.

Original: Let L J = (cid:16) 

Revised: Let L J = (cid:16)

Original: + Lµ g (cid:17) (cid:16) τµ g + ρLµ 2 g (cid:17) , with
Revised: + Lµ g (cid:17) (cid:16) τµ g + ρLµ 2 g (cid:17) , with

Original: L = max { L f , L g } .
Revised: L = max { L f , L g } .

Original: Then, the Jacobian J N ( x ) is Lipschitz continuous with constant L J :
Revised: Then, the Jacobian J N ( x ) is Lipschitz continuous with constant L J :

Original: [Equation]
Revised: [Equation]

Original: We next provide an upper-bound on the hypergradient estimation error for ESJ.
Revised: We next provide an upper-bound on the hypergradient estimation error for ESJ.

Original: Proposition 2.
Revised: Proposition 2.

Original: Suppose that Assumptions 1, 2, and 3 hold.
Revised: Suppose that Assumptions 1, 2, and 3 hold.

Original: Consider the ESJ algorithm.
Revised: Consider the ESJ algorithm.

Original: Its hypergradient estimation error can be upper-bounded as:
Revised: Its hypergradient estimation error can be upper-bounded as:

Original: [Equation]
Revised: [Equation]

Original: [Equation] where E [ · ] is conditioned on x k and taken over the Gaussian vectors { u k,j : j = 1 , ..., Q } . 
Revised: where E [ · ] is conditioned on x k and taken over the Gaussian vectors { u k,j : j = 1 , ..., Q } . [Equation]

Original: By using the smoothness property in Proposition 1 and the upper-bound in Proposition 2, we provide the following characterization of the convergence rate for ESJ.
Revised: By using the smoothness property in Proposition 1 and the upper-bound in Proposition 2, we provide the following characterization of the convergence rate for ESJ.

Original: Theorem 1 (Convergence of ESJ) .
Revised: Theorem 1 (Convergence of ESJ) .

Original: Suppose that Assumptions 1, 2, and 3 hold.
Revised: Suppose that Assumptions 1, 2, and 3 hold.

Original: Choose the innerand outer-loop stepsizes respectively as α ≤ 1 L and β = 14 L Φ , where L Φ =
Revised: Choose the innerand outer-loop stepsizes respectively as α ≤ 1 L and β = 14 L Φ , where L Φ =

Original: L + 2 L 2 + τM 2 µ g + ρLM + L 3 + τML µ 2 g + ρL
Revised: L + 2 L 2 + τM 2 µ g + ρLM + L 3 + τML µ 2 g + ρL

Original: 2 M µ 3 g .
Revised: 2 M µ 3 g .

Original: Then, the iterates x k for k = 0 , ..., K − 1 of ESJ in Algorithm 1 satisfy:
Revised: Then, the iterates x k for k = 0 , ..., K − 1 of ESJ in Algorithm 1 satisfy:

Original: [Equation]
Revised: [Equation]

Original: with Φ ∗ = inf x Φ( x ) and ∆ 1 = O (cid:16) (1 − αµ g ) N + pQ +
Revised: with Φ ∗ = inf x Φ( x ) and ∆ 1 = O (cid:16) (1 − αµ g ) N + pQ +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Further, choose K , N , µ , and Q at the order of O ( 1 (cid:15) ) , O (log 1 (cid:15) ) , O (min { 1 p (cid:113) (cid:15)dp , 1 p (cid:113) 1 dp } ) and
Revised: Further, choose K , N , µ , and Q at the order of O ( 1 (cid:15) ) , O (log 1 (cid:15) ) , O (min { 1 p (cid:113) (cid:15)dp , 1 p (cid:113) 1 dp } ) and

Original: O ( p(cid:15) ) .
Revised: O ( p(cid:15) ) .

Original: Then ESJ achieves an (cid:15) -accurate stationary point with O ( p(cid:15) 2 log 1 (cid:15) ) computations of the gradient
Revised: Then ESJ achieves an (cid:15) -accurate stationary point with O ( p(cid:15) 2 log 1 (cid:15) ) computations of the gradient

Original: ∇ y g ( x, y ) and O ( 1 (cid:15) ) computations of the gradients ∇ x f ( x, y ) and ∇ y f ( x, y ) .
Revised: ∇ y g ( x, y ) and O ( 1 (cid:15) ) computations of the gradients ∇ x f ( x, y ) and ∇ y f ( x, y ) .

Original: Theorem 1 characterizes the sublinear convergence of ESJ with respect to the number K of outer iterations due to the nonconvexity of the outer objective.
Revised: Theorem 1 characterizes the sublinear convergence of ESJ with respect to the number K of outer iterations due to the nonconvexity of the outer objective.

Original: The convergence error in ∆ 1 is due to the estimation error of the Jacobian matrix J ∗ via our ES oracle, which captures three types of errors: (a) the approximation error between J N and J ∗ via inner-loop gradient descent, which decreases exponentially w.r.t.
Revised: The convergence error in ∆ 1 is due to the estimation error of the Jacobian matrix J ∗ via our ES oracle, which captures three types of errors: (a) the approximation error between J N and J ∗ via inner-loop gradient descent, which decreases exponentially w.r.t.

Original: the number N of inner iterations due to the strong convexity of the inner objective; (b) the error between our ES oracle and the Jacobian J µ of the smoothed output E u y N ( x k + µu ) , which decreases sublinearly w.r.t.
Revised: the number N of inner iterations due to the strong convexity of the inner objective; (b) the error between our ES oracle and the Jacobian J µ of the smoothed output E u y N ( x k + µu ) , which decreases sublinearly w.r.t.

Original: the batch size Q , and (c) the error between the
Revised: the batch size Q , and (c) the error between the

Original: Jacobians J N and J µ , which can be controlled by the smoothness parameter µ .
Revised: Jacobians J N and J µ , which can be controlled by the smoothness parameter µ .

Original: Theorem 1 also indicates that ESJ requires only gradient computations to converge, and eliminates
Revised: Theorem 1 also indicates that ESJ requires only gradient computations to converge, and eliminates

Original: Hessian- and Jacobian-vector products required by the existing AID and ITD based bilevel optimizers (Pedregosa (2016), Franceschi et al.
Revised: Hessian- and Jacobian-vector products required by the existing AID and ITD based bilevel optimizers (Pedregosa (2016), Franceschi et al.

Original: (2018), Grazzi et al.
Revised: (2018), Grazzi et al.

Original: Thus, ESJ is computationally much more efﬁcient particularly for high-dimensional problems.
Revised: Thus, ESJ is computationally much more efﬁcient particularly for high-dimensional problems.

Original: 3. C ONVERGENCE A NALYSIS FOR ESJ-S
Revised: 3. C ONVERGENCE A NALYSIS FOR ESJ-S

Original: In this section, we apply the stochastic algorithm ESJ-S to the ﬁnite-sum objective in eq.
Revised: In this section, we apply the stochastic algorithm ESJ-S to the ﬁnite-sum objective in eq.

Original: (2) and analyze its convergence rate.
Revised: (2) and analyze its convergence rate.

Original: We ﬁrst note that the Lipschitzness property established in Propositionis general and can also be used here.
Revised: We ﬁrst note that the Lipschitzness property established in Propositionis general and can also be used here.

Original: The following proposition establishes an upper bound on the estimation error of Jacobian J ∗ by J N = ∂Y N ∂x , where Y N is the output of N inner SGD updates.
Revised: The following proposition establishes an upper bound on the estimation error of Jacobian J ∗ by J N = ∂Y N ∂x , where Y N is the output of N inner SGD updates.

Original: Proposition 3.
Revised: Proposition 3.

Original: Suppose that Assumptions 1, 2, and 4 hold.
Revised: Suppose that Assumptions 1, 2, and 4 hold.

Original: Consider the ESJ-S algorithm.
Revised: Consider the ESJ-S algorithm.

Original: Choose the inner-loop stepsize as α = 2 L + µ g , where L = max { L f , L g } .
Revised: Choose the inner-loop stepsize as α = 2 L + µ g , where L = max { L f , L g } .

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: where λ , Γ , and C γ < 1 are constants (see appendix F for their forms).
Revised: where λ , Γ , and C γ < 1 are constants (see appendix G for their forms).

Original: We next provide an upper-bound on the estimation error of the hypergradient by (cid:98) ∇ Φ( x k ) in eq.
Revised: We next provide an upper-bound on the estimation error of the hypergradient by (cid:98) ∇ Φ( x k ) in eq.

Original: Proposition 4.
Revised: Proposition 4.

Original: Suppose that Assumptions 1, 2, 3, and 4 hold.
Revised: Suppose that Assumptions 1, 2, 3, and 4 hold.

Original: Consider the ESJ-S algorithm.
Revised: Consider the ESJ-S algorithm.

Original: Set the inner-loop stepsize as α = 2 L + µ g where
Revised: Set the inner-loop stepsize as α = 2 L + µ g where

Original: L = max { L f , L g } .
Revised: L = max { L f , L g } .

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: where S and D f are the sizes of the inner and outer mini-batches, respectively.
Revised: where S and D f are the sizes of the inner and outer mini-batches, respectively.

Original: Based on Propositions 1, 3, and 4, we characterize the convergence rate for ESJ-S.
Revised: Based on Propositions 1, 3, and 4, we characterize the convergence rate for ESJ-S.

Original: Theorem 2 (Convergence of ESJ-S) .
Revised: Theorem 2 (Convergence of ESJ-S) .

Original: Suppose that Assumptions 1, 2, 3, and 4 hold.
Revised: Suppose that Assumptions 1, 2, 3, and 4 hold.

Original: Set the innerand outer-loop stepsizes respectively as α = 2 L + µ g and β = 14 L Φ , where L = max { L f , L g } and the constant L Φ are deﬁned in Theorem 1.
Revised: Set the innerand outer-loop stepsizes respectively as α = 2 L + µ g and β = 14 L Φ , where L = max { L f , L g } and the constant L Φ are deﬁned in Theorem 1.

Original: Then, the iterates x k , k = 0 , ..., K − 1 of ESJ-S satisfy:
Revised: Then, the iterates x k , k = 0 , ..., K − 1 of ESJ-S satisfy:

Original: [Equation]
Revised: [Equation]

Original: where ∆ 2 = O (cid:16) (1 − αµ g ) N + 1 S
Revised: where ∆ 2 = O (cid:16) (1 − αµ g ) N + 1 S

Original: + 1 D f + pQ + µ 2 dp 4 Q + µ 2 dp 3 (cid:17) .
Revised: + 1 D f + pQ + µ 2 dp 4 Q + µ 2 dp 3 (cid:17) .

Original: For ESJ-S to attain an (cid:15) -accurate stationary point, it sufﬁces to select K , S , and D f at the order of O ( 1 (cid:15) ) , and N , µ , and Q respectively at the order of O (log 1 (cid:15) ) , O (min { 1 p (cid:113) (cid:15)dp , 1 p (cid:113) 1 dp } ) and
Revised: For ESJ-S to attain an (cid:15) -accurate stationary point, it sufﬁces to select K , S , and D f at the order of O ( 1 (cid:15) ) , and N , µ , and Q respectively at the order of O (log 1 (cid:15) ) , O (min { 1 p (cid:113) (cid:15)dp , 1 p (cid:113) 1 dp } ) and

Original: O ( p(cid:15) ) , which thus amount to O ( p(cid:15) 3 log 1 (cid:15) ) computations of the gradient ∇ y G ( x, y, ξ ) and O ( 1 (cid:15) 2 ) computations of the gradients ∇ x F ( x, y, ξ ) and ∇ y F ( x, y, ξ ) .
Revised: O ( p(cid:15) ) , which thus amount to O ( p(cid:15) 3 log 1 (cid:15) ) computations of the gradient ∇ y G ( x, y, ξ ) and O ( 1 (cid:15) 2 ) computations of the gradients ∇ x F ( x, y, ξ ) and ∇ y F ( x, y, ξ ) .

Original: Comparing to the convergence error ∆ 1 in Theorem 1 for the deterministic algorithm ESJ, Theorem 2 for the stochastic algorithm ESJ-S captures two more sublinearly decreasing error terms 1 S and 1 D f respectively due to the sampling of inner and outer batches to estimate the objectives.
Revised: Comparing to the convergence error ∆ 1 in Theorem 1 for the deterministic algorithm ESJ, Theorem 2 for the stochastic algorithm ESJ-S captures two more sublinearly decreasing error terms 1 S and 1 D f respectively due to the sampling of inner and outer batches to estimate the objectives.

Original: To compare between ESJ-S and ESJ on the ﬁnite-sum objective in eq.
Revised: To compare between ESJ-S and ESJ on the ﬁnite-sum objective in eq.

Original: (2), Theorem 1 suggests that
Revised: (2), Theorem 1 suggests that

Original: ESJ attains an (cid:15) -accurate stationary point of eq.
Revised: ESJ attains an (cid:15) -accurate stationary point of eq.

Original: (2) with O ( pm(cid:15) 2 log 1 (cid:15) ) computations of ∇ G and
Revised: (2) with O ( pm(cid:15) 2 log 1 (cid:15) ) computations of ∇ G and

Original: O ( n(cid:15) ) computations of ∇ F .
Revised: O ( n(cid:15) ) computations of ∇ F .

Original: As a comparison, Theorem 2 suggests that ESJ-S achieves O ( p(cid:15) 3 log 1 (cid:15) ) computations of ∇ G and O ( 1 (cid:15) 2 ) computations of ∇ F , which outperforms ESJ in the large sample regime where the sample sizes n, m > 1 (cid:15) .
Revised: As a comparison, Theorem 2 suggests that ESJ-S achieves O ( p(cid:15) 3 log 1 (cid:15) ) computations of ∇ G and O ( 1 (cid:15) 2 ) computations of ∇ F , which outperforms ESJ in the large sample regime where the sample sizes n, m > 1 (cid:15) .

Original: This also reﬂects the advantage of SGD over GD.
Revised: This also reﬂects the advantage of SGD over GD.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We validate our algorithms over three bilevel problems: shallow hyper-representation (HR) with linear/2-layer net embedding model on synthetic data, deep HR with LeNet network (LeCun et al., 1998) on MNIST dataset, and few-shot meta-learning with ResNet-12 on miniImageNet dataset.
Revised: We validate our algorithms over three bilevel problems: shallow hyper-representation (HR) with linear/2-layer net embedding model on synthetic data, deep HR with LeNet network (LeCun et al., 1998) on MNIST dataset, and few-shot meta-learning with ResNet-12 on miniImageNet dataset.

Original: We run all models using a single NVIDIA Tesla P100 GPU.
Revised: We run all models using a single NVIDIA Tesla P100 GPU.

Revised: Hyperparamter Optimization (HO) experiments can be found in Appendix D.

Revised: All running time are in seconds.

Original: [Figure]
Revised: [Figure]

Original:  S HALLOW H YPER - R EPRESENTATION ON S YNTHETIC D ATA
Revised: 4. S HALLOW H YPER -R EPRESENTATION ON S YNTHETIC D ATA

Original: The hyper-representation (HR) problem (Franceschi et al., 2018; Grazzi et al., 2020) searches for a regression (or classiﬁcation) model following a two-phased optimization process.
Revised: The hyper-representation (HR) problem (Franceschi et al., 2018; Grazzi et al., 2020) searches for a regression (or classiﬁcation) model following a two-phased optimization process.

Original: The inner-level identiﬁes the optimal linear regressor parameters w , and the outer level solves for the optimal em- bedding model (i.e., representation) parameters λ .
Revised: The inner-level identiﬁes the optimal linear regressor parameters w , and the outer level solves for the optimal em- bedding model (i.e., representation) parameters λ .

Original: Mathematically, the problem can be modeled by the following bilevel optimization:
Revised: Mathematically, the problem can be modeled by the following bilevel optimization:

Original: [Equation]
Revised: [Equation]

Original: where X 2 ∈ R n 2 × m and X 1 ∈ R n 1 × m are matrices of synthesized training and validation data, and Y 2 ∈ R n 2 , Y 1 ∈ R n 1 are the corresponding response vectors.
Revised: where X 2 ∈ R n 2 × m and X 1 ∈ R n 1 × m are matrices of synthesized training and validation data, and Y 2 ∈ R n 2 , Y 1 ∈ R n 1 are the corresponding response vectors.

Original: In the case of shallow HR, the embedding function T ( · ; λ ) is either a linear transformation or a two-layer network.
Revised: In the case of shallow HR, the embedding function T ( · ; λ ) is either a linear transformation or a two-layer network.

Original: We generate data matrices X 1 , X 2 and labels Y 1 , Y 1 following the same process in Grazzi et al.
Revised: We generate data matrices X 1 , X 2 and labels Y 1 , Y 1 following the same process in Grazzi et al.

Original: We compare our ESJ algorithm with the baseline bilevel optimizers AID-FP and AID-CG and HOZOG (see Appendix C.1 for details about the baseline algorithms and hyperparameters used).
Revised: We compare our ESJ algorithm with the baseline bilevel optimizers AID-FP, AID-CG, ITD-R, and HOZOG (see Appendix C.1 for details about the baseline algorithms and hyperparameters used).

Original: Figure 1 show the performance comparison among the algorithms under linear and two-layer net embedding models.
Revised: Figure 1 show the performance comparison among the algorithms under linear and two-layer net embedding models.

Original: It can be observed that for both cases, our proposed method ESJ converges faster than all the other approaches, and the advantage of ESJ becomes more signiﬁcant in Figure(d), which is under a higher-dimensional model of a two-layer net.
Revised: It can be observed that for both cases, our proposed method ESJ converges faster than all the other approaches, and the advantage of ESJ becomes more signiﬁcant in Figure(d), which is under a higher-dimensional model of a two-layer net.

Original: In particular, ESJ outperforms the existing ES-based algorithm HOZOG.
Revised: In particular, ESJ outperforms the existing ES-based algorithm HOZOG.

Original: This is because HOZOG uses the ES technique to approximate the entire hypergradient, which likely incurs a large estimation error.
Revised: This is because HOZOG uses the ES technique to approximate the entire hypergradient, which likely incurs a large estimation error.

Original: In contrast, our ESJ exploits the structure of the hypergradient and uses ES only to estimate the response Jacobian so that the estimation of hypergradient is more accurate.
Revised: In contrast, our ESJ exploits the structure of the hypergradient and uses ES only to estimate the response Jacobian so that the estimation of hypergradient is more accurate.

Original: Such an advantage is more evident under a twolayer net model, where HOZOG does not converge as shown in Figure 1 (d).
Revised: Such an advantage is more evident under a twolayer net model, where HOZOG does not converge as shown in Figure 1 (d).

Original: This can be explained by the ﬂat hypergradient norm as shown in Figure 1 (e), which indicates that the hypergradient estimator in HOZOG fails to provide a good descent direction for the outer optimizer.
Revised: This can be explained by the ﬂat hypergradient norm as shown in Figure 1 (e), which indicates that the hypergradient estimator in HOZOG fails to provide a good descent direction for the outer optimizer.

Original: Figure 1 (c) and (f) further show that the convergence of ESJ does not change substantially with the number N of inner GD steps, and hence tuning of N in practice is not costly.
Revised: Figure 1 (c) and (f) further show that the convergence of ESJ does not change substantially with the number N of inner GD steps, and hence tuning of N in practice is not costly.

Original: 4. D EEP H YPER -R EPRESENTATION ON MNIST D ATASET
Revised: 4. D EEP H YPER -R EPRESENTATION ON MNIST D ATASET

Original: In order to demonstrate the advantage of our proposed algorithms in large neural net models, we perform deep hyper-representation to classify MNIST images by learning an entire LeNet network.
Revised: In order to demonstrate the advantage of our proposed algorithms in large neural net models, we perform deep hyper-representation to classify MNIST images by learning an entire LeNet network.

Original: The corresponding bilevel problem is given by
Revised: The corresponding bilevel problem is given by

Original: [Equation]
Revised: [Equation]

Original: (13) where f ( x i ; λ )
Revised: (13) where f ( x i ; λ )

Original: ∈ R p corresponds to features extracted from data point x i , L ( · , · ) is the crossentropy loss function, c
Revised: ∈ R p corresponds to features extracted from data point x i , L ( · , · ) is the crossentropy loss function, c

Original: = 10 is the number of categories, and D in and D out are data used to compute respectively inner and outer loss functions.
Revised: = 10 is the number of categories, and D in and D out are data used to compute respectively inner and outer loss functions.

Original: Since the sizes of D out and D in are large in the case of MNIST dataset, we apply the more efﬁcient stochastic algorithm ESJ-S in Algorithm 2 with a minibatch size B = 256 to estimate the inner and outer losses L in and L out .
Revised: Since the sizes of D out and D in are large in the case of MNIST dataset, we apply the more efﬁcient stochastic algorithm ESJ-S in Algorithm 2 with a minibatch size B = 256 to estimate the inner and outer losses L in and L out .

Original: [Figure]
Revised: [Figure]

Original: Figure 2 compares the classiﬁcation accuracy on the outer dataset D out among the different methods.
Revised: Figure 2 compares the classiﬁcation accuracy on the outer dataset D out among the different methods.

Original: Our algorithm ESJ-S converges with the fastest rate and attains the best accuracy with the lowest variance among all algorithms.
Revised: Our algorithm ESJ-S converges with the fastest rate and attains the best accuracy with the lowest variance among all algorithms.

Original: Note that ESJ-S is able to attain the same accuracy of 0 .
Revised: Note that ESJ-S is able to attain the same accuracy of 0 .

Original: + obtained by the standard training of all parameters with one-phased optimization on the MNIST dataset using the same backbone network.
Revised: + obtained by the standard training of all parameters with one-phased optimization on the MNIST dataset using the same backbone network.

Original: All other methods fail to recover such a level of performance, and instead saturate around an accuracy of 0 .
Revised: All other methods fail to recover such a level of performance, and instead saturate around an accuracy of 0 .

Original: Further, Figure 2(c) indicates that the convergence of
Revised: Further, Figure 2(c) indicates that the convergence of

Original: ESJ-S does not change substantially with the number N of inner SGD steps.
Revised: ESJ-S does not change substantially with the number N of inner SGD steps.

Original: This demonstrates the robustness of our method when applied to complex function geometries such as deep nets.
Revised: This demonstrates the robustness of our method when applied to complex function geometries such as deep nets.

Original: [Equation] EARNING OVER M INI I MAGE N ET
Revised: 4. F EW -S HOT M ETA -L EARNING OVER M INI I MAGE N ET

Original: To study our algorithms over larger neural nets, we study the few-shot image recognition problem, where classiﬁcation tasks T i , i = 1 , ..., m are sampled over a distribution P T .
Revised: To study our algorithms over larger neural nets, we study the few-shot image recognition problem, where classiﬁcation tasks T i , i = 1 , ..., m are sampled over a distribution P T .

Original: In particular, we consider the ANIL meta-learning method (Raghu et al., 2019; Ji et al., 2020a), where all tasks share common embedding features parameterized by φ , and each task T i has its task-speciﬁc parameter w i for i = 1 , ..., m .
Revised: In particular, we consider the ANIL meta-learning method (Raghu et al., 2019; Ji et al., 2020a), where all tasks share common embedding features parameterized by φ , and each task T i has its task-speciﬁc parameter w i for i = 1 , ..., m .

Original: More speciﬁcally, we set φ to be the parameters of the convolutional part of a deep CNN model (e.g., ResNet-12 network) and w includes the parameters of the last classiﬁcation layer.
Revised: More speciﬁcally, we set φ to be the parameters of the convolutional part of a deep CNN model (e.g., ResNet-12 network) and w includes the parameters of the last classiﬁcation layer.

Original: All model parameters ( φ , w ) are trained following a bilevel procedure.
Revised: All model parameters ( φ , w ) are trained following a bilevel procedure.

Original: In the inner-loop, the base learner of each task T i ﬁxes φ and minimizes its loss function over a training set S i to obtain its adapted parameters w ∗ i .
Revised: In the inner-loop, the base learner of each task T i ﬁxes φ and minimizes its loss function over a training set S i to obtain its adapted parameters w ∗ i .

Original: At the outer stage, the meta-learner computes the test loss for each task T i using the parameters ( φ , w ∗ i ) over a test set D i , and optimizes the parameters φ of the common embedding function by minimizing the meta-objective L meta over all classiﬁcation tasks.
Revised: At the outer stage, the meta-learner computes the test loss for each task T i using the parameters ( φ , w ∗ i ) over a test set D i , and optimizes the parameters φ of the common embedding function by minimizing the meta-objective L meta over all classiﬁcation tasks.

Original: The problem can be expressed as the following bilevel optimization
Revised: The problem can be expressed as the following bilevel optimization

Original: [Equation]
Revised: [Equation]

Original: 14) where we collect all task-speciﬁc parameters into (cid:101)
Revised: where we collect all task-speciﬁc parameters into (cid:101)

Original: w = ( w 1 , ..., w m ) and the corresponding minimizers into (cid:101)
Revised: w = ( w 1 , ..., w m ) and the corresponding minimizers into (cid:101)

Original: w ∗ =
Revised: w ∗ =

Original: ( w ∗ 1 , ..., w ∗ m ) .
Revised: ( w ∗ 1 , ..., w ∗ m ) .

Original: The functions L S i ( φ, w i )
Revised: The functions L S i ( φ, w i )

Original: = 1 |S i | (cid:80) ζ ∈S
Revised: = 1 |S i | (cid:80) ζ ∈S

Original: i
Revised: i

Original: L ( φ, w i ; ζ ) + R ( w i )) and L D i ( φ, w ∗ i ) =
Revised: L ( φ, w i ; ζ ) + R ( w i )) and L D i ( φ, w ∗ i ) =

Original: 1 |D i | (cid:80) ξ ∈D i L ( φ, w ∗ i ; ξ ) correspond respectively to the training and test loss functions for task T i , with R a strongly-convex regularizer and L a classiﬁcation loss function.
Revised: 1 |D i | (cid:80) ξ ∈D i L ( φ, w ∗ i ; ξ ) correspond respectively to the training and test loss functions for task T i , with R a strongly-convex regularizer and L a classiﬁcation loss function.

Original: In our setting, since the task-speciﬁc parameters correspond to the weights of the last linear layer, the inner-level objective L adapt ( φ, (cid:101) w ) is strongly convex with respect to (cid:101)
Revised: In our setting, since the task-speciﬁc parameters correspond to the weights of the last linear layer, the inner-level objective L adapt ( φ, (cid:101) w ) is strongly convex with respect to (cid:101)

Original: w = ( w 1 , ..., w m ) .
Revised: w = ( w 1 , ..., w m ) .

Original: We note that the problem studied in Section 4.2 can be seen as single-task instances of the more general multi-task learning problem in eq.
Revised: We note that the problem studied in Section 4.2 can be seen as single-task instances of the more general multi-task learning problem in eq.

Original: However, in contrast to the problem in Section 4.2, the sizes of the datasets D i and S i are usually small in few-shot learning and full GD can be applied here.
Revised: However, in contrast to the problem in Section 4.2, the sizes of the datasets D i and S i are usually small in few-shot learning and full GD can be applied here.

Original: Hence, we use ESJ (Algorithm 1) here.
Revised: Hence, we use ESJ (Algorithm 1) here.

Original: Also since the number m of tasks in few-shot classiﬁcation datasets is often very large, it is preferable to sample a minibatch of i.i.d.
Revised: Also since the number m of tasks in few-shot classiﬁcation datasets is often very large, it is preferable to sample a minibatch of i.i.d.

Original: tasks by P T at each meta (i.e., outer) iteration and update the meta parameters based on these tasks.
Revised: tasks by P T at each meta (i.e., outer) iteration and update the meta parameters based on these tasks.

Original: [Figure]
Revised: [Figure]

Original: Algorithm
Revised: Algorithm

Original: Time 
Revised: Time (hours)

Original: [Figure]
Revised: ESJ 2. MetaOptNet 20+ ProtNet 4.

Original: [Figure]
Revised: [Figure]

Original: We conduct few-shot meta-learning on the miniImageNet dataset (Vinyals et al., 2016) using ResNet-12 as a backbone network for feature extraction.
Revised: We conduct few-shot meta-learning on the miniImageNet dataset (Vinyals et al., 2016) using two different backbone networks for feature extraction: ResNet-12 and CNN4 (Vinyals et al., 2016).

Original: The dataset description and hyperparameter details can be found in Appendix C.4.
Revised: The dataset description and hyperparameter details can be found in Appendix C.4.

Original: Since the baseline bilevel optimizers AID and ITD scale poorly to the ResNet-12 setting, we do not compare with these methods. Instead, we compare our method ESJ with two baseline methods  MetaOptNet (Lee et al., 2019) and ProtoNet (Snell et al., 2017), which have previously been adopted for such a network.
Revised: We compare our algorithm ESJ with four baseline methods for few-shot meta-learning MAML (Finn et al., 2017), ANIL (Raghu et al., 2019), MetaOptNet (Lee et al., 2019), and ProtoNet (Snell et al., 2017).

Original: Note that both of these two approaches are not bilevel optimizers but empirical methods designed speciﬁcally to solve the fewshot meta-learning problem.

Original: We run their efﬁcient Pytorch Lightning implementations available at the learn2learn repository (Arnold et al., 2019).
Revised: We run their efﬁcient Pytorch Lightning implementations available at the learn2learn repository (Arnold et al., 2019).

Original: Figure 3(a) and (b) show that our algorithm ESJ converges faster than the othertwo baseline methods.
Revised: Figure 3(a) and (b) show that our algorithm ESJ converges faster than the other baseline methods.

Revised: Also, Comparing Figure 3(a) and (b), the advantage of our method over the baselines MAML and

Revised: ANIL becomes more signiﬁcant as the size of the network increases.

Original: Further, Figure 3(c) shows that MetaOptNet did not reach 69% accuracy after 20 hours of training.
Revised: Further, Figure 3(c) shows that MetaOptNet did not reach 69% accuracy after 20 hours of training with ResNet-12 network.

Original: In comparison, our ESJ is able to attain 69% in less than three hours, which is about 1. times less than the time taken for ProtoNet to reach the same performance level.
Revised: In comparison, our ESJ is able to attain 69% in less than three hours, which is about 1.5 times less than the time taken for ProtoNet to reach the same performance level.

Original: Both ESJ and ProtoNet saturate around 70% accuracy after 10 hours of training.
Revised: Both ESJ and ProtoNet saturate around 70% accuracy after 10 hours of training.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose a novel ES-based approach for bilevel optimization, which eliminates all second-order computations in the gradient-based approaches.
Revised: In this paper, we propose a novel ES-based approach for bilevel optimization, which eliminates all second-order computations in the gradient-based approaches.

Original: Compared to the existing ES-based algorithm, our approach explores the analytical structure of the hypergradient, and hence leads to much more efﬁcient and accurate hypergradient estimation.
Revised: Compared to the existing ES-based algorithm, our approach explores the analytical structure of the hypergradient, and hence leads to much more efﬁcient and accurate hypergradient estimation.

Original: Thus, our algorithm outperforms the existing algorithms in the experiments, particularly in the high-dimensional applications.
Revised: Thus, our algorithm outperforms the existing algorithms in the experiments, particularly in the high-dimensional applications.

Original: We also characterize the convergence rate for our proposed algorithms and show that the polynomial-time complexity can be achieved.
Revised: We also characterize the convergence rate for our proposed algorithms and show that the polynomial-time complexity can be achieved.

Original: We anticipate that our approach will be useful for accelerating bilevel algorithms in various machine learning problems.
Revised: We anticipate that our approach will be useful for accelerating bilevel algorithms in various machine learning problems.

Original:  R EPRODUCIBILITY C HECKLIST
Revised:  R EPRODUCIBILITY C HECKLIST

Original: To ensure reproducibility, we use the Machine Learning Reproducibility Checklist v2.0, Apr.2020 (Pineau et al., 2021).
Revised: To ensure reproducibility, we use the Machine Learning Reproducibility Checklist v2.0, Apr.2020 (Pineau et al., 2021).

Original: An earlier version of this checklist (v1.2) was used for NeurIPS 2019 (Pineau et al., 2021).
Revised: An earlier version of this checklist (v1.2) was used for NeurIPS 2019 (Pineau et al., 2021).

Original: • For all models and algorithms presented, – A clear description of the mathematical settings, algorithm, and/or model.
Revised: • For all models and algorithms presented, – A clear description of the mathematical settings, algorithm, and/or model.

Original: We clearly describe all of the settings, formulations, and algorithms in Section 2.
Revised: We clearly describe all of the settings, formulations, and algorithms in Section 2.

Original: – A clear explanation of any assumptions.
Revised: – A clear explanation of any assumptions.

Original: All assumptions are stated in Section 3.
Revised: All assumptions are stated in Section 3.

Original: and details are clearly explained in Section 3.1.
Revised: and details are clearly explained in Section 3.1.

Original: – An analysis of the complexity (time, space, sample size) of any algorithm.
Revised: – An analysis of the complexity (time, space, sample size) of any algorithm.

Original: We provide the time/computational complexity analysis for our algorithms in Theoremsand 2.
Revised: We provide the time/computational complexity analysis for our algorithms in Theoremsand 2.

Original: • For any theoretical claim , – A clear statement of the claim.
Revised: • For any theoretical claim , – A clear statement of the claim.

Original: A clear statement of theoretical claims are made in Section 3.
Revised: A clear statement of theoretical claims are made in Section 3.

Original: – A complete proof of the claim.
Revised: – A complete proof of the claim.

Original: The complete proofs of all claims are available in AppendixD, Appendix E,  and
Revised: The complete proofs of all claims are available in Appendix E, Appendix F, and

Original: Appendix F. • For all datasets used, check if you include: – The relevant statistics, such as number of examples.
Revised: Appendix G. • For all datasets used, check if you include: – The relevant statistics, such as number of examples.

Original: We use widely adopted datasets MNIST and miniImageNet in Section 4.
Revised: We use widely adopted datasets MNIST and miniImageNet in Section 4.

Original: The related statistics can be seen at http://yann.lecun.com/exdb/mnist/ .
Revised: The related statistics can be seen at http://yann.lecun.com/exdb/mnist/ .

Original: – The details of train/validation/test splits.
Revised: – The details of train/validation/test splits.

Original: We give this information in the Supplementary Appendix C. – An explanation of any data that were excluded, and all pre-processing step.
Revised: We give this information in the Supplementary Appendix C. – An explanation of any data that were excluded, and all pre-processing step.

Original: We did not exclude any data or perform any pre-processing.
Revised: We did not exclude any data or perform any pre-processing.

Original: – For new data collected,a complete description of the data collection process, such as instructions to annotators and methods for quality control.
Revised: – For new data collected,a complete description of the data collection process, such as instructions to annotators and methods for quality control.

Original: We do not collect or release new datasets.
Revised: We do not collect or release new datasets.

Original: • For all shared code related to this work, check if you include: – Training code.
Revised: • For all shared code related to this work, check if you include: – Training code.

Original: The training code is available in our code in supplementary material.
Revised: The training code is available in our code in supplementary material.

Original: – Evaluation code.
Revised: – Evaluation code.

Original: The evaluation code is available in our code in supplementary material.
Revised: The evaluation code is available in our code in supplementary material.

Original: – (Pre-)trained model(s).
Revised: – (Pre-)trained model(s).

Original: We do not release any pre-trained models.
Revised: We do not release any pre-trained models.

Original: • For all reported experimental results , check if you include: – The range of hyper-parameters considered, method to select the best hyperparameter conﬁguration, and speciﬁcation of all hyper-parameters used to generate results.
Revised: • For all reported experimental results , check if you include: – The range of hyper-parameters considered, method to select the best hyperparameter conﬁguration, and speciﬁcation of all hyper-parameters used to generate results.

Original: We provide all details of the hyper-parameter tuning in Supplementary Appendix C. – A clear deﬁnition of the speciﬁc measure or statistics used to report results.
Revised: We provide all details of the hyper-parameter tuning in Supplementary Appendix C. – A clear deﬁnition of the speciﬁc measure or statistics used to report results.

Original: We use the classiﬁcation accuracy on test-set and the loss on the train-set.
Revised: We use the classiﬁcation accuracy on test-set and the loss on the train-set.

Original: – A description of results with central tendency (e.g. mean) & variation (e.g. error bars).
Revised: – A description of results with central tendency (e.g. mean) & variation (e.g. error bars).

Original: We do not report the mean and standard deviation for experiments.
Revised: We do not report the mean and standard deviation for experiments.

Original: – The average runtime for each result, or estimated energy cost.
Revised: – The average runtime for each result, or estimated energy cost.

Original: We report the running time of the algorithms in Section 4.
Revised: We report the running time of the algorithms in Section 4.

Original: – A description of the computing infrastructure used.
Revised: – A description of the computing infrastructure used.

Original: All detailed descriptions are presented in Section 4.
Revised: All detailed descriptions are presented in Section 4.

Original: Supplementary Material A S TOCHASTIC B ILEVEL O PTIMIZER B ASED ON ES M ETHOD
Revised: Supplementary Material A S TOCHASTIC B ILEVEL O PTIMIZER B ASED ON ES M ETHOD

Original: In the following, we present the algorithm speciﬁcation for our proposed stochastic bilevel optimizer based on ES method, which we call ESJ-S .
Revised: In the following, we present the algorithm speciﬁcation for our proposed stochastic bilevel optimizer based on ES method, which we call ESJ-S .

Original: Algorithm2 Stochastic bilevel
Revised: Algorithm

Original: 1: Input: lower- and upper-level stepsizes α, β > 0 , initializations x 0 ∈ R p and y 0 ∈ R d  .
Revised: 1: Input: lower- and upper-level stepsizes α, β > 0 , initializations x 0 ∈ R p and y 0 ∈ R d , inner and outer iterations numbers K and N , and number of Gaussian vectors Q .

Original: 2: for k = 0 , 1 , ..., K do 3:
Revised: 2: for k = 0 , 1 , ..., K do 3:

Original: Set Y 

Revised: Set Y

Original: k = y 0 ,
Revised: k = y 0 ,

Original: Y 0 k,j = y 0 , j = 1 , ..., Q 4:
Revised: Y 0 k,j = y 0 , j = 1 , ..., Q 4:

Original: Generate u k,j = N (0 , I ) ∈
Revised: Generate u k,j = N (0 , I ) ∈

Original: R p , j = 1 , ..., Q 5: for t = 1 , 2 , ..., N do 6:
Revised: R p , j = 1 , ..., Q 5: for t = 1 , 2 , ..., N do 6:

Original: Draw a sample batch S t −7:
Revised: Draw a sample batch S t −7:

Original: Update Y t k
Revised: Update Y tk =

Original: = Y t − 1 k − α ∇ y G ( x k , Y t − 1 k ; S t − 1 ) 8: for j = 1 , ..., Q do 9:
Revised: Y t − 1 k − α ∇ y G ( x k , Y t − 1 k ; S t − 1 ) 8: for j = 1 , ..., Q do 9:

Original: Update Y tk,j = Y t − 1 k,j − α ∇ y G (cid:16) x k + µu k,j , Y t − 1 k,j ; S t − 1 (cid:17) 10: end for 11: end for 12:
Revised: Update Y tk,j = Y t − 1 k,j − α ∇ y G (cid:16) x k + µu k,j , Y t − 1 k,j ; S t − 1 (cid:17) 10: end for 11: end for 12:

Original: Compute δ j = Y Nk,j − Y Nk µ , j = 1 , ..., Q 13:
Revised: Compute δ j = Y Nk,j − Y Nk µ , j = 1 , ..., Q 13:

Original: Draw a sample batch D F 14:
Revised: Draw a sample batch D F 14:

Original: Compute (cid:98) ∇ Φ( x k ) =
Revised: Compute (cid:98) ∇ Φ( x k ) =

Original: ∇ x F ( x k , Y Nk ; D F ) + 1 Q (cid:80)
Revised: ∇ x F ( x k , Y Nk ; D F ) + 1 Q (cid:80)

Original: Qj =1 (cid:10) δ j , ∇ y F ( x k , Y Nk ; D F ) (cid:11) u k,j 15:
Revised: Qj =1 (cid:10) δ j , ∇ y F ( x k , Y Nk ; D F ) (cid:11) u k,j 15:

Original: Update x k +

Revised: Update x k +

Original: =
Revised: =

Original: x
Revised: x

Original: k − β (cid:98) ∇ Φ( x k ) 16: end for
Revised: k − β (cid:98) ∇ Φ( x k ) 16: end for

Original: B A DDITIONAL R ELATED W ORK
Revised: B A DDITIONAL R ELATED W ORK

Original: In this section, we provide further related work on the applications of bilevel optimization and ES methods.
Revised: In this section, we provide further related work on the applications of bilevel optimization and ES methods.

Original: Bilevel optimization applications.
Revised: Bilevel optimization applications.

Original: Bilevel optimization has been employed in various applications such as few-shot meta-learning (Snell et al., 2017; Franceschi et al., 2018; Rajeswaran et al., 2019;
Revised: Bilevel optimization has been employed in various applications such as few-shot meta-learning (Snell et al., 2017; Franceschi et al., 2018; Rajeswaran et al., 2019;

Original: Z ¨ ugner & G ¨ unnemann, 2019; Ji et al., 2020a;b), hyperparameter optimization (Franceschi et al., 2017; Mackay et al., 2018; Shaban et al., 2019), neural architecture search (Liu et al., 2018a; Zhang et al., 2021), etc.
Revised: Z¨ugner & G¨unnemann, 2019; Ji et al., 2020a;b), hyperparameter optimization (Franceschi et al., 2017; Mackay et al., 2018; Shaban et al., 2019), neural architecture search (Liu et al., 2018a; Zhang et al., 2021), etc.

Original: This paper demonstrates the superior performance of the proposed ES-based bilevel optimizer in meta-learning and hyperparameter optimization.
Revised: This paper demonstrates the superior performance of the proposed ES-based bilevel optimizer in meta-learning and hyperparameter optimization.

Original: ES applications.
Revised: ES applications.

Original: Evolution Strategies (ES) method has been studied for more than two decades (see review papers (Hansen, 2006; Varelas et al., 2018)).
Revised: Evolution Strategies (ES) method has been studied for more than two decades (see review papers (Hansen, 2006; Varelas et al., 2018)).

Original: In particular, Nesterov & Spokoiny (2017) proposed a simple and effective ES-gradient via Gaussian smoothing, which was further extended to the stochastic setting by Ghadimi & Lan (2013).
Revised: In particular, Nesterov & Spokoiny (2017) proposed a simple and effective ES-gradient via Gaussian smoothing, which was further extended to the stochastic setting by Ghadimi & Lan (2013).

Original: Such an ES technique has exhibited great effectiveness in various applications including meta-reinforcement learning (Song et al., 2019), hyperparameter optimization (Gu et al., 2021), adversarial machine learning (Ji et al., 2019;
Revised: Such an ES technique has exhibited great effectiveness in various applications including meta-reinforcement learning (Song et al., 2019), hyperparameter optimization (Gu et al., 2021), adversarial machine learning (Ji et al., 2019;

Original: Liu et al., 2018b), minimax optimization (Liu et al., 2020b; Xu et al., 2020), etc.
Revised: Liu et al., 2018b), minimax optimization (Liu et al., 2020b; Xu et al., 2020), etc.

Original: This paper proposes a novel ES-based
Revised: This paper proposes a novel ES-based

Original: Jacobian estimator for accelerating bilevel optimization.
Revised: Jacobian estimator for accelerating bilevel optimization.

Revised: The ES method has also been used for studying hyperparameter optimization problems.

Revised: For example, Mackay et al.

Revised: (2018) models the response function itself as a neural network (where each layer involves an afﬁne transformation of hyperparameters) using the Self-Tuning Networks (STNs).

Revised: An improved and more stable version of STNs was further proposed in Bae & Grosse (2020), which focused on accurately approximating the response Jacobian rather than the response function itself.

Original: C F URTHER S PECIFICATIONS FOR E XPERIMENTS IN S ECTION
Revised: C F URTHER S PECIFICATIONS FOR E XPERIMENTS IN S ECTION

Original: We note that the smoothing parameter µ (in Algorithms 1 and 2) was easy to set and a value of 0 .or 0 .
Revised: We note that the smoothing parameter µ (in Algorithms 1 and 2) was easy to set and a value of 0 .or 0 .

Original: 01 yields a good starting point across all our experiments.
Revised: 01 yields a good starting point across all our experiments.

Original: The batch size Q (in Algorithmsand 2) is ﬁxed to 1 (i.e., we use one Jacobian oracle) in all our experiments.
Revised: The batch size Q (in Algorithmsand 2) is ﬁxed to 1 (i e., we use one Jacobian oracle) in all our experiments.

Original: C. S PECIFICATIONS ON B ASELINE B ILEVEL A PPROACHES IN S ECTION 4. 
Revised: C. S PECIFICATIONS ON B ASELINE B ILEVEL A PPROACHES IN S ECTION 4.

Original: We compare our algorithm ESJ with the following baseline methods:
Revised: We compare our algorithm ESJ with the following baseline methods:

Original: • HOZOG (Gu et al., 2021): a hyperparameter optimization algorithm that uses evolution strategies to estimate the entire hypergradient (both the direct and indirect component).
Revised: • HOZOG (Gu et al., 2021): a hyperparameter optimization algorithm that uses evolution strategies to estimate the entire hypergradient (both the direct and indirect component).

Original: We use our own implementation for this method.
Revised: We use our own implementation for this method.

Original: • AID-CG (Grazzi et al., 2020; Rajeswaran et al., 2019): approximate implicit differentiation with conjugate gradient.
Revised: • AID-CG (Grazzi et al., 2020; Rajeswaran et al., 2019): approximate implicit differentiation with conjugate gradient.

Original: We use its implementation provided at https://github.com/ prolearner/ hypertorch • AID-FP (Grazzi et al., 2020): approximate implicit differentiation with ﬁxed-point.
Revised: We use its implementation provided at https://github.com/ prolearner/hypertorch • AID-FP (Grazzi et al., 2020): approximate implicit differentiation with ﬁxed-point.

Original: We experimented with its implementation at the repositery https://github.com/prolearner/ hypertorch • reverse (Franceschi et al., 2017) an iterative differentiation method that computes hypergradients by backpropagating through the unrolled inner loop optimization steps.
Revised: We experimented with its implementation at the repositery https://github.com/prolearner/ hypertorch • ITD-R (REVERSE) (Franceschi et al., 2017): an iterative differentiation method that computes hypergradients using reverse mode automatic differention (RMAD).

Original: We use its implementation provided at https://github.com/prolearner/hypertorch .
Revised: We use its implementation provided at https://github.com/prolearner/hypertorch .

Original: C. H YPERPARAMETERS DETAILS FOR SHALLOW HR EXPERIMENTS IN S ECTION 4. 
Revised: C. H YPERPARAMETERS DETAILS FOR SHALLOW HR EXPERIMENTS IN S ECTION 4.

Original: For the linear embedding case, we set the smoothing parameter µ to be 0 .
Revised: For the linear embedding case, we set the smoothing parameter µ to be 0 .

Original: 01 for ESJ and HOZOG.
Revised: 01 for ESJ and HOZOG.

Original: We use the following hyperparameters for all compared methods.
Revised: We use the following hyperparameters for all compared methods.

Original: The number of inner GD steps is ﬁxed to N = 20 with the learning rate of α = 0 .
Revised: The number of inner GD steps is ﬁxed to N = 20 with the learning rate of α = 0 .

Original: For the outer optimizer, we use Adam (Kingma & Ba, 2014) with a learning rate of 0 .
Revised: For the outer optimizer, we use Adam (Kingma & Ba, 2014) with a learning rate of 0 .

Original: The value of γ in eq.
Revised: The value of γ in eq.

Original: (12) is set to be 0 .
Revised: (12) is set to be 0 .

Original: For the two-layer net case, we use µ = 0 .
Revised: For the two-layer net case, we use µ = 0 .

Original: 1 for ESJ and HOZOG.
Revised: 1 for ESJ and HOZOG.

Original: For all methods, we set N = 10 , α = 0 .
Revised: For all methods, we set N = 10 , α = 0 .

Original: 001 , β = 0 .
Revised: 001 , β = 0 .

Original: 001 , and use Adam with a learning rate of 0 .
Revised: 001 , and use Adam with a learning rate of 0 .

Original: as the outer optimizer.
Revised: as the outer optimizer.

Original: C. S PECIFICATIONS ON B ASELINE S TOCHASTIC A LGORITHMS IN S ECTION 4. 
Revised: C. S PECIFICATIONS ON B ASELINE S TOCHASTIC A LGORITHMS IN S ECTION 4.

Original: We compare our stochastic algorithm ESJ-S with the following baseline stochastic bilevel algorithms.
Revised: We compare our stochastic algorithm ESJ-S with the following baseline stochastic bilevel algorithms.

Original: • stocBiO: an approximate implicit differentiation method that uses Neumann Series to estimate the Hessian inverse.
Revised: • stocBiO: an approximate implicit differentiation method that uses Neumann Series to estimate the Hessian inverse.

Original: We use its implementation available at https://github.com/ JunjieYang97/StocBio .
Revised: We use its implementation available at https://github.com/ JunjieYang97/StocBio .

Original: • AID-CG-S and AID-FP-S: stochastic versions of AID-CG and AID-FP, respectively.
Revised: • AID-CG-S and AID-FP-S: stochastic versions of AID-CG and AID-FP, respectively.

Original: We use their implementations in the repository https://github.com/prolearner/ hypertorch .
Revised: We use their implementations in the repository https://github.com/prolearner/ hypertorch .

Original: C. S PECIFICATIONS FOR F EW - SHOT M ETA -L EARNING IN S ECTION 4. 
Revised: C. S PECIFICATIONS FOR F EW - SHOT M ETA -L EARNING IN S ECTION 4.

Original: The miniImageNet dataset (Vinyals et al., 2016) is a large-scale benchmark for few-shot learning generated from ImageNet (Russakovsky et al., 2015) Russakovsky.
Revised: The miniImageNet dataset (Vinyals et al., 2016) is a large-scale benchmark for few-shot learning generated from ImageNet (Russakovsky et al., 2015) Russakovsky.

Original: The dataset consists ofclasses with each class containing 600 images of size 84 × 84.
Revised: The dataset consists ofclasses with each class containing 600 images of size 84 × 84.

Original: Following (Arnold et al., 2019), we split the classes into 64 classes for meta-training, 16 classes for meta-validation, and 20 classes for meta-testing.
Revised: Following (Arnold et al., 2019), we split the classes into 64 classes for meta-training, 16 classes for meta-validation, and 20 classes for meta-testing.

Original: More specﬁcally, we use 20000 tasks for meta-training, 600 tasks for meta-validation, and 600 tasks for meta-testing.
Revised: More specﬁcally, we use 20000 tasks for meta-training, 600 tasks for meta-validation, and 600 tasks for meta-testing.

Original: We normalize all image pixels by their means and standard deviations over RGB channels and do not perform any additional data augmentation.
Revised: We normalize all image pixels by their means and standard deviations over RGB channels and do not perform any additional data augmentation.

Original: At each meta-iteration, we sample a batch of 16 training tasks and update the parameters based on these tasks.
Revised: At each meta-iteration, we sample a batch of 16 training tasks and update the parameters based on these tasks.

Original: We set the smoothness parameter to be µ = 0 .
Revised: We set the smoothness parameter to be µ = 0 .

Original: 1 and use N = 30 inner steps.
Revised: 1 and use N = 30 inner steps.

Original: We use SGD with a learning rate of α = 0 .
Revised: We use SGD with a learning rate of α [Figure]

Original: 01 as  inner optimizer and Adam with a learning rate of β = 0 .
Revised: Hyperparameter optimization (HO) is the problem of ﬁnding the set of the best hyperparamters (either representational or regularization parameters) that yield the optimal value of some criterion of model quality (usually a validation loss on unseen data). HO can be posed as a bilevel optimization problem in which the inner problem corresponds to ﬁnding the model parameters by minimizing a training loss (usually regularized) for the given hyperparameters and then the outer problem minimizes over the hyperparameters. Hence, HO can be mathematically expressed as follows

Original: 01 as outer (meta) optimizer.
Revised: [Equation] where L is a loss function (e.g., logistic loss), R ( w, λ ) is a regularizer, and D tr and D val are respectively training and validation data. Note that the loss function used to identify hyperparameters must be different from the one used to ﬁnd model parameters; otherwise models with higher complexities would be always favored. This is usually achieved in HO by using different data splits (here D val and D tr ) to compute validation and training losses, and by adding a regularizer term on the training loss.

Original: D S UPPORTING T ECHNICAL L EMMAS
Revised: Following (Franceschi et al., 2017; Grazzi et al., 2020), we perform classiﬁcation on the 20 Newsgroup dataset, where the classiﬁer is modeled by an afﬁne transformation, the cost function L is the cross-entrpy loss, and R ( w, λ ) is a strongly-convex regularizer. We set one l 2 -regularization hyperparameter for each weight in w , so that λ and w have the same size. For ESJ and HOZOG, we use GD with a learning rate of 100 and a momentum of 0 . 9 to perform the inner updates.

Original: In this section, we provide auxiliary lemmas that are used for proving the convergence results for the algorithms ESJ  and ESJ-S.
Revised: The outer learning rate is set to be 0 . We set the smoothing parameter ( µ in Algorithm 1) to be 0 . For AID-FP, AID-CG, and REVERSE we use the suggested hyperparameters in their implementations accompanying the paper Grazzi et al. It can be seen from Figure 4, our method ESJ slightly outperforms HOZOG and converges faster than the other AID and ITD based approaches.

Original: In the following proofs, we let L = max { L g , L f } and D be such that y ∗ ( x )
Revised: We note that the similar performance for ESJ and HOZOG can be explained by the fact that in HO, the hypergradient expession in eq. (6) contains only the second term (the ﬁrst term is zero), which is very close to the approximation in HOZOG method.

Original: ≤ D .
Revised: However, as we have seen in the other experiments (not for HO), ESJ is a more robust and stable bilevel optimizer than HOZOG, and it achieves good performance across many bilevel problems. E S UPPORTING T ECHNICAL L EMMAS In this section, we provide auxiliary lemmas that are used for proving the convergence results for the algorithms ESJ and ESJ-S. In the following proofs,

Original: First we recall that for any two matrices A ∈ R m × r and B ∈ R r × n , we have the following upperbound on the Frobenius norm of their product,
Revised: First we recall that for any two matrices A ∈ R m × r and B ∈ R r × n , we have the following upperbound on the Frobenius norm of their product,

Original: [Equation]
Revised: [Equation]

Original: The following lemma follows directly from the Lipschitz properties in Assumptions 2 and 3.
Revised: The following lemma follows directly from the Lipschitz properties in Assumptions 2 and 3.

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose that Assumptions 2 and 3 hold.
Revised: Suppose that Assumptions 2 and 3 hold.

Original: Then, the stochastic derivatives ∇ F ( x, y ; ξ ) , ∇ x ∇ y G ( x, y ; ξ ) , and ∇ 2 y G ( x, y ; ξ ) have bounded variances, i.e., for any ( x, y ) and ξ we have:
Revised: Then, the stochastic derivatives ∇ F ( x, y ; ξ ) , ∇ x ∇ y G ( x, y ; ξ ) , and ∇ 2 y G ( x, y ; ξ ) have bounded variances, i.e., for any ( x, y ) and ξ we have:

Original: • E ξ (cid:13)(cid:13) ∇ F ( x, y ; ξ ) − ∇ f ( x, y ) (cid:13)(cid:13) 2 ≤ M 2 ; • E ξ (cid:13)(cid:13) ∇ x ∇ y G ( x, y ; ξ ) − ∇ x ∇ y g
Revised: • E ξ (cid:13)(cid:13) ∇ F ( x, y ; ξ ) − ∇ f ( x, y ) (cid:13)(cid:13) 2 ≤ M 2 ; • E ξ (cid:13)(cid:13) ∇ x ∇ y G ( x, y ; ξ ) − ∇ x ∇ y g

Original: ( x, y ) (cid:13)(cid:13)
Revised: ( x, y ) (cid:13)(cid:13)

Original: 2 F ≤ L 2 ; • E ξ (cid:13)(cid:13) ∇ 2 y G ( x, y ; ξ ) − ∇ 2 y g ( x, y )
Revised: 2 F ≤ L 2 ; • E ξ (cid:13)(cid:13) ∇ 2 y G ( x, y ; ξ ) − ∇ 2 y g ( x, y )

Original: (cid:13)(cid:13) 2 F ≤ L 2 .
Revised: (cid:13)(cid:13) 2 F ≤ L 2 .

Original: Using Lemma 2.2 in Ghadimi & Wang (2018), the following lemma characterizes the Lipschitz property of the gradient of the total objective Φ( x ) =
Revised: Using Lemma 2.2 in Ghadimi & Wang (2018), the following lemma characterizes the Lipschitz property of the gradient of the total objective Φ( x ) =

Original: f ( x, y ∗ ( x )) .
Revised: f ( x, y ∗ ( x )) .

Original: Lemma 2.
Revised: Lemma 2.

Original: Suppose that Assumptions 1, 2, and 3 hold.
Revised: Suppose that Assumptions 1, 2, and 3 hold.

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: where the constant L Φ =
Revised: where the constant L Φ =

Original: L + 2 L 2 + τM 2 µ
Revised: L + 2 L 2 + τM 2 µ

Original: [Equation]
Revised: [Equation]

Original: We next provide some essential properties of the ES-based (i.e., zeroth-order) gradient oracle in eq.
Revised: We next provide some essential properties of the ES-based (i.e., zeroth-order) gradient oracle in eq.

Original: (4), due to Nesterov & Spokoiny (2017).
Revised: (4), due to Nesterov & Spokoiny (2017).

Original: Lemma 3.
Revised: Lemma 3.

Original: Let h : R n → R be a differentiable function with L -Lipsctitz gradient.
Revised: Let h : R n → R be a differentiable function with L -Lipsctitz gradient.

Original: Deﬁne its
Revised: Deﬁne its

Original: Gaussian smooth approximation h µ ( x ) =
Revised: Gaussian smooth approximation h µ ( x ) =

Original: E u
Revised: E u

Original: [ h ( x + µu )] , where µ > 0 and u ∈ R n is a standard
Revised: [ h ( x + µu )] , where µ > 0 and u ∈ R n is a standard

Original: Gaussian random vector.
Revised: Gaussian random vector.

Original: Then, h µ is differentiable and we have:
Revised: Then, h µ is differentiable and we have:

Original: • The gradient of h µ takes the form
Revised: • The gradient of h µ takes the form

Original: [Equation]
Revised: [Equation]

Original: • For any x ∈ R n ,
Revised: • For any x ∈ R n ,

Original: [Equation]
Revised: [Equation]

Original: • For any x ∈ R n ,
Revised: • For any x ∈ R n ,

Original: [Equation]
Revised: [Equation]

Original: Note the ﬁrst item in Lemma 3 implies that the oracle in eq.
Revised: Note the ﬁrst item in Lemma 3 implies that the oracle in eq.

Original: (4) is in indeed an unbiased estimator of the gradient of the smoothed function h µ .
Revised: (4) is in indeed an unbiased estimator of the gradient of the smoothed function h µ .

Original: Lemma 4.
Revised: Lemma 4.

Original: Suppose that Assumptions 1 and 2 hold.
Revised: Suppose that Assumptions 1 and 2 hold.

Original: The Jacobian J ∗ =
Revised: The Jacobian J ∗ =

Original: ∂y ∗ ( x ) ∂x has bounded norm:
Revised: ∂y ∗ ( x ) ∂x has bounded norm:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: From the ﬁrst order optimality condition of y ∗ ( x ) , we have ∇ y g ( x, y ∗ ( x ))
Revised: From the ﬁrst order optimality condition of y ∗ ( x ) , we have ∇ y g ( x, y ∗ ( x ))

Original: = 0 .
Revised: = 0 .

Original: Hence, the Implicit Function Theorem implies:
Revised: Hence, the Implicit Function Theorem implies:

Original: [Equation]
Revised: [Equation]

Original: Taking norms and applying eq.
Revised: Taking norms and applying eq.

Original: (15) together with Assumptions 1 and 2 yield the desired result
Revised: (16) together with Assumptions 1 and 2 yield the desired result

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma 5.
Revised: Lemma 5.

Original: Suppose that Assumptions 1 and 2 hold.
Revised: Suppose that Assumptions 1 and 2 hold.

Original: The Jacobian J N = ∂y Nk ∂x k has bounded norm:
Revised: The Jacobian J N = ∂y Nk ∂x k has bounded norm:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 5.
Revised: Proof of Lemma 5.

Original: The inner loop gradient descent updates writes
Revised: The inner loop gradient descent updates writes

Original: [Equation]
Revised: [Equation]

Original: Taking derivatives w.r.t.
Revised: Taking derivatives w.r.t.

Original: x k yields
Revised: x k yields

Original: [Equation]
Revised: [Equation]

Original: Telescoping over t from 1 to N yields
Revised: Telescoping over t from 1 to N yields

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Assumption 2 and ( ii ) applies the strong-convexity of function g ( x, · ) .
Revised: where ( i ) follows from Assumption 2 and ( ii ) applies the strong-convexity of function g ( x, · ) .

Original: This completes the proof.
Revised: This completes the proof.

Original: Lemma 6.
Revised: Lemma 6.

Original: Suppose that Assumptions 1 and 2 hold.
Revised: Suppose that Assumptions 1 and 2 hold.

Original: Then, the Jacobian in the stochastic algorithm
Revised: Then, the Jacobian in the stochastic algorithm

Original: ESJ-S J N = ∂Y
Revised: ESJ-S J N = ∂Y

Original: Nk ∂x k has bounded norm, as shown below.
Revised: Nk ∂x k has bounded norm, as shown below.

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 6.
Revised: Proof of Lemma 6.

Original: The proof follows similarly to Lemma 5.
Revised: The proof follows similarly to Lemma 5.

Original: D. P ROOF OF P ROPOSITION
Revised: E. P ROOF OF P ROPOSITION

Original: Proposition 1 (Re-stated) .
Revised: Proposition 1 (Re-stated) .

Original: Suppose that Assumptions 1 and 2 hold.
Revised: Suppose that Assumptions 1 and 2 hold.

Original: Deﬁne the constant
Revised: Deﬁne the constant

Original: [Equation]
Revised: [Equation]

Original: Then, the Jacobian J N ( x ) = ∂y N ( x ) ∂x is L J -Lipschitz with respect to x under the Frobenious norm:[Equation]
Revised: Then, the Jacobian J N ( x ) = ∂y N ( x ) ∂x is L J -Lipschitz with respect to x under the Frobenious norm:

Original: [Equation]
Revised: [Equation]

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: Using eq.
Revised: Using eq.

Original: (20), we have
Revised: (21), we have

Original: [Equation]
Revised: [Equation]

Original: x ∈
Revised: x ∈

Original: [Equation]
Revised: [Equation]

Original: Hence, for x 1 ∈ R p and x 2 ∈ R p , we have
Revised: Hence, for x 1 ∈ R p and x 2 ∈ R p , we have

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne A t ( x ) = (cid:81) N − 1 m =
Revised: where we deﬁne A t ( x ) = (cid:81) N − 1 m =

Original: t +1 (cid:0)
Revised: t +1 (cid:0)

Original: I − α ∇ 2 y g ( x, y m ( x )) (cid:1) and ( i ) follows from eq.
Revised: I − α ∇ 2 y g ( x, y m ( x )) (cid:1) and ( i ) follows from eq.

Original: Next we upper bound the quantity (cid:13)(cid:13)
Revised: Next we upper bound the quantity (cid:13)(cid:13)

Original: A t ( x 1 ) − A t ( x 2 ) (cid:13)(cid:13) , as shown below.
Revised: A t ( x 1 ) − A t ( x 2 ) (cid:13)(cid:13) , as shown below.

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from Lemma 5 and Assumptions 1 and
Revised: where the last inequality follows from Lemma 5 and Assumptions 1 and

Original: [Equation]
Revised: [Equation]

Original: Telescoping eq.
Revised: Telescoping eq.

Original: (25) over t yields
Revised: (26) over t yields

Original: [Equation]
Revised: [Equation]

Original: =(
Revised: =(

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Lemma 5 and Assumption 2.
Revised: where ( i ) follows from Lemma 5 and Assumption 2.

Original: Replacing eq.
Revised: Replacing eq.

Original: (26) in eq.
Revised: (27) in eq.

Original: (24) and using
Revised: (25) and using

Original: Assumption 2, we have
Revised: Assumption 2, we have

Original: [Equation]
Revised: [Equation]

Original: where we use (cid:80) N − 1 m =

Revised: where we use (cid:80) N − 1 m =

Original: mx m − 1 ≤ 1 α 2 µ 2 g in eq.
Revised: mx m − 1 ≤ 1 α 2 µ 2 g in eq.

Original: (27), which can be obtained by taking derivatives for the expression (cid:80) N − 1 m [-=
-]
Revised: (28), which can be obtained by taking derivatives for the expression (cid:80) N − 1 m [-=
-]=

Original: x m with respect to x .
Revised: x m with respect to x .

Original: Hence, rearranging and using the deﬁnition of L J in eq.
Revised: Hence, rearranging and using the deﬁnition of L J in eq.

Original: (22) ﬁnishes the proof.
Revised: (23) ﬁnishes the proof.

Original: Lemma 7.
Revised: Lemma 7.

Original: Suppose that Assumptions 1 and 2 hold.
Revised: Suppose that Assumptions 1 and 2 hold.

Original: Deﬁne the constant
Revised: Deﬁne the constant

Original: [Equation]
Revised: [Equation]

Original: Then, the Jacobian J N ( x ) = ∂Y N ( x ; · ) ∂x is L J -Lipschitz with respect to x under the Frobenius norm:
Revised: Then, the Jacobian J N ( x ) = ∂Y N ( x ; · ) ∂x is L J -Lipschitz with respect to x under the Frobenius norm:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 7.
Revised: Proof of Lemma 7.

Original: The proof follows similarly to that for Proposition 1.
Revised: The proof follows similarly to that for Proposition 1.

Original: E P ROOFS FOR D ETERMINISTIC B ILEVEL O PTIMIZATION
Revised: F P ROOFS FOR D ETERMINISTIC B ILEVEL O PTIMIZATION

Original: For notation convenience, we deﬁne the following quantities:
Revised: For notation convenience, we deﬁne the following quantities:

Original: [Equation]
Revised: [Equation]

Original: where u j ∈ R p , j = 1 , . .
Revised: where u j ∈ R p , j = 1 , . .

Original: , Q are standard Gaussian vectors.
Revised: , Q are standard Gaussian vectors.

Original: Let y Ni,µ ( x k ) be the Gaussian smooth approximation of y Ni ( x k ) .
Revised: Let y Ni,µ ( x k ) be the Gaussian smooth approximation of y Ni ( x k ) .

Original: We collect y Ni,µ ( x k ) for i = 1 , . . .
Revised: We collect y Ni,µ ( x k ) for i = 1 , . . .

Original: , d together as a vector y Nµ ( x k ) , which is the Gaussian approximation of the vector y N ( x k ) .
Revised: , d together as a vector y Nµ ( x k ) , which is the Gaussian approximation of the vector y N ( x k ) .

Original: If µ > 0 , y Nµ ( x k ) is differentiable and we let
Revised: If µ > 0 , y Nµ ( x k ) is differentiable and we let

Original: J µ be the Jocobian given by
Revised: J µ be the Jocobian given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We approximate ∂y Nk ∂x k using the average ES estimator given by ˆ J N = 1 Q (cid:80) Q j =1 ˆ J N,j .
Revised: We approximate ∂y Nk ∂x k using the average ES estimator given by ˆ J N = 1 Q (cid:80) Q j =1 ˆ J N,j .

Original: The hypergradient is then approximated as
Revised: The hypergradient is then approximated as

Original: [Equation]
Revised: [Equation]

Original: Let δ j = y N ( x k + µu j ) − y N ( x k ) µ , and let δ i,j be the i -th component of δ j .
Revised: Let δ j = y N ( x k + µu j ) − y N ( x k ) µ , and let δ i,j be the i -th component of δ j .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: Using eq.
Revised: Using eq.

Original: (30) and eq.
Revised: (31) and eq.

Original: [Equation]
Revised: [Equation]

Original: the estimator for the hypergradient can thus be computed as
Revised: the estimator for the hypergradient can thus be computed as

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF P ROPOSITION
Revised: F. P ROOF OF P ROPOSITION

Original: Proposition 2 (Re-stated) .
Revised: Proposition 2 (Re-stated) .

Original: Suppose that Assumptions 1,
Revised: Suppose that Assumptions 1,

Original: [Equation]
Revised: [Equation]

Original: and 3 hold.
Revised: and 3 hold.

Original: Then, the expected estimation error can be upper-bounded as follows:
Revised: Then, the expected estimation error can be upper-bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: where the expectation E [ · ] is conditioned on x k and y Nk .
Revised: where the expectation E [ · ] is conditioned on x k and y Nk .

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: Based on the deﬁnitions of ∇ Φ( x k ) and (cid:98) ∇ Φ( x k ) and conditioning on x k and y Nk
Revised: Based on the deﬁnitions of ∇ Φ( x k ) and (cid:98) ∇ Φ( x k ) and conditioning on x k and y Nk

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Lemma 4 and Assumption 3, and ( i ) and ( ii ) also use the following result for full GD (when applied to a strongly-convex function).
Revised: where ( i ) follows from Lemma 4 and Assumption 3, and ( i ) and ( ii ) also use the following result for full GD (when applied to a strongly-convex function).

Original: [Equation]
Revised: [Equation]

Original: Next, we upper-bound the last term E (cid:13)(cid:13)
Revised: Next, we upper-bound the last term E (cid:13)(cid:13)

Original: ˆ J N − J ∗ (cid:13)(cid:13) 2 F at the last line of eq.
Revised: ˆ J N − J ∗ (cid:13)(cid:13) 2 F at the last line of eq.

Original: First note that
Revised: First note that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ( i ) follows by applying Lemma 3 to the components of vector y N ( x k ) which have Lipschitz gradients by Proposition 1.
Revised: where ( i ) follows by applying Lemma 3 to the components of vector y N ( x k ) which have Lipschitz gradients by Proposition 1.

Original: Then, noting that (cid:13)(cid:13) ∇ y Ni,µ (cid:13)(cid:13) 2 ≤ 2 (cid:13)(cid:13) ∇ y Ni (cid:13)(cid:13) 2 + 12 µ 2 L 2 J ( p +3) 3 and replacing in eq.
Revised: Then, noting that (cid:13)(cid:13) ∇ y Ni,µ (cid:13)(cid:13) 2 ≤ 2 (cid:13)(cid:13) ∇ y Ni (cid:13)(cid:13) 2 + 12 µ 2 L 2 J ( p +3) 3 and replacing in eq.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Lemma 5 and P 4 is a polynomial of degree 4 in p .
Revised: where ( i ) follows from Lemma 5 and P 4 is a polynomial of degree 4 in p .

Original: Combining eq.
Revised: Combining eq.

Original: (35) and eq.
Revised: (36) and eq.

Original: (37) yields
Revised: (38) yields

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We next upper-bound the second term at the right hand side of eq.
Revised: We next upper-bound the second term at the right hand side of eq.

Original: (34), which can be upper-bounded using eq.
Revised: (35), which can be upper-bounded using eq.

Original: (41) in Ji et al.
Revised: (41) in Ji et al.

Original: (2021), as shown below.
Revised: (2021), as shown below.

Original: [Equation]
Revised: [Equation]

Original: We ﬁnally upper-bound the last term at the right hand side of eq.
Revised: We ﬁnally upper-bound the last term at the right hand side of eq.

Original: (34) using Lemma 3.
Revised: (35) using Lemma 3.

Original: [Equation]
Revised: [Equation]

Original: Substituting eq.
Revised: Substituting eq.

Original: (39), eq. (40) and eq.
Revised: (40), eq. (41) and eq.

Original: (41) into eq.
Revised: (42) into eq.

Original: (34) yields
Revised: (35) yields

Original: [Equation]
Revised: [Equation]

Original: Finally [Equation] the bound for the expected estimation error in eq.
Revised: Finally, the bound for the expected estimation error in eq.

Original: (33) becomes
Revised: (34) becomes

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof

Revised: [Equation]

Original: E. P ROOF OF T HEOREM
Revised: F. P ROOF OF T HEOREM

Original: Theorem 1 (Re-stated) .
Revised: Theorem 1 (Re-stated) .

Original: Suppose that Assumptions 1, 2, and 3 hold.
Revised: Suppose that Assumptions 1, 2, and 3 hold.

Original: Choose the inner- and outerloop stepsizes respectively as α ≤ 1 L and β = 14 L Φ , where L Φ =
Revised: Choose the inner- and outerloop stepsizes respectively as α ≤ 1 L and β = 14 L Φ , where L Φ =

Original: L + 2 L 2 + τM 2 µ g + ρLM + L 3 + τML µ 2 g + ρL
Revised: L + 2 L 2 + τM 2 µ g + ρLM + L 3 + τML µ 2 g + ρL

Original: 2 M µ 3 g .
Revised: 2 M µ 3 g .

Original: Then, the iterates x k for k = 0 , ..., K − 1 of ESJ in Algorithm 1 satisfy:
Revised: Then, the iterates x k for k = 0 , ..., K − 1 of ESJ in Algorithm 1 satisfy:

Original: [Equation]
Revised: [Equation]

Original: with Φ ∗ = inf x Φ( x ) and D is the upper-bound established in Proposition 2 and is given by
Revised: with Φ ∗ = inf x Φ( x ) and D is the upper-bound established in Proposition 2 and is given by

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Using the Lipschitzness of function Φ( x k ) , we have Φ( x k +1 ) ≤ Φ(
Revised: Using the Lipschitzness of function Φ( x k ) , we have Φ( x k +1 ) ≤ Φ(

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:104)∇
Revised: ∇

Original: [Equation]
Revised: [Equation]

Original: (cid:98) ∇
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x k ) (cid:105) 
Revised: ) (cid:105) + L φ

Original: + L φ 2 β 2 (cid:13)(cid:13)(cid:98) ∇ Φ( x k ) (cid:13)(cid:13)≤ Φ( x k ) − β (cid:104)∇ Φ( x k ) , (cid:98) ∇ Φ( x k ) − ∇ Φ( x k ) (cid:105) − β (cid:13)(cid:13) ∇ Φ( x k ) (cid:13)(cid:13)+ L φ β 2 (cid:16)(cid:13)(cid:13) ∇ Φ( x k ) (cid:13)(cid:13) 

Revised: [Equation] β 2 (cid:13)(cid:13)(cid:98) ∇ Φ( x k ) (cid:13)(cid:13)≤ Φ( x k ) − β (cid:104)∇ Φ( x k ) , (cid:98) ∇ Φ( x k ) − ∇ Φ( x k ) (cid:105) − β (cid:13)(cid:13) ∇ Φ( x k ) ( cid:13)(cid:13)+ L φ β 2 (cid:16)(cid:13)(cid:13) ∇ Φ( x k ) (cid:13)(cid:13)

Original: + (cid:13)(cid:13)(cid:98)
Revised: + (cid:13)(cid:13)(cid:98)

Original: [Equation]
Revised: [Equation]

Original: Let E k [ · ] =
Revised: Let E k [ · ] =

Original: E u k
Revised: E u k

Original: [Equation]
Revised: [Equation]

Original: 1: Q [ ·| x k [Equation] y N k ] be the expectation over the Gaussian vectors u k, 1 , . . .
Revised: 1: Q [ ·| x k , y N k ] be the expectation over the Gaussian vectors u k, 1 , . . .

Original: , u k,Q conditioned on x k and y Nk .
Revised: , u k,Q conditioned on x k and y Nk .

Original: Applying the expectation E k [ · ] to eq.
Revised: Applying the expectation E k [ · ] to eq.

Original: (46) yields
Revised: (47) yields

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where D represent the upper-bound established for the estimation error in Proposition 2.
Revised: where D represent the upper-bound established for the estimation error in Proposition 2.

Original: Now taking total expectation over U k
Revised: Now taking total expectation over U k

Original: = { u 1 , 1: Q , . .
Revised: = { u 1 , 1: Q , . .

Original: , u k, 1: Q } , we have
Revised: , u k, 1: Q } , we have

Original: [Equation]
Revised: [Equation]

Original: where E k = E U k − 1 Φ( x k ) .
Revised: where E k = E U k − 1 Φ( x k ) .

Original: Summing up the inequalities in eq.
Revised: Summing up the inequalities in eq.

Original: (48) for k = 0 , . . .
Revised: (49) for k = 0 , . . .

Original: , K − 1 yields
Revised: , K − 1 yields

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Setting β
Revised: Setting β

Original: =
Revised: =

Original: 14 L φ , denoting by Φ ∗ =
Revised: 14 L φ , denoting by Φ ∗ =

Original: inf x Φ( x )
Revised: inf x Φ( x )

Original: [Equation]
Revised: [Equation]

Original: and rearranging eq.
Revised: and rearranging eq.

Original: (49), we have
Revised: (50), we have

Original: [Equation]
Revised: [Equation]

Original: Hence, the proof is ﬁnished.
Revised: Hence, the proof is ﬁnished.

Original: F P ROOFS FOR S TOCHASTIC B ILEVEL O PTIMIZATION
Revised: G P ROOFS FOR S TOCHASTIC B ILEVEL O PTIMIZATION

Original: Deﬁne the following quantities
Revised: Deﬁne the following quantities

Original: [Equation]
Revised: [Equation]

Original: where u j ∈ R p , j = 1 , . .
Revised: where u j ∈ R p , j = 1 , . .

Original: , Q are standard Gaussian vectors and Y Nk is the output of SGD obtained with the minibatches {S 0 , ..., S N − 1 } .
Revised: , Q are standard Gaussian vectors and Y Nk is the output of SGD obtained with the minibatches {S 0 , ..., S N − 1 } .

Original: Conditioning on x k and Y Nk and taking expectation over u j yields
Revised: Conditioning on x k and Y Nk and taking expectation over u j yields

Original: [Equation]
Revised: [Equation]

Original: where Y Ni,µ ( x k ; S ) is the i -th component of vector Y Nµ ( x k ; S ) , which is the entry-wise Gaussian smooth approximation of vector Y N ( x k ; S ) .
Revised: where Y Ni,µ ( x k ; S ) is the i -th component of vector Y Nµ ( x k ; S ) , which is the entry-wise Gaussian smooth approximation of vector Y N ( x k ; S ) .

Original: Let E k [ · ] =
Revised: Let E k [ · ] =

Original: E [ ·| x k , Y Nk ] =
Revised: E [ ·| x k , Y Nk ] =

Original: E D F ,u 1: q be the expectation over the Gaussian vectors and the sample minibatch D F conditioned on x k and Y Nk .
Revised: E D F ,u 1: q be the expectation over the Gaussian vectors and the sample minibatch D F conditioned on x k and Y Nk .

Original: F. P ROOF OF P ROPOSITION
Revised: G. P ROOF OF P ROPOSITION

Original: Proposition 3 (Re-stated) .
Revised: Proposition 3 (Re-stated) .

Original: Suppose that Assumptions 1, 2, and 4 hold.
Revised: Suppose that Assumptions 1, 2, and 4 hold.

Original: Choose the inner-loop stepsize as α = 2 L + µ g .
Revised: Choose the inner-loop stepsize as α = 2 L + µ g .

Original: Deﬁne the constants
Revised: Deﬁne the constants

Original: [Equation]
Revised: [Equation]

Original: where γ is such that γ ≥ L + µ g µ 2 g .
Revised: where γ is such that γ ≥ L + µ g µ 2 g .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Proof of Proposition 3.
Revised: Proof of Proposition 3.

Original: Based on the SGD updates
Revised: Based on the SGD updates

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Taking the derivatives w.r
Revised: Taking the derivatives w.r

Original: [Equation]
Revised: [Equation]

Original: t. x k yields
Revised: t. x k yields

Original: [Equation]
Revised: [Equation]

Original: Hence, using the triangle inequality
Revised: Hence, using the triangle inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Assumption 1.
Revised: where ( i ) follows from Assumption 1.

Original: We then further have
Revised: We then further have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: The terms P 1 , P 2 and P 3 in the above inequality can be transformed as follows using the Peter-Paul version of Young’s inequality.
Revised: The terms P 1 , P 2 and P 3 in the above inequality can be transformed as follows using the Peter-Paul version of Young’s inequality.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Let E t − 1 [ · ] = E [ ·| x k , Y t − 1 k ] .
Revised: Let E t − 1 [ · ] = E [ ·| x k , Y t − 1 k ] .

Original: Conditioning on x k and Y t − 1 k and taking expectations yield
Revised: Conditioning on x k and Y t − 1 k and taking expectations yield

Original: [Equation]
Revised: [Equation]

Original: where C γ , C xy and C y are deﬁned as follows
Revised: where C γ , C xy and C y are deﬁned as follows

Original: [Equation]
Revised: [Equation]

Original: Conditioning on x k and Y t − 1 k , we have
Revised: Conditioning on x k and Y t − 1 k , we have

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from Lemma 1 and Assumption 2.
Revised: where ( i ) follows from Lemma 1 and Assumption 2.

Original: Similarly we can derive
Revised: Similarly we can derive

Original: [Equation]
Revised: [Equation]

Original: Combining eq.
Revised: Combining eq.

Original: (52), eq. (53),  and eq.
Revised: (53), eq. (54), and eq.

Original: (54) we obtain
Revised: (55) we obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Unconditioning on x k and Y t − 1 k and taking total expectations of eq.
Revised: Unconditioning on x k and Y t − 1 k and taking total expectations of eq.

Original: (55) yield
Revised: (56) yield

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows from the analysis of SGD for a strongly-convex function.
Revised: where ( i ) follows from the analysis of SGD for a strongly-convex function.

Original: Let Γ = 2( τ 2 C xy + ρ 2 C y )
Revised: Let Γ = 2( τ 2 C xy + ρ 2 C y )

Original: σ 2 µ g LS + 2 L 2 S ( C xy + C y ) and λ = 2( τ 2 C xy + ρ 2 C y )
Revised: σ 2 µ g LS + 2 L 2 S ( C xy + C y ) and λ = 2( τ 2 C xy + ρ 2 C y )

Original: D 2 .
Revised: D 2 .

Original: Then, we have
Revised: Then, we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Telescoping eq.
Revised: Telescoping eq.

Original: (56) over t from N down to 1 yields
Revised: (57) over t from N down to 1 yields

Original: [Equation]
Revised: [Equation]

Original: which, in conjunction with (cid:16)
Revised: which, in conjunction with (cid:16)

Original: [Equation]
Revised: [Equation]

Original: such that C γ ≤ 1 − αµ g , yields
Revised: such that C γ ≤ 1 − αµ g , yields

Original: [Equation]
Revised: [Equation]

Original: The proof is then complete
Revised: The proof is then complete

Original: [Equation]
Revised: [Equation]

Original: Lemma 8.
Revised: Lemma 8.

Original: Suppose that Assumptions 1, 2,
Revised: Suppose that Assumptions 1, 2,

Original: [Equation]
Revised: [Equation]

Original: and 4 hold
Revised: and 4 hold

Original: [Equation]
Revised: [Equation]

Original: Set the inner-loop stepsize as α = 2 L + µ g .
Revised: Set the inner-loop stepsize as α = 2 L + µ g .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the expectation E k [ · ] is conditioned on x k and Y Nk .
Revised: where the expectation E k [ · ] is conditioned on x k and Y Nk .

Original: Proof of Lemma 8.
Revised: Proof of Lemma 8.

Original: Conditioning on x k and Y Nk , we have
Revised: Conditioning on x k and Y Nk , we have

Original: [Equation]
Revised: [Equation]

Original: Recall ∇ Φ( x k ) =
Revised: Recall ∇ Φ( x k ) =

Original: ∇ x f ( x k , y ∗ k )
Revised: ∇ x f ( x k , y ∗ k )

Original: + J (cid:62)∗ ∇ y f
Revised: + J (cid:62)∗ ∇ y f

Original: ( x k , y ∗ k ) .
Revised: ( x k , y ∗ k ) .

Original: Thus, we have (cid:13)(cid:13) E k (cid:98) ∇ Φ( x k ) −∇
Revised: Thus, we have (cid:13)(cid:13) E k (cid:98) ∇ Φ( x k ) −∇

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)≤
Revised: (cid:13)(cid:13)≤

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Nk
Revised: Nk

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: Y Nk − y
Revised: Y Nk − y

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13) 2 + 4 (cid:13)(cid:13)
Revised: (cid:13)(cid:13) 2 + 4 (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:62)
Revised: (cid:62)

Original: [Equation]
Revised: [Equation]

Original: y f
Revised: y f

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: , Y Nk )
Revised: , Y Nk )

Original: [Equation]
Revised: [Equation]

Original: J (cid:62)∗
Revised: J (cid:62)∗

Original: [Equation]
Revised: [Equation]

Original: y f
Revised: y f

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: Y Nk ) (cid:13)(cid:13)+ 4 (cid:13)(cid:13)
Revised: Y Nk ) (cid:13)(cid:13)+ 4 (cid:13)(cid:13)

Original: J (cid:62)∗ ∇ y f
Revised: J (cid:62)∗ ∇ y f

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: , Y Nk ) − J (cid:62)∗ ∇ y f
Revised: , Y Nk ) − J (cid:62)∗ ∇ y f

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: , y
Revised: , y

Original: [Equation]
Revised: [Equation]

Original: ) (cid:13)(cid:13)( i ) ≤ 2 L
Revised: ) (cid:13)(cid:13)( i ) ≤ 2 L

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: Y Nk − y
Revised: Y Nk − y

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: J
Revised: J

Original: [Equation]
Revised: [Equation]

Original: − J
Revised: − J

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: F +
Revised: F +

Original: [Equation]
Revised: [Equation]

Original: L 2 µg (cid:13)(cid:13) ∇ y f
Revised: L 2 µg (cid:13)(cid:13) ∇ y f

Original: ( x k , Y Nk ) − ∇ y f
Revised: ( x k , Y Nk ) − ∇ y f

Original: [Equation]
Revised: [Equation]

Original: where ( i ) applies Lemma 4 and Assumption 3.
Revised: where ( i ) applies Lemma 4 and Assumption 3.

Original: Taking expectation of the above inequality yields
Revised: Taking expectation of the above inequality yields

Original: [Equation]
Revised: [Equation]

Original: Using the fact that Y Ni ( x k ; · ) has Lipschitz gradient (Proposition 1) and Lemma
Revised: Using the fact that Y Ni ( x k ; · ) has Lipschitz gradient (Proposition 1) and Lemma

Original: [Equation]
Revised: [Equation]

Original: the last term at the right hand side of eq.
Revised: the last term at the right hand side of eq.

Original: (60) can be directly upper-bounded as
Revised: (61) can be directly upper-bounded as

Original: [Equation]
Revised: [Equation]

Original: where L J is the Lipschitz constant of the Jacobian J N (and also of its rows ∇ Y Ni ( x k ; S ) ) as deﬁned in Proposition 1.
Revised: where L J is the Lipschitz constant of the Jacobian J N (and also of its rows ∇ Y Ni ( x k ; S ) ) as deﬁned in Proposition 1.

Original: Combining eq.
Revised: Combining eq.

Original: (60), eq.
Revised: (61), eq.

Original: (61), Proposition 3, and SGD analysis (as in eq.
Revised: (62), Proposition 3, and SGD analysis (as in eq.

Original: (60) in Ji et al.
Revised: (60) in Ji et al.

Original: (2021)) yields
Revised: (2021)) yields

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This ﬁnishes the proof.
Revised: This ﬁnishes the proof.

Original: F. P ROOF OF P ROPOSITION
Revised: G. P ROOF OF P ROPOSITION

Original: Proposition 4 (Re-stated) .
Revised: Proposition 4 (Re-stated) .

Original: Suppose that Assumptions 1, 2, 3, and 4 hold.
Revised: Suppose that Assumptions 1, 2, 3, and 4 hold.

Original: Set the inner-loop stepsize as α = 2 L + µ g .
Revised: Set the inner-loop stepsize as α = 2 L + µ g .

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: 63) where ∆ = 8 M 2 (cid:16)(cid:16) 1 + 1 D f (cid:17) 4 p +15 Q + 1 D f (cid:17)
Revised: 64) where ∆ = 8 M 2 (cid:16)(cid:16) 1 + 1 D f (cid:17) 4 p +15 Q + 1 D f (cid:17)

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: dL
Revised: dL

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: and B 1 respresents the upper bound established in Lemma 8.
Revised: and B 1 respresents the upper bound established in Lemma 8.

Original: Proof of Proposition 4.
Revised: Proof of Proposition 4.

Original: We have, conditioning on x k and Y Nk
Revised: We have, conditioning on x k and Y Nk

Original: [Equation]
Revised: [Equation]

Original: Our next step is to upper-bound the ﬁrst term in eq.
Revised: Our next step is to upper-bound the ﬁrst term in eq.

Original: [Equation]
Revised: [Equation]

Original: where the last two steps follows from Lemma 1.
Revised: where the last two steps follows from Lemma 1.

Original: Next, we upper-bound the term E k (cid:13)(cid:13)
Revised: Next, we upper-bound the term E k (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Recall that for a function h with L -Lipschitz gradient, we have
Revised: Recall that for a function h with L -Lipschitz gradient, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, applying eq.
Revised: Then, applying eq.

Original: (67) to function Y N i ( · ; S ) yields
Revised: (68) to function Y N i ( · ; S ) yields

Original: [Equation]
Revised: [Equation]

Original: Hence, eq. (66) becomes
Revised: Hence, eq. (67) becomes

Original: [Equation]
Revised: [Equation]

Original: where P 4 is a polynomial of degree 4 in p .
Revised: where P 4 is a polynomial of degree 4 in p .

Original: Combining eq.
Revised: Combining eq.

Original: (65), eq.
Revised: (66), eq.

Original: (68) and Lemma 3 yields
Revised: (69) and Lemma 3 yields

Original: [Equation]
Revised: [Equation]

Original: where ∆
Revised: where ∆

Original: [Equation]
Revised: [Equation]

Original: Taking total expectations of both eq.
Revised: Taking total expectations of both eq.

Original: (64) and eq.
Revised: (65) and eq.

Original:  and combining
Revised: (70) and combining

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where B 1 representing the upper-bound established in eq
Revised: where B 1 representing the upper-bound established in eq

Original: [Equation]
Revised: [Equation]

Original: The proof is then complete.
Revised: The proof is then complete.

Original: F. P ROOF OF T HEOREM
Revised: G. P ROOF OF T HEOREM

Original: Theorem 2 (Re-stated) .
Revised: Theorem 2 (Re-stated) .

Original: Suppose that Assumptions 1, 2, 3, and 4 hold.
Revised: Suppose that Assumptions 1, 2, 3, and 4 hold.

Original: Set the inner- and outer-loop stepsizes respectivelly as α = 2 L + µ g and β = 14 L Φ , where L = max { L f , L g } and the constant L Φ is deﬁned as in Theorem 1.
Revised: Set the inner- and outer-loop stepsizes respectivelly as α = 2 L + µ g and β = 14 L Φ , where L = max { L f , L g } and the constant L Φ is deﬁned as in Theorem 1.

Original: Then, the iterates x k , k = 0 , ..., K − 1 of the ESJ-S algorithm satisfy
Revised: Then, the iterates x k , k = 0 , ..., K − 1 of the ESJ-S algorithm satisfy

Original: [Equation]
Revised: [Equation]

Original: where ∆ and B 1 have the following forms
Revised: where ∆ and B 1 have the following forms

Original: [Equation]
Revised: [Equation]

Original: and the constants Γ , λ , and C γ are deﬁned in Proposition
Revised: and the constants Γ , λ , and C γ are deﬁned in Proposition

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Using the Lipschitzness of function Φ( x k )
Revised: Using the Lipschitzness of function Φ( x k )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence, taking expectation over the above inequality yields
Revised: Hence, taking expectation over the above inequality yields

Original: [Equation]
Revised: [Equation]

Original: Also
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: which, in conjunction with eq.
Revised: which, in conjunction with eq.

Original: [Equation]
Revised: [Equation]

Original: yields
Revised: yields

Original: [Equation]
Revised: [Equation]

Original: Setting β
Revised: Setting β

Original: =
Revised: =

Original: 14 L Φ and using the bounds established in Lemma 8 and Proposition
Revised: 14 L Φ and using the bounds established in Lemma 8 and Proposition

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Summing up the above inequality over k from k = 0 to k = K − 1 yields
Revised: Summing up the above inequality over k from k = 0 to k = K − 1 yields

Original: [Equation]
Revised: [Equation]

Original: Rearranging the above inequality yields
Revised: Rearranging the above inequality yields

Original: [Equation]
Revised: [Equation]

Original: The proof is then complete
Revised: The proof is then complete

Original: [Equation]
Revised: [Equation]