A MORTIZED I MPLICIT D IFFERENTIATION FOR S TOCHASTIC B ILEVEL O PTIMIZATION
A BSTRACT
We study a class of algorithms for solving bilevel optimization problems in both stochastic and deterministic settings when the inner-level objective is strongly convex.
Speciﬁcally, we consider algorithms based on inexact implicit differentiation and we exploit a warm-start strategy to amortize the estimation of the exact gradient.
We then introduce a uniﬁed theoretical framework inspired by the study of singularly perturbed systems (Habets, 1974) to analyze such amortized algorithms.
By using this framework, our analysis shows these algorithms to match the computational complexity of oracle methods that have access to an unbiased estimate of the gradient, thus outperforming many existing results for bilevel optimization.
We illustrate these ﬁndings on synthetic experiments and demonstrate the efﬁciency of these algorithms on hyper-parameter optimization experiments involving several thousands of variables.
I NTRODUCTION
Bilevel optimization refers to a class of algorithms for solving problems with a hierarchical structure involving two levels: an inner and an outer level.
The inner -level problem seeks a solution y (cid:63) ( x ) minimizing a cost g ( x y ) over a set Y given a ﬁxed outer variable x in a set X .
The outer -level problem minimizes an objective of the form L ( x )=
f
( x, y (cid:63) ( x )) over X for some upper -level cost f .
When the solution y (cid:63) ( x ) is unique, the bilevel optimization problem takes the following form:
[Equation]
that
[Equation]
First introduced in the ﬁeld of economic game theory by Stackelberg (1934) and long studied in optimization (Ye and Zhu, 1995; Ye and Ye, 1997; Ye et al., 1997), this problem has recently received increasing attention in the machine learning community (Blondel et al., 2021;
Liu et al., 2021; Shaban et al., 2019).
Indeed, many machine learning applications can be reduced to (1) including hyper-parameter optimization (Feurer and Hutter, 2019), meta-learning (Bertinetto et al., 2018), reinforcement learning (Hong et al., 2020b; Liu et al., 2021) or dictionary learning (Mairal et al., 2011; Lecouat et al., 2020a;b).
The hierarchical nature of (1) introduces additional challenges compared to standard optimization problems, such as ﬁnding a suitable trade-off between the computational budget for approximating the inner and outer level problems (Ghadimi and Wang, 2018;
Dempe and Zemkoho, 2020).
These considerations are exacerbated in machine learning applications, where the costs f and g often come as an average of functions over a large or inﬁnite number of data points (Franceschi et al., 2018).
All these challenges highlight the need for methods that are able to control the computational costs inherent to (1) while dealing with the large-scale setting encountered in machine learning.
Gradient-based bilevel optimization methods appear to be viable approaches for solving (1) in largescale settings (Lorraine et al., 2020).
They can be divided into two categories: Iterative differentiation (ITD) and Approximate implicit differentiation (AID).
ITD approaches approximate the map y (cid:63) ( x ) by a differentiable optimization
algorithm A ( x ) viewed as a function of x .
The resulting surrogate loss ˜ L ( x ) = f
( x, A ( x )) is optimized instead of L ( x ) using reverse-mode automatic differentiation (see Baydin et al., 2018).
AID approaches (Pedregosa, 2016) rely on an expression of the gradient ∇L resulting from the implicit function theorem (Lang, 2012, Theorem 5.9).
Unlike ITD, AID avoids differentiating the algorithm approximating y (cid:63) ( x ) and, instead, approximately solves a linear system using only Hessian and Jacobian-vector products to estimate the gradient ∇L (Rajeswaran et al., 2019).
These methods can also rely on stochastic approximation to increase scalability (Franceschi et al., 2018; Grazzi et al., 2020; 2021).
In the context of machine-learning, Ghadimi and Wang (2018) provided one of the ﬁrst comprehensive studies of the computational complexity for a class of bilevel algorithms based on AID approaches.
Subsequently, Hong et al.
(2020b)
; Ji et al.
; Ji and Liang (2021); Yang et al.
(2021) proposed different algorithms for solving (1) and obtained improved overall complexity by achieving a better trade-off between the cost of the inner and outer level problems.
Still, the question of whether these complexities can be improved by better exploiting the structure of (1) through heuristics such as warm-start remains open (Grazzi et al., 2020).
Moreover, these studies proposed separate analysis of their algorithms depending on the convexity of the loss L and whether a stochastic or deterministic setting is considered.
This points out to a lack of uniﬁed and systematic theoretical framework for analyzing bilevel problems, which is what the present work addresses.
We consider the Am ortized I mplicit G radient O ptimization (AmIGO) algorithm, a bilevel optimization algorithm based on Approximate Implicit Differentiation (AID) approaches that exploits a warm-start strategy when estimating the gradient of L .
We then propose a uniﬁed theoretical framework for analyzing the convergence of AmIGO when the inner-level problem is strongly convex in both stochastic and deterministic settings.
The proposed framework is inspired from the early work of Habets (1974) on singularly perturbed systems and analyzes the effect of warm start by viewing the iterates of AmIGO algorithm as a dynamical system.
The evolution of such system is described by a total energy function which allows to recover the convergence rates of unbiased oracle methods which have access to an unbiased estimate of ∇L (c.f. Table 1).
To the best of our knowledge, this is the ﬁrst time a bilevel optimization algorithm based on a warm-start strategy provably recovers the rates of unbiased oracle methods across a wide range of settings including the stochastic ones.
R ELATED WORK
Singularly perturbed systems (SPS) are continuous-time deterministic dynamical systems of coupled variables ( x ( t ) , y ( t )) with two time-scales where y ( t ) evolves much faster than x ( t ) .
As such, they exhibit a hierarchical structure similar to (1).
The early work of Habets (1974); Saberi and
Khalil (1984) provided convergence rates for SPS towards equilibria by studying the evolution of a single scalar energy function summarizing these systems.
The present work takes inspiration from these works to analyze the convergence of AmIGO which involves three time-scales.
Two time-scale Stochastic Approximation (TTSA) can be viewed as a discrete-time stochastic version of SPS.
(Kaledin et al., 2020) showed that TTSA achieves a ﬁnite-time complexity of O (cid:0) (cid:15) − 1 (cid:1) for linear systems while Doan (2020) obtained a complexity of O (cid:0) (cid:15) − 3 / 2 (cid:1) for general non-linear systems by extending the analysis for SPS.
Hong et al.
(2020b) further adapted the non-linear TTSA for solving (1).
In the present work, we obtain faster rates by taking into account the dynamics of a third variable z k appearing in AmIGO, thus resulting in a three time-scale dynamics.
Warm-start in bilevel optimization.
Ji et al.
; Ji and Liang (2021) used a warm-start for the inner-level algorithm to obtain an improved computational complexity over algorithms without warm-start.
In the deterministic non-convex setting, Ji et al.
(2021) used a warm-start strategy when solving the linear system appearing in AID approaches to obtain improved convergence rates.
However, it remained open whether using a warm-start when solving both inner-level problem and linear system arising in AID approaches can yield faster algorithms in the more challenging stochastic setting (Grazzi et al., 2020).
In the present work, we provide a positive answer to this question.
A MORTIZED I MPLICIT G RADIENT O PTIMIZATION 3. G ENERAL SETTING AND MAIN ASSUMPTIONS
Notations.
In all what follows, X and Y are Euclidean spaces.
For a differentiable function h ( x, y ) :
X × Y → R , we denote by ∇ h its gradient w.r.t.
( x, y ) , by ∂ x h and ∂ y h its partial derivatives w.r.t.
x and y and by ∂ xy h
and ∂ yy
h
the partial derivatives of ∂ y h w.r.t x and y , respectively.
[Table]
an (cid:15) -accurate solution as measured by E [ L ( x k ) −L (cid:63) ] ∧ 2 − 1 µ E (cid:104) (cid:107) x k − x (cid:63) (cid:107) 2 (cid:105) when L is µ -strongly-convex (SC) and 1 k (cid:80) ki
=1 E (cid:104) (cid:107)∇L ( x i ) (cid:107) 2 (cid:105) when L is non-convex (NC) .
The settings (D) and (S) stand for the deterministic and stochastic settings.
The cost corresponds to the total number of gradients, Jacobian and Hessian-vector products used by the algorithm.
κ L and κ g are the conditioning numbers of L and g whenever applicable.
The dependence on κ L and κ g for
TTSA and AccBio are derived in Proposition 11 of Appendix A.4.
The rate of MRBO/VRBO is obtained under the additional mean-squared smoothness assumption (Arjevani et al., 2019).
To ensure that (1) is well-deﬁned, we consider the setting where the inner-level problem is strongly convex so that the solution y (cid:63) ( x ) is unique as stated by the following assumption:
Assumption 1.
For any x ∈ X , the function y
(cid:55)→ g ( x, y ) is L g -smooth and µ g -strongly convex.
Assumption 1 holds in the context of hyper-parameter selection when the inner-level is a kernel regression problem (Franceschi et al., 2018), or when the variable y represents the last linear layer of a neural network as in many meta-learning tasks (Ji et al., 2021).
Under Assumption 1 and additional smoothness assumptions on f and g , the next proposition shows that L is differentiable:
Proposition 1.
Let g be a twice differentiable function satisfying Assumption 1.
Assume that f is differentiable and consider the quadratic problem:
[Equation]
Then, (2) admits a unique minimizer z (cid:63) ( x, y ) for any ( x, y ) in X × Y .
Moreover, y (cid:63) ( x ) is unique and well-deﬁned for any x in X and L is differentiable with gradient given by:
[Equation]
Proposition 1 follows by application of the implicit function theorem (Lang, 2012, Theorem 5.9) and provides an expression for ∇L solely in terms of partial derivatives of f and g evaluated at ( x, y (cid:63) ( x )) .
Following Ghadimi and Wang (2018), we further make two smoothness assumptions on f and g :
Assumption 2.
There exist positive constants L f and B such that for all x, x (cid:48) ∈ X
and y, y (cid:
) ∈ Y :
[Equation]
Assumption 3.
There exit positive constants L (cid:48) g , M g such that for any x, x (cid:48) ∈ X and y, y (cid:48)
∈ Y : max {(cid:107) ∂
xy g ( x, y ) − ∂ xy g
( x (cid:48) , y (cid:48) ) (cid:107) , (cid:107) ∂
yy g
( x, y ) − ∂ yy g
( x (cid:48) , y (cid:48) ) (cid:107)}
≤ M g (cid:107) ( x, y ) − ( x (cid:48) , y (cid:48) )
(cid:107) (cid:107)
[Table]
Assumptions 1 to 3 allow a control of the variations of y (cid:63) and z (cid:63) and ensure L is L -smooth for some positive constant L as shown in Proposition 6 of Appendix B.2.
As an L -smooth function, L is necessarily weakly convex (Davis et al., 2018), meaning that L satisﬁes the inequality L ( x )
−L ( y ) ≤ ∇L ( x ) (cid:62) ( x − y ) − µ 2 (cid:107) x − y (cid:107) 2 for some ﬁxed µ ∈ R with | µ | ≤ L .
In particular, L is convex when µ ≥ 0 , strongly convex when µ > 0 and generally non-convex when µ < 0 .
We thus consider two cases for L , the strongly convex case ( µ > 0) and the non-convex case ( µ < 0) .
When L is convex, we denote by L (cid:63) its minimum value achieved at a point x (cid:63) and deﬁne κ L = L/µ when µ > 0 .
Stochastic/deterministic settings.
We consider the general setting where f ( x, y ) and g ( x, y ) are expressed as an expectation of stochastic functions ˆ f ( x, y, ξ ) and ˆ g ( x, y, ξ ) over a noise variable ξ .
We recover the deterministic setting as a particular case when the variable ξ has zero variance, thus allowing us to treat both stochastic (S) and deterministic (D) settings in a uniﬁed framework.
As often in machine-learning, we assume we can always draw a new batch D of i.i.d.
samples of the noise variable ξ with size |D| ≥ 1 and use it to compute stochastic approximations of f and g deﬁned by abuse of notation as ˆ f ( x, y, D ) := 1 |D| (cid:80) ξ ∈D ˆ f ( x, y, ξ ) and ˆ g ( x, y, D ) := 1 |D| (cid:80) ξ ∈D ˆ g ( x, y, ξ ) .
We make the following noise assumptions which are implied by those in Ghadimi and Wang (2018):
Assumption 4.
For any batch D , ∇ ˆ f ( x, y, D ) and ∂ y ˆ g ( x, y, D ) are unbiased estimator of ∇ f ( x, y ) and ∂ y g
( x, y ) with a uniformly bounded variance, i.e. for all x, y ∈
X × Y :
[Equation]
Assumption 5.
For any batch D , the matrices F 1 ( x, y, D ) :
= ∂
xy ˆ g ( x, y, D ) − ∂ xy g
( x, y ) and
F [Equation] x, y, D ) :
=
∂ yy ˆ g
( x, y, D ) − ∂
yy g
( x, y ) have zero mean and satisfy for all x,
y ∈
X × Y :
[Equation]
For conciseness, we will use the notations σ 2 f :=
˜ σ 2 f |D| − 1 , σ 2 g :=
˜ σ 2 g |D| − 1 , σ 2 g xy :=
˜ σ 2 g xy |D| − 1 and σ 2 g yy :
=
˜ σ 2 g yy |D| − 1 , without explicit reference to the batch D .
Next, we describe the algorithm.
3. A LGORITHMS
Amortized Implicit Gradient Optimization (AmIGO) is an iterative algorithm for solving (1).
It constructs iterates x k , y k and z k such that x k approaches a stationary point of L while y k and z k track the quantities y (cid:63) ( x k ) and z (cid:63) ( x k , y k ) .
AmIGO computes the iterate x k +1 using an update equation x k +
=
x
k − γ k ˆ ψ k for some given step-size γ k and a stochastic estimate ˆ ψ k of ∇L ( x k ) based on (3) and deﬁned according to (4) below for some new batches of samples D f and D g xy .
[Equation]
end AmIGO computes ˆ ψ k in 4 steps given iterates x k , y k −and
z k − 1 .
A ﬁrst step computes an approximation y k to y (cid:63) ( x k ) using a stochastic algorithm
A k initialized at y k − 1 .
A second step computes unbiased estimates u k = ∂
x
ˆ f ( x k , y k , D f ) and v k = ∂
y ˆ f ( x k , y k , D f ) of the partial derivatives of f w.r.t.
x and y .
A third step computes an approximation z k to z (cid:63) ( x k , y k ) using a second stochastic algorithm B k for solving (2) initialized at z k − 1 .
To increase efﬁciency, algorithm B k uses the pre-computed vector v k for approximating the partial derivative ∂ y f in (2).
Finally, the stochastic estimate ˆ ψ k is computed using (4) by summing the pre-computed vector u k with the jacobian-vector product
[Equation]
AmIGO is summarized in Algorithm 1.
Algorithms A k and B k .
While various choices for A k and B k are possible, such as adaptive algorithms (Kingma and Ba, 2015), or accelerated stochastic algorithms (Ghadimi and Lan, 2012), we focus on simple stochastic gradient descent algorithms with a pre-deﬁned number of iterations T and
N .
These algorithms compute intermediate iterates y t and z n optimizing the functions
y
(cid:55)→ g ( x k , y ) and z (cid:55)→ Q ( x k , y k , z ) starting from some initial values y 0 and z 0 and returning the last iterates y T and z N as described in Algorithms 2 and 3.
Algorithm A k updates the current iterate y t − 1 using a stochastic gradient
∂ y ˆ g
( x k , y t − 1 , D g ) for some new batch of samples D g and a ﬁxed step-size α k .
Algorithm B k updates the current iterate z t − 1 using a stochastic estimate of ∂ z Q ( x k , y k ,
z t − 1 ) with step-size β k .
The stochastic gradient is computed by evaluating the Hessian-vector product
∂
yy
ˆ g
( x k , y k , D g yy )
z t − 1 for some new batch of samples D g yy and summing it with a vector v k approximating ∂ y f
( x k , y k ) provided as input to algorithm B k .
Warm-start for y 0 and z 0 .
Following the intuition that y (cid:63) ( x k ) remains close to y (cid:63) ( x k − 1 ) when x k (cid:39) x k − 1 , and assuming that y k − 1 is an accurate approximation to y (cid:63) ( x k − 1 ) , it is natural to initialize A k with the iterate y k − 1 .
The same intuition applies when initializing B k with z k − 1 .
Next, we introduce a framework for analyzing the effect of warm-start on the convergence speed of AmIGO.
Algorithm [-
-]2 A
[Table]
Sample
[Table] [Table] ,k (cid:17) . 5: end for 6: Return y T .
Algorithm 3 B k ( x, y, v, z 0 ) 1: Parameters: β k , N .
2: for n ∈ { 1 , ..., N } do 3:
Sample batch D g yy n,k .
4: z n ← z n − 1 − β k (cid:16) ∂
yy
ˆ g (cid:16) x, y, D g yy n,k (cid:17)
z n − 1 + v (cid:17) .
5: end for 6: Return z N .
A NALYSIS OF A MORTIZED I MPLICIT G RADIENT O PTIMIZATION 4. G ENERAL APPROACH AND MAIN RESULT
The proposed approach consists in three main steps: (1) Analysis of the outer-level problem , (2)
Analysis of the inner-level problem and (3) Analysis of the joint dynamics of both levels.
Outer-level problem.
We consider a quantity E xk describing the evolution of x k deﬁned as follows:
[Equation]
where u ∈ { 0 , 1 } is set to 1 in the stochastic setting and to 0 in the deterministic one and δ k is a positive sequence that determines the convergence rate of the outer-level problem and is deﬁned by:
[Equation]
with η 0 such that γ − 1 0 ≥ η 0 ≥ µ if µ ≥ 0 and η 0 =
L if µ< 0 and where we choose the step-size γ k to be a non-increasing sequence with γ 0 ≤ 1 L .
With this choice for δ k and by setting u = 1 in (5), E xk recovers the quantity considered in the stochastic estimate sequences framework of Kulunchakov and
Mairal (2020) to analyze the convergence of stochastic optimization algorithms when L is convex.
When L is non-convex, E xk recovers a standard measure of stationarity (Davis and Drusvyatskiy, 2018).
In Section 4.3, we control E xk using bias and variance error E ψk − 1 and V ψk − 1 of ˆ ψ k given by (6) below where E k denotes expectation conditioned on ( x k , y k , z k − 1 ) .
[Equation]
Inner-level problems.
We consider the mean-squared errors E yk and E zk between initializations ( y 0 =
y k − 1 and z
=
z
k − 1 ) and stationary values ( y (cid:63) ( x k ) and z (cid:63) ( x k , y k ) ) of algorithms
A k and B k :
[Equation]
In Section 4.3, we show that the warm-start strategy allows to control E yk and E zk in terms of previous iterates E yk − 1 and E zk − 1 as well as the bias and variance errors in (6).
We further prove that such bias and variance errors are, in turn, controlled by E yk and E z k .
Joint dynamics.
Following Habets (1974), we consider an aggregate error E totk deﬁned as a linear combination of E xk , E yk and E zk with carefully selected coefﬁcients a k and b k :
[Equation]
As such E totk represents the dynamics of the whole system.
The following theorem provides an error bound for E totk in both convex and non-convex settings for a suitable choice of the coefﬁcients a k and b k provided that T and N are large enough:
Theorem 1.
Choose a batch-size (cid:12)(cid:12)
D g yy (cid:12)(cid:12) ≥ 1 ∨ ˜ σ 2 gyy µ g L g and the step-sizes
α k
= L − 1 g , β k =(2 L g ) − 1 , γ k
= L − 1 .
Set the coefﬁcients a k and b k to be a k :=
δ 0 (1 − α k µ g ) 1 / 2 and b k :
= δ 0 (cid:0) 1 − 12 β k µ g (cid:1) 1 /and set the number of iterations T and N of Algorithms 2 and 3 to be of order T
= O
( κ g ) and
N = O
( κ g ) up to a logarithmic dependence on κ g .
Let ˆ x k = u
(1 − δ k )
ˆ x k − 1 +(1 − u (1 − δ k ))
x k , with ˆ x 0 = x 0 .
Then, under Assumptions 1 to 5, E totk satisﬁes:
[Equation]
where W 2 , deﬁned in (20) of Appendix A.2, is the effective variance of the problem with W 2 =0 in the deterministic setting and, in the stochastic setting, W 2 > 0 is of the following order:
[Equation]
We describe the strategy of the proof in Section 4.3 and provide a proof outline in Appendix A. with exact expressions for all variables including the expressions of T , N and W 2 .
The full proof is provided in Appendix A.2.
The choice of a k and b k ensures that E yk and E zk contribute less to
E totk as the algorithms A k and B k become more accurate.
The effective variance W 2 accounts for interactions between both levels in the presence of noise and becomes proportional to the outer-level variance σ 2 f when the inner-level problem is solved exactly.
In the deterministic setting, all variances ˜ σ 2 f , ˜ σ 2 g , ˜ σ 2 g xy and ˜ σ 2 g yy vanish so that W 2 =0 .
Hence, we characterize such setting by W 2 =0 and the stochastic one by W 2 > 0 .
Next, we apply Theorem 1 to obtain the complexity of AmIGO.
4. C OMPLEXITY ANALYSIS
We deﬁne the complexity C ( (cid:15) ) of a bilevel algorithm to be the total number of queries to the gradients of f and g , Jacobian/hessian-vector products needed by the algorithm to achieve an error (cid:15) according to some pre-deﬁned criterion.
Let the number of iterations k , T and N and sizes of the batches |D g | , |D f | , (cid:12)(cid:12)
D g xy (cid:12)(cid:12) and (cid:12)(cid:12)
D g yy (cid:12)(cid:12) , be such that AmIGO achieves a precision (cid:15) .
Then C ( (cid:15) ) is given by:
[Equation]
We provide the complexity of AmIGO in the 4 settings of Table 1 in the form of Corrolaries 1 to 4 .
Corollary 1 (Case µ> 0 and W 2 =0 ) .
Use batches of size 1 .
Achieving L ( x k ) −L (cid:63) + µ 2 (cid:107) x
k − x (cid:63) (cid:107) 2 ≤ (cid:15) requires C ( (cid:15) )
=
O (cid:16) κ L κ g log (cid:16) E tot 0 (cid:15) (cid:17)(cid:17) .
Corollary 1 outperforms the complexities in Table 1 in terms of the dependence on (cid:15) .
It is possible to improve the dependence on κ g to κ 1 / 2 g using acceleration in A k and B k as discussed in
Appendix A.5.1, or using generic acceleration methods such as Catalyst (Lin et al., 2018).
Corollary 2 (Case µ W 2 > 0 ) .
Choose |
[Equation]
[Equation]
˜ σ 2 f (cid:15) (cid:17) and (cid:12)(cid:12)
D g yy (cid:12)(cid:12) =
Θ (cid:16) ˜ σ 2 g yy (cid:0) 1 (cid:15) ∨ κ g (cid:1)(cid:17) .
Achieving E
[ L (ˆ x k ) −L (cid:63) ]+ µ 2 E (cid:104) (cid:107) x k − x (cid:63) (cid:107) 2 (cid:105) ≤ (cid:15) requires:
[Equation]
= O
[Equation]
Corollary 2 improves over the results in Table 1 in the stochastic strongly-convex setting and recovers the dependence on (cid:15) of stochastic gradient descent for smooth and strongly convex functions up to a logarithmic factor.
Corollary 3 (Case µ< 0 and W 2 =0 ) .
Choose batches of size 1 .
Achieving 1 k (cid:80) ki
=1 (cid:107)∇L ( x i ) (cid:107) 2 ≤ (cid:15) requires C ( (cid:15) )
=
O (cid:16) κ 4 g (cid:15) (( L ( x 0 ) − L (cid:63) )
+ E y
+ E z 0 ) (cid:17) .
Corollary 3 recovers the complexity of AID-BiO (Ji et al., 2021) in the deterministic non-convex setting.
This is expected since AID-BiO also exploits warm-start for both A k and B k .
Corollary 4 (Case µ< 0 and W > 0 ) .
Choose (cid:12)(cid:12)
[Equation]
[Equation] (cid:0)
[Equation] (cid:15)µ
[Equation]
(cid:1)(cid:17) , |D f | = Θ (cid:16) [Equation] 2 g ˜
[Equation] 2 f (cid:15) (cid:17) and |D g | =
Θ (cid:16) κ 5 g ˜ σ 2 g (cid:15) (cid:17) .
Achieving an error 1 k (cid:80) ki =1 E (cid:104) (cid:107)∇L ( x i ) (cid:107) 2 (cid:105) ≤ (cid:15) requires:
[Equation]
Corollary 4 recovers the optimal dependence on (cid:15) of O ( 1 (cid:15) 2 ) achieved by stochastic gradient descent in the smooth non-convex case (Arjevani et al., 2019, Theorem 1).
It also improves over the results in (Ji et al., 2021) which involve an additional logarithmic factor log( (cid:15) − 1 ) as N is required to be
O ( κ g log( (cid:15) − 1 )) .
In our case, N remains constant since B k beneﬁts from warm-start.
The faster rates of MRBO/VRBO (cid:63) (Yang et al., 2021) are obtained under the additional mean-squared smoothness assumption (Arjevani et al., 2019), which we do not investigate in the present work.
Such assumption allows to achieve the improved complexity of O ( (cid:15) − 3 / 2 log( (cid:15) − 1 )) .
However, these algorithms still require N =
O (log( (cid:15) − 1 )) , indicating that the use of warm-start in B k could further reduce the complexity to O ( (cid:15) − 3 / 2 ) which would be an interesting direction for future work.
4. O UTLINE OF THE PROOF
The proof of Theorem 1 proceeds by deriving a recursion for both outer-level error E xk and innerlevel errors E yk and E zk
and then combining those to obtain an error bound on the total error E totk .
Outer-level recursion.
To allow a uniﬁed analysis of the behavior of E xk in both convex and nonconvex settings, we deﬁne F k as follows:
[Equation]
The following proposition, with a proof in Appendix C.1, provides a recursive inequality on E xk involving the errors in (6) due to the inexact gradient ˆ ψ k :
Proposition 2.
Let ρ k be a non-increasing sequence with 0 <ρ k < 2 .
Assumptions 1 to 3 ensure that:
[Equation]
with s k deﬁned as
[Equation]
[Equation] In the ideal case where y k = y (cid:63) ( x k ) and z k = z (cid:63) ( x k , y k ) , the bias E ψk vanishes and (10) simpliﬁes to (Kulunchakov and Mairal, 2019, Proposition 1) which recovers the convergence rates for stochastic gradient methods in the convex case.
However, y k and z k are generally inexact solutions and introduce a positive bias E ψk .
Therefore, controlling the inner-level iterates is required to control the bias E ψk which, in turn, impacts the convergence of the outer-level as we discuss next.
Controlling the inner-level iterates y k and z k .
Proposition 3 below controls the expected mean squared errors between iterates y k and z k and their limiting values y (cid:63) ( x k ) and z (cid:63) ( x k , y k ) :
Proposition 3.
Let the step-sizes α k and β k be such that α k ≤ L − 1 g and β k ≤ 1 2 L g ∧ µ g µ 2 g + σ 2 g yy .
Let Λ k :=(1 − α k µ g ) T and Π k :=
(cid:16) 1 − β k µ g 2 (cid:17) N .
Under Assumptions 1, 4 and 5, it holds that:
[Equation]
where R yk =
O (cid:0) κ g σ 2 g (cid:1) and
R z k
= O (cid:16) κ 3 g
σ 2 g yy
+ κ 2 g σ 2 f (cid:17) are deﬁned in (14) of Appendix A.2.
While Proposition 3 is speciﬁc to the choice of the algorithms
A k and B k in Algorithms 2 and 3, our analysis directly extends to other algorithms satisfying inequalities similar to (11) such as accelerated or variance reduced algorithms discussed in Appendices A.5.1 and A.5.2.
Proposition 4 below controls the bias and variance terms V ψk and E ψk in terms of the warm-start error E yk and E zk .
Proposition 4.
Under Assumptions 1 to 5, the following inequalities hold:
[Equation]
where w [Equation] O [Equation]
+
[Equation]
σ
[Equation]
yy (cid:17) , σ 2 x = O (cid:16) σ [Equation] g [Equation]
κ
[Equation] g σ [Equation] g yy (cid:17) and L ψ = O ( κ 2 g ) are positive constants deﬁned in (13) and (16) of Appendix A.2 with L ψ controlling the variations of E k [ ˆ ψ k ] .
Proposition 4 highlights the dependence of E ψk and V ψk on the inner-level errors.
It suggests analyzing the evolution of E yk and E zk to quantify how large the bias and variances can get:
Proposition 5.
Let ζ k > 0 , a 2 × 2 matrix P k , two vectors
U k and V k in R 2 all independent of x k , y k and z k be as deﬁned in Proposition 8 of Appendix A.2.
Under Assumptions 1 to 5, it holds that:
[Equation]
Proposition 5 describes the evolution of the inner-level errors as the number of iterations k increases.
The matrix P k and vectors U k and V k arise from discretization errors and depend on the step-sizes and constants of the problem.
The second term of (12) represents interactions with the outer-level through E xk − 1 , V ψk − 1 and E ψk − 1 .
Propositions 2, 4 and 5 describe the joint dynamics of ( E xk , E yk , E zk ) from which the evolution of E totk can be deduced as shown in Appendices A.1 and A.2.
E XPERIMENTS
We run three sets of experiments described in Sections 5.1 to 5.3.
In all cases, we consider AmIGO with either gradient descent (AmIGO-GD) or conjugate gradient (AmIGO-CG) for algorithm B k .
We then compare these algorithms with AID methods without warm-start for B k which we refer to as (AID-GD) and (AID-CG) and with (AID-CG-WS) which uses warm-start for B k but not for
A k .
We also consider other variants using either a ﬁxed-point algorithm (AID-FP)
(Grazzi et al., 2020) or Neumann series expansion (AID-N) (Lorraine et al., 2020) for B k .
Finally, we consider two algorithms based on iterative differentiation which we refer to as (ITD)
(Grazzi et al., 2020) and (Reverse) (Franceschi et al., 2017).
For all methods except (AID-CG-WS), we use warm-start in algorithm A k , however only AmIGO, AmIGO-CG and AID-CG-WS exploits warm-start in B k the other AID based methods initializing B k with z 0 =0 .
In Sections 5.2 and 5.3, we also compare with
BSA algorithm
(Ghadimi and Wang, 2018), TTSA algorithm (Hong et al., 2020a) and stocBiO (Ji et al., 2021).
5. S YNTHETIC PROBLEM
To study the behavior of AmIGO in a controlled setting, we consider a synthetic problem where both inner and outer level losses are quadratic functions with thousands of variables as described in details in Appendix F.1.
Figure 1(a) shows the complexity C ( (cid:15) ) needed to reach 10 − 6 relative error amongst the best choice for T and M over a grid as the conditioning number κ g increases.
AmIGO-CG achieves the lowest time and is followed by AID-CG thus showing a favorable effect of warm-start for B k .
The same conclusion holds for AmIGO-GD compared to AID-GD.
Note that
AID-CG is still faster than AmIGO-CG for larger values of κ g highlighting the advantage of using algorithms B k with O ( √ κ g ) complexity such as (CG) instead non-accelerated ones with O ( κ − 1 g ) such as (GD).
Figure 1(b) shows the relative error after 10 s and maintains the same conclusions.
For moderate values of κ g , only AmIGO and AID-CG reach an error of 10 − 20 as shown in Figure 1(c).
We refer to Figures 2 and 3 of Appendix F for additional results on the effect of the choice of T and
M showing that AmIGO consistently performs well for a wide range of values of T and M .
5. H YPER - PARAMETER OPTIMIZATION
We consider a classiﬁcation task on the 20Newsgroup dataset using a logistic loss and a linear model.
Each dimension of the linear model is regularized using a different hyper-parameter.
The
[Figure]
collection of those hyper-parameters form a vector x of dimension d =101631 optimized using an unregularized regression loss over the validation set while using the training set.
We consider two evaluations settings: A default setting based on Grazzi et al.
; Ji et al.
(2021) and a grid-seach search setting near the default values of β k , T and N as detailed in Appendix F.2.
We also vary the batch-size from 10 3 ∗ { 0 . 1 , 1 , 2 , 4 } and report the best performing choices for each method.
Figure 1(d,e,f) show AmIGO-CG to be the fastest, achieving the lowest error and highest validation and test accuracies.
The test accuracy of AmIGO-CG decreases after exceeding 80% indicating a potential overﬁtting as also observed in Franceschi et al.
Similarly, AmIGO-GD outperformed all other methods that uses an algorithm B k with O ( κ g ) complexity.
Moreover, all remaining methods achieved comparable performance matching those reported in Ji et al.
(2021), thus indicating that the warm-start in B k and acceleration in B k were the determining factors for obtaining an improved performance.
Additionally, Figure 4 of Appendix F report similar results for each choice of the batch-size indicating robustness to the choice of the batch-size.
5. D ATASET DISTILLATION
Dataset distillation (Wang et al., 2018) consists in learning a synthetic dataset so that a model trained on this dataset achieves a small error on the training set.
Figure 5 of Appendix F.3 shows the training loss (outer loss), the training and test accuracies of a model trained on MNIST by dataset distillation.
Similarly to Figure 1, AmIGO-CG achieves the best performance followed by AID-CG.
AmIGO obtains the best performance by far among methods without acceleration for B k while all the remaining ones fail to improve.
This ﬁnding is indicative of an ill-conditioned inner-level problem as conﬁrmed when computing the conditioning number of the hessian ∂
yy g
( x, y ) which we found to be of order 7 × 10 4 .
Indeed, when compared to the synthetic example for κ g =10 4 as shown in Figure 2, we also observe that only AmIGO-CG, AmIGO and AID-CG could successfully optimize the loss.
Hence, these results conﬁrm the importance of warm-start for an improved performance.
C ONCLUSION
We studied AmIGO, an algorithm for bilevel optimization based on amortized implicit differentiation and introduced a uniﬁed framework for analyzing its convergence.
Our analysis showed that
AmIGO achieves the same complexity as unbiased oracle methods , thus achieving improved rates compared to methods without warm-start in various settings.
We then illustrated empirically such improved convergence in both synthetic and a hyper-optimization experiments.
A future research direction consists in extending the proposed framework to non-smooth objectives and analyzing acceleration in both inner and outer level problems as well as variance reduction techniques.  A CKNOWLEDGMENTS AND F UNDING
This project was supported by the ERC grant number 714381 (SOLARIS project) and by ANR 3IA
MIAI@Grenoble Alpes, (ANR19-P3IA-0003).
A C ONVERGENCE OF A M IGO ALGORITHM
In this section, we provide a proof of Theorem 1 as well as its corollaries Corrolaries 1 to 4.
In
Appendix A.1, we provide an outline of the proof Theorem 1 that states the main intermediary results needed for the proof and provide explicit expressions for the quantities needed throughout the rest of the paper.
Appendices A.2 and A.3 provide the proofs of Theorem 1 and Corrolariesto 4.
The proofs of the intermediary results are deferred to Appendices B and C.
A. P ROOF OUTLINE OF T HEOREM
The proof of Theorem 1 proceeds in 8 steps as discussed bellow.
Step 1: Smoothness properties.
This step consists in characterizing the smoothness of ∇L , y (cid:63) , z (cid:63) as well as the conditional expectation E [ ˆ ψ k | x k , y k , z k ] knowing x k , y k and z k .
For this purpose, we consider the function Ψ : X × Y × Y → X deﬁned as follows:
[Equation]
Hence, by deﬁnition of ˆ ψ k , it is easy to see that E [ ˆ ψ k | x k , y k , z k ]
= Ψ( x k , y k , z k ) .
The following proposition controls the smoothness of ∇L , y (cid:63) , z (cid:63) and Ψ and is adapted from (Ghadimi and Wang, 2018, Lemma 2.2).
We provide a proof in Appendix B.2 for completeness.
Proposition 6.
Under Assumptions 1 to 3, L , Ψ , y (cid:63) and z (cid:63) satisfy:
[Equation]
where L y , L z , L ψ and L are given by:
[Equation]
The expressions of L y , L z , L ψ and L suggests the following dependence on the conditioning κ g of the inner-level problem which will be useful for the complexity analysis: L y =
O ( κ g ) , L ψ =
O ( κ 2 g ) , L z
= O ( κ 2 g ) and L
= O ( κ 3 g ) .
Step 2: Convergence of the inner-level iterates.
In this step, we control the mean squared errors E (cid:104) (cid:107) y k − y (cid:63) ( x k ) (cid:107) 2 (cid:105) and E (cid:104) (cid:107)
z k − z (cid:63) ( x k , y k ) (cid:107) 2 (cid:105) as stated in Proposition 3.
In fact we prove a slightly stronger version stated below:
Proposition 7.
Let α k and β k be two positive sequences with α k ≤ L − 1 g and β k ≤ 12 L g min (cid:18) 1 ,
[Equation]
(cid:19) and deﬁne
Λ k :=(1 − α k µ g ) T and Π k := (cid:16) 1 − β k µ g 2 (cid:17) N .
Denote by ¯ z k the conditional expectation of z k knowing x k , y k and z 0 k .
Let R yk and R zk be deﬁned as:
[Equation]
Then, under Assumptions 1, 4 and 5 the iterates z k and ¯ z k satisfy:
[Equation]
It is easy to see from the above expressions that R yk
=
O (cid:0) κ g σ 2 g (cid:1) while R zk
= O (cid:16) κ 3 g
σ 2 g y y
+ κ 2 g σ 2 f (cid:17) as stated in Proposition 3.
Controlling E [ (cid:107) y k − y (cid:63) ( x k ) (cid:107) ] 2 follows by standard results on SGD (Kulunchakov and Mairal, 2020, Corollary 31) since the iterates of Algorithm 2 uses i.i.d.
samples.
The error terms E (cid:104) (cid:107) z k − z (cid:63) ( x k , y k ) (cid:107) 2 (cid:105) is more delicate since Algorithm 3 uses the same sample
∂
y ˆ f ( x k , y k ) for updating the iterates, therefore introducing additional correlations between these iterates.
We defer the proof of Proposition 7 to Appendix B.3 which relies on a general result for stochastic linear systems with correlated noise provided in Appendix E.
Step 3:
Controlling the bias and variance errors V ψk and E ψk is achieved by Proposition 4.
The bias E ψk is controlled simply by using the smoothness of the potential Ψ near the point ( x k , y (cid:63) ( x k ) , z (cid:63) ( x k , y (cid:63) ( x k )) as shown in Proposition 6.
The variance term V ψk is more delicate to control due to the multiplicative noise resulting from the Jacobian-vector product between ∂ xy ˆ g ( x k , y k , D g xy )
z k .
We defer the proof to Appendix B.4 and provide below explicit expressions for the constants σ 2 x and w 2 x :
[Equation]
16a
[Equation]
Note that w 2 x = O (cid:16) κ 2 g (cid:16) σ 2 f + σ 2 g xy (cid:17) + κ 3 g σ 2 g yy (cid:17) and σ 2 x = O (cid:16) σ 2 g xy + κ 2 g σ 2 g yy (cid:17) , as stated in Proposition 4.
Step 4: Outer-level error bound.
This step consists in obtaining the inequality in Propositionwhich extends the result of (Kulunchakov and Mairal, 2020, Proposition 1) to biased gradients and to the non-convex case.
We defer the proof of such result to Appendix C.1.
Step 5: Inner-level error bound.
This step consists in proving Proposition 5.
For clarity, we provide a second statement with explicit expressions for the quantities of interest:
Proposition 8.
Let r k
and θ k be two positive non-increasing sequences no greater than 1 .
For any 0 ≤ v ≤ 1 , denote by φ k and ˜ R k
the following non-negative scalars:
[Equation]
Finally, consider the following matrices and vectors:
[Equation]
Then, under Assumptions 1 to 5, the following holds:
[Equation]
We defer the proof of the above result to Appendix C.2.
Step 6: General error bound.
By combining the inequalities in Propositions 2 and 8 resulting from the analysis of both outer and inner levels, we obtain a general error bound on E totk in Propositionwith a proof in Appendix C.4.
Proposition 9.
Choose the step-sizes α k and β k such that they are non-increasing in k and choose r k and θ k such that δ k r − 1 k and δ k θ − 1 k are non-increasing sequences.
Choose the coefﬁcients a k and b k deﬁning E totk in (7) to be of the form a k = δ k r − 1 k Λ sk and b k = δ k θ − 1 k Π sk for some 0 < s < 1 .
and ﬁx a non-increasing sequence 0 < ρ k < 1 .
Then, under Assumptions 1 to 5 E totk satisﬁes:
[Equation]
where V k and A k are given by:
[Equation]

[Equation]
where we introduced u
Ik = a k U (1) k
+ b k U (2) k for conciseness with U (1) k and U (2)
k being the components of the vector U k deﬁned in Proposition 8.
Proposition 9 holds without conditions on the error made by Algorithms 2 and 3.
The general form of a k and b k allows to account for potentially decreasing step-sizes γ k , α k and β k .
However, in the present work, we will restrict to the constant step-size for ease of presentation as we discuss next.
Step 7: Controlling the precision of the inner-level algorithms.
In this step, we provide conditions on T and N in Proposition 10 bellow so that (cid:107) A k (cid:107) ∞ ≤ 1 − [Equation]
1 −
[Equation]
k )
[Equation]
k in the constant step-size case.
These conditions are expressed in terms of the following constants:
[Equation]
Proposition 10.
Let Assumptions 1 to 5 hold.
Choose ρ k = 12 , the step-sizes to be constant: α k = α ≤ 1 L g , β k = β ≤ 12 L g and γ k = γ ≤ 1 L and choose the batch-size (cid:12)(cid:12)
D g yy (cid:12)(cid:12) =
Θ (cid:18) ˜
σ 2 gyy µ g L [Equation] (cid:19) .
Moreover, set s = 12 , r k = θ
k
= 1 .
Finally, choose T and N as follows:
[Equation]
with C 1 , C 2 , C 3 , C (cid:48) 1 , C (cid:48) 2 and C (cid:48) 3 deﬁned in (19a) to (19f).
Then, (cid:107) A k (cid:107) ∞ ≤ 1 − (1 − ρ k ) δ k and
V totk ≤ γδ 0 W 2 , with W 2 given by:
[Equation]
We provide a proof of Proposition 10 in Appendix D.
It is easy to see from Proposition 10 that that
T =
O ( κ g ) and N =
O ( κ g ) when α = 1 L g and β = 12 L g ,
where the big-
O notation hides a logarithmic dependence in κ g coming from the constants { C i , C (cid:48)
i | i ∈ { 1 , 2 , 3 }} .
Step 8: Proving the main inequalities.
The ﬁnal step combines Propositions 9 and 10 to get the desired inequality.
We provide a full proof in Appendix A.2 assuming Propositions 9 and 10 hold.
A. P ROOF OF T HEOREM
In order to prove Theorem 1 in the convex case, we need the following averaging strategy lemma , a generalization of (Kulunchakov and Mairal, 2020, Lemma 30):
Lemma 1.
Let L be a convex function on X .
Let x k be a (potentially stochastic) sequence of iterates in X .
Let ( E k ) k ≥ 0 , ( V k ) k ≥ 0 and ( δ k ) k ≥ 0 be non-negative sequences such that δ k ∈ (0 , 1) .
Fix some non-negative number u ∈ [0 , 1] and deﬁne the following averaged iterates ˆ x k recursively by ˆ x k = u (1 − δ k )
ˆ x k − 1 +(1 − (1 − δ k ) u ) x k and starting from any initial point ˆ x 0 .
Assume the iterates ( x k ) k ≥ 1 satisfy the following relation for all k ≥ 1 :
[Equation]
Let Γ k := (cid:81) kt
=1 (1 − δ k ) .
Then the averaged iterates (ˆ x k ) k ≥ 1 satisfy the following:
[Equation]
Proof.
For simplicity, we write F k = E
[ L ( x k ) − L (cid:63) ] and ˆ
F k = E
[ L (ˆ x k ) − L (cid:63) ] .
We ﬁrst multiply (21) by Γ − 1 k and sum the resulting inequalities for all 1 ≤ k ≤ K to get:
K (cid:88)
[Equation]
Grouping the terms in F k together and recalling that Γ − 1 k (1 − δ k )
=
Γ − 1 k − 1 yields:
K (cid:88)
[Equation]
Simplifying the telescoping sums and multiplying by Γ k
[Equation]
we get:
[Equation]
Consider now the quantity ˆ F k .
Recalling that L is convex and by deﬁnition of the iterates
ˆ x k we apply Jensen’s inequality to write:
[Equation]
By iteratively applying the above inequality
[Equation]
we get that:
[Equation]
We can therefore apply (22) to the above inequality to get the desired result.
We now proceed to prove Theorem 1.
Proof of Theorem 1.
By application of Proposition 9 and using the choice of T and N given by
Proposition 10, the following inequality holds:
[Equation]
with W deﬁned in Proposition 10.
We then distinguish two cases depending on the sign of µ :
Case µ ≥ 0 .
Recall that F k and E xk are given by:
[Equation]
Since µ > 0 , L is a convex function and we can apply Lemma 1 with V k = γδ 0 W 2 and
E k
=
δ k 2 γ k (cid:107) x
k − x (cid:63) (cid:107) {+
+} + a k E yk
+ b k E zk .
The result follows by noting that Γ k (cid:80) kt
=1 Γ − 1 t ≤ δ − 1 0 .
Case µ < 0 .
In this case, we recall that F k and E xk are given by:
[Equation]
We then sum (23) for all iterations 0 ≤ t ≤ k which, by telescoping, yields:
[Equation]
Using that E [ L ( x k ) − L (cid:63) ]
+ E totk − E xk is non-negative since E totk − E xk = a k E yk
+ b k E
zk , we get: (cid:88)
[Equation]
Finally, since ρ t = 12 , δ t =
Lγ , the result follows after dividing both sides by 12 kLγ .
A. P ROOF OF C ORROLARIES 1 TO
Proof of Corollary 1.
Choosing u = 0 implies that E xk = µ
[Equation]
We can then apply Theorem 1 for µ > 0 which yields the following:
[Equation]
In the deterministic setting, it holds all variances vanish : σ 2 f = σ 2 g = σ 2 g yy =
σ 2 g xy =0 .
Hence, W 2 =by deﬁnition of W 2 .
Therefore, to achieve an error L ( x k ) −L (cid:63) ≤ (cid:15) for some (cid:15)> 0 , (24) suggests choosing k
= O (cid:16) κ L log (cid:16) E tot 0 (cid:15) (cid:17)(cid:17) .
Additionally, T =
Θ( κ g ) and N =Θ( κ g ) as required by Theoremand since σ 2 g
yy =0 , it holds that N
=
O
( κ g ) .
Using batches of size 1 , yields the desired complexity.
Proof of Corollary 2.
Here we choose u = 1 and apply Theorem 1 for µ > 0 which yields:
[Equation]
Hence, to achieve an error E
[ L (ˆ x k ) − L (cid:63) ] ≤ (cid:15) , we need k = O (cid:16) κ L log (cid:16)
[Equation]
(cid:17)(cid:17) to guarantee that the ﬁrst term in the l.h.s.
of the above inequality is O ( (cid:15) ) .
Moreover, we recall that
L − 1 =
O ( κ − 3 g ) from Proposition 6 and that Theorem 1 ensure the variance W satisﬁes:
[Equation]
Hence, ensuring the variance term 2 L − 1 W 2 is of order (cid:15) is achieved by choosing the size of the batches as follows:
[Equation]
Recall that T =
Θ( κ g ) and N =Θ( κ g ) as required by, Theorem 1, thus yielding the desired result.
Proof of Corollary 3.
In the non-convex deterministic case, recall that E xk = 1 L (cid:107)∇L ( x k ) (cid:107) 2 ≤ E totk .
We thus apply Theorem 1 for µ < 0 , multiply by L to get:
[Equation]
The setting being deterministic, it holds that W 2 =0 .
Moreover, recall that L =
O ( κ 3 g ) from
Proposition 6.
Hence, to achieve an error of order min 1 ≤ t ≤ k (cid:107)∇L ( x t ) (cid:107) 2 ≤ (cid:15) , it sufﬁce to choose k = O (cid:16) κ 3 g (cid:15)
( L ( x 0 ) − L (cid:63) + ( E y
+ E z 0 )) (cid:17) .
Thus using batches of size 1 and T and N of order κ g .
Proof of Corollary 4.
In the non-convex stochastic case, E xk = 1 L E (cid:104) (cid:107)∇L ( x k ) (cid:107) 2 (cid:105) ≤ E totk .
We thus apply Theorem 1 for µ < 0 , multiply by L to get:
[Equation]
to achieve an error of order (cid:15) , we need to ensure each term in the l.h.s.
of the above inequality is of order (cid:15) .
For the ﬁrst term, similarly to the deterministic setting Corollary 3, we simply need k = O (cid:16) κ 3 g (cid:15)
( L ( x 0 ) − L (cid:63) + ( E y {+
+} + E z 0 )) (cid:17) .
For the second term, we need to have W 2 =
O ( (cid:15) ) , which is achieved using the following choice for the sizes of the batches:
[Equation]
Finally, as required by Theorem 1, we set T =
Θ( κ g ) and N = Θ( κ g ) thus yielding the desired complexity.
A. C OMPARAISONS WITH OTHER METHODS
In this subsection we derive and discuss the complexities of methods presented in Table 1.
A.4. C OMPARAISON WITH TTSA (H ONG ET AL ., 2020 B )
Proposition 11.
strongly-convex case µ > 0 .
The complexity of the TTSA algorithm in Hong et al.
(2020b) to achieve an error µ 2 E (cid:104) (cid:107) x k − x (cid:63) (cid:107) 2 (cid:105) ≤ (cid:15) is given by:
[Equation]
non-convex case µ < 0 .
The complexity of the TTSA algorithm in Hong et al.
(2020b) to achieve an error 1 k (cid:80) 1 ≤ i ≤
[Equation]
E [Equation] is given by:
[Equation]
Proof.
strongly-convex case µ >
Using the choice of step-sizes in Hong et al.
(2020b), the following bound holds:
[Equation]
where ∆ 0 x = E (cid:104) (cid:107) x 0 − x (cid:63) (cid:107) 2 (cid:105) , E (cid:104) (cid:107) y 1 − y (cid:63) ( x 0 ) (cid:107) 2 (cid:105) and k
α given by:
[Equation]
By a simple calculation, it is easy to see that (cid:81) ki =0 (cid:16) 1 −
[Equation]
[Equation]
Moreover, using that L ψ = O ( µ − 2 g ) , L y =
O ( µ − 1 g ) , we get that
[Equation]
Using that L
= O ( µ − 3 g ) , we get µ − 6 g µ − 13 (cid:0) 1 + µ − 2 (cid:1) =
O (cid:16) µ − 5 g κ 13 L + µ g κ 73 L (cid:17) .
Hence, to reach an error (cid:15) , we need to control both terms in the above inequality.
This suggests the following condition on k to control the second term which dominates the error:
[Equation]
Moreover, the result in (Hong et al., 2020b, Theorem 1) requires N = Θ (cid:0) κ g log 1 (cid:15) (cid:1) , where N is the number of terms in the Neumann series used to approximate the hessian inverse ( ∂ yy g ( x, y )) − 1 in the expression of the gradient ∇L .
Hence, the total complexity is given by the following expression:
[Equation]
Smooth Non-convex case µ < 0 .
Following Hong et al.
(2020b), consider the proximal map of L for a ﬁxed ρ > 0 :
[Equation]
and deﬁne the quantity ˜∆ kx :
= E (cid:104) (cid:107) ˆ x ( x k ) − x k (cid:107) 2 (cid:105) , where x k are the iterates produced by the TTSA algorithm.
Let K be a random variable uniformly distributed on { 0 , ..., K − 1 } and independent from the remaining r.v. used in the TTSA algorithm.
The result in (Hong et al., 2020b, Theorem 2) provide the following error bound on ˜∆ kx
[Equation]
where ρ is set to 2 L and ∆ 0 ≤ max (cid:16) E
[ L ( x 0 ) −L
[Equation] ] , E (cid:104) (cid:107) y 1 − y (cid:63) ( x 0 ) (cid:107) 2 (cid:105)(cid:17) .
Now, recall that by deﬁnition of the proximal map, we have the following identity:
[Equation]
Hence, we obtain the following error bound:
[Equation]
where we used that L ψ =
O ( κ 2 g ) .
Therefore, to reach an error of order (cid:15) , TTSA requires:
[Equation]
Moreover, controlling the bias in the estimation of the gradient requires N =
O
( κ g log 1 (cid:15) ) terms in the Neumann series approximating the hessian.
Hence, the total complexity of the algorithm is:
[Equation]
A.4. C OMPARAISON WITH A CC B IO (J I AND L IANG , 2021)
Complexity of AccBio.
The bilevel algorithm AccBio introduced in Ji and Liang (2021) uses acceleration for both the inner and outer loops.
This allows to obtain the following conditions on k , T and N to achieve an (cid:15) accuracy:
[Equation]
Note that, since AccBio do not use a warm-start strategy when solving the linear system, N is required to grow as log 1 (cid:15) in order to achieve an (cid:15) accuracy.
This contributes an additional logarithmic factor to the total complexity so that C ( (cid:15) )
=
O ( κ 12 L κ 12 g (cid:0) log 1 (cid:15) (cid:1) 2 ) .
This is by contrast with AmIGO which exploits warm start when solving the linear system and thus only needs a constant number of iterations N =
O ( κ g )
although the dependence on κ g is worse compared to AccBio.
However, it is possible to improve such dependence by using acceleration in the inner-level algorithms A k and B k as we discuss in Appendix A.5.1.
Complexity of AccBio as a function of µ and µ g .
The authors choose to report the complexity as a function of µ and µ g instead of the conditioning numbers κ L and κ g .
To achieve this, they observe that, under the additional assumption that the hessian ∂ yy g ( x, y ) is constant w.r.t.
y , the Lipschitz constant L has an improved dependence on µ g : L = O ( µ − 2 g ) instead of L
= O ( µ − 3 g ) in the general case
where ∂
yy g
( x, y ) is only Lipschitz in y .
This allows them to express κ L =
Lµ
=
O
( µ − 2 g µ − 1 ) and to report the following complexities in terms of µ and µ g :
[Equation]
Note that, in the general case where L = O ( µ − 3 g ) , the complexity as a function of µ and µ g becomes
O (cid:16) µ − 12 µ − 2 g (cid:0) log 1 (cid:15) (cid:1) 2 (cid:17) , while still maintaining the same expression in terms of κ L and κ g .
Hence, using the expression in terms of conditioning allows a more general expression for the complexity that is less sensitive to the speciﬁc assumptions on g and is therefore more suitable for comparaison with other results in the literature.
A. C HOICE OF THE INNER - LEVEL ALGORITHMS A k AND B k .
The choice of A k and B k has an impact on the total complexity of the algorithm.
We discuss two choices for A k and B k which improve the total complexity of AmIGO: Accelerated algorithms (in
Appendix A.5.1) and variance reduced algorithms (in Appendix A.5.2).
A.5. A CCELERATION OF THE INNER - LEVEL FOR A M IGO
AmIGO could beneﬁt from acceleration in the inner-loop by using standard acceleration schemes
Nesterov (2003) for A k and B k .
As a consequence, and using analysis of accelerated algorithms (Nesterov, 2003) in the deterministic setting, the error of the inner-level iterates would satisfy:
[Equation]
where ˜Λ k and ˜Π k are accelerated rates of the form ˜Λ k
= O
((1 −√ κ g ) T ) and ˜Π k
= O
((1 −√ κ g ) N ) .
The rest of the proofs are similar provided that Λ k and Π k are replaced by their accelerated rates
˜ Λ k and ˜Π k .
This implies that T and N need to be only of order T
=
O
( √ κ g ) and N = O ( √ κ g ) so that the ﬁnal complexity becomes:
[Equation]
Note that using conjugate gradient for B k also enjoys an accelerated convergence rate Shewchuk et al.
This is conﬁrmed in our experiments of Figure 1 where AmIGO-CG enjoys the fastest convergence.
In order to further improve the dependence on κ L to κ 1 / 2 L , one would need to use an accelerated scheme when updating the iterates x k .
The analysis of such scheme along with warm-start would be an interesting direction for future work.
A.5. V ARIANCE REDUCED ALGORITHMS FOR A k AND B k
When the inner-level cost function g is a ﬁnite average of functions g ( x, y )
=
1 n (cid:80) 1 ≤ i ≤ n g i ( x, y ) empirical average, it is possible to use variance reduced algorithms such as SAG (Schmidt et al.,
If every function g i is L g -smooth, then by (Schmidt et al., 2017, Proposition 1), the inner level error becomes:
[Equation]
with ˜Λ k = (cid:0) 1 − κ g 8 n (cid:1) T .
This has the advantage that the error due to the variance decays exponentially with the number of iterations T .
As a consequence, the dependence of the effective variance W 2 on the conditioning numbers κ L and κ g can be improved to:
[Equation]
This can be achieved by taking T
= O ( nκ g ) up to a logarithmic dependence on the condition numbers.
As a consequence, the complexity in the strongly convex stochastic setting becomes:
[Equation]
In the non-convex setting
[Equation]
the complexity becomes:
[Equation]
The downside of this approach is the dependence on the number n of functions g i in the total complexity.
B P RELIMINARY RESULTS B. E XPRESSION OF THE GRADIENT
We provide a proof of Proposition 1 which shows that L is differentiable and provides an expression of its gradient.
Proof.
Assumption 1 ensures that y (cid:55)→ g ( x, y ) admits a unique minimizer y (cid:63) ( x ) deﬁned as the unique solution to the implicit equation ∂ y g ( x, y (cid:63) ( x ))
= 0 .
Moreover, since g is twice continuously differentiable and strongly convex, it follows that ∂
yy
g
( x, y (cid:63) ( x )) is invertible for any x ∈ X .
Therefore the implicit function theorem (Lang, 2012, Theorem 5.9), ensures that x (cid:55)→ y (cid:63) ( x ) is continuously differentiable with Jacobian given by ∇ y (cid:63) ( x ) = − ∂ xy g
( x, y (cid:63) ( x ))
∂
yy g
( x, y (cid:63) ( x )) − 1 .
Hence, by composition of differentiable functions, L is also differentiable with gradient given by:
[Equation]
We can thus deﬁne z (cid:63) ( x, y )
= − ∂
yy g
( x, y ) − 1 ∂
y f
( x, y ) to get the desired expression for ∇L ( x ) and note that z (cid:63) is the solution to (2).
B. S MOOTHNESS PROPERTIES OF L , Y (cid:63) , z (cid:63) AND Ψ
Proof of Proposition 6.
Lipschitz continuity of x (cid:55)→ y (cid:63) ( x ) .
By Assumptions 1 and 3, the implicit function theorem (Lang, 2012, Theorem 5.9) ensures y (cid:63) ( x ) is differentiable with Jacobian given by:
[Equation]
Moreover, by Assumption 3, we know that ∂ y g
( x, y ) is L (cid:48)
g -Lipchitz in x for any y ∈ Y , hence, (cid:107) ∂
xy g ( x, y (cid:63) ( x ))
(cid:107) op is upper-bounded by L (cid:48) g .
Moreover, by Assumption 1, g is µ g -strongly convex in y uniformly on X .
Therefore, it holds that (cid:13)(cid:13)
∂
yy g
( x, y (cid:63) ( x )) − 1 (cid:13) (cid:13) op ≤ µ − 1 g .
This allows to deduce that (cid:107)∇ y
[Equation] x ) (cid:107) op ≤ µ − 1 g L [Equation] g , and by application of the fundamental theorem of calculus that:
[Equation]
This shows that y (cid:63) is L y -Lipschitz continuous with L y := µ − 1 g L (cid:48) g .
Lipchitz continuity of x (cid:55)→ z (cid:63) ( x, y ) .
Let ( x, y ) and ( x (cid:48) , y (cid:48) ) be two points in X × Y .
Recalling the deﬁnition of z (cid:63) ( x, y ) in Proposition 1, it is easy to see that z (cid:63) ( x, y ) admits the following expression:
[Equation]
Recalling the expression of z (cid:63) ( x, y )
[Equation]
the following holds:
[Equation]
Hence
[Equation]
by taking the norm of the above quantity a triangular inequality followed by operator inequalities, it follows that:
[Equation]
where we introduced H 1 = ∂
yy g
( x, y ) and H 2 = ∂
yy g
( x (cid:48) , y (cid:48) ) for conciseness.
Using Assumption 1, we can upper-bound (cid:13)(cid:13)
H − 1 1 (cid:13)(cid:13) op and (cid:13)(cid:13)
H − 1 2 (cid:13)(cid:13) op by µ − 1 g .
By Assumption 3, we know that (cid:107) H 1 − H 2 (cid:107) op ≤ M g (cid:107) ( x, y ) − ( x (cid:48) , y (cid:48) )
(cid:107) .
Finally by Assumption 2, we also have that (cid:107) ∂
y f ( x (cid:48) , y (cid:48) ) − ∂ y f
( x, y ) (cid:107) ≤ L f (cid:107) ( x, y ) − ( x (cid:48) , y (cid:48) )
(cid:107) and that (cid:107) ∂
y
f
( x (cid:48) , y (cid:48) ) (cid:107) ≤ B ensuring that:
[Equation]
Hence, we conclude that z (cid:63) is L z -Lipchitz continuous with L z deﬁned as in (13).
boundedness of z (cid:63) ( x, y )
Recalling the expression of z (cid:63) in (25), it is easy to see that (cid:107)
z
(cid:63) ( x, y ) (cid:107) is upper-bounded by µ − 1 g B since ∂
yy g
( x, y ) is µ g -strongly convex in y by Assumption 1 and ∂ y f ( x, y ) is bounded by B by Assumption 2.
Regularity of Ψ .
[Equation]
By taking the norm of the above expression and applying a triangular inequality followed by operator inequalities, it follows that:
[Equation]
To get the ﬁrst term of the last inequality above, we used that ∂ x f is L f -Lipschitz by Assumption 2.
To get the second term, we used that ∂ xy g ( x, y ) is bounded since ∂ y g ( x, y ) is L (cid:48)
g -Lipschitz by
Assumption 3.
Finally, for the last term, we used that ∂ xy g ( x, y ) is M g -Lipschitz by Assumption 3.
By choosing x (cid:48)
= x , y (cid:48)
=
y (cid:63) ( x ) and z (cid:48) =
z (cid:63) ( x, y (cid:63) ( x )) , it is easy to see from Proposition
that Ψ( x, y (cid:63) ( x ) , z (cid:63) ( x, y (cid:63) ( x ))) =
∇L ( x ) .
Hence, applying the above inequality yields:
[Equation]
As shown earlier, (cid:107) z (cid:63) ( x, y (cid:63) ( x ))
(cid:107) is upper-bounded by µ − 1 g B , while (cid:107)
z (cid:63) ( x, y )
− z (cid:63) ( x, y (cid:63) ( x )) (cid:107) is bounded by L z (cid:107) y − y (cid:63) ( x ) (cid:107) .
This allows to conclude that (cid:107) Ψ( x, y, z ) − ∇L ( x ) (cid:107)
≤ L ψ with L ψ deﬁned in (13).
Lipschitz continuity of x (cid:55)→ ∇L ( x ) .
We apply (26) with ( y, z ) =
( y (cid:63) ( x ) , z (cid:63) ( x, y (cid:63) ( x ))) and ( y (cid:48) , z (cid:48) )
= ( y (cid:63) ( x (cid:48) ) , z (cid:63) ( x, y (cid:63) ( x (cid:48) ))) which yields:
[Equation]
where we used that (cid:107) z (cid:63) ( x (cid:48) , y (cid:63) ( x (cid:48) ))
(cid:107) is upper-bounded by µ − 1 g B , z (cid:63) is
L z -Lipschitz and y (cid:63)
is L yLipschitz.
Hence, ∇L is L -Lipschitz continuous, with L as given by (13).
B. C ONVERGENCE OF THE ITERATES OF ALGORITHMS A k AND B k
Proof.
Controlling the iterates y t of A k .
Consider a new batch D g of samples ξ .
We have by deﬁnition of the update equation of y t that:
[Equation]
Taking the expectation conditionally on x k and y t −
[Equation]
we get:
[Equation]
The ﬁrst line uses that ∂ y ˆ g (cid:0) x k , y t − 1 , D g (cid:1) is an unbiased estimator of ∂ y g ( x k , y t − 1 ) .
For the second line, we use Assumption 4 which allows to upper-bound the variance of ∂ y ˆ g by σ 2 g .
Moreover, since g is convex and L g -smooth and since α k ≤ L − 1 g , it follows that the last term in the above inequality is non-positive and can thus be upper-bounded by 0 .
By unrolling the resulting inequality recursively for 1 < t ≤ k , we obtain the desired result.
Controlling the iterates z n of B k .
The poof follows by direct application of Proposition 15 with β = β k and the following choices for A n , A , ˆ b , b :
[Equation]
This directly yields the following inequalities:
[Equation]
where ˜Π k and ˜ R zk are given by: [Equation]
− β k µ g ) N
[Equation]
First we have that ˜Π k ≤ Π k .
Moreover, Proposition 6, we have that (cid:107)
z (cid:63) ( x k , y k ) (cid:107) ≤ Bµ − 1 g hence, ˜ R zk ≤
R
zk thus yielding the desired inequalities.
Finally (15a) also follows similarly using (45) from
Proposition 15.
B. C ONTROLLING THE BIAS AND VARIANCE E ψk AND V ψk
Proof of Proposition 4 .
Recall that the expressions of E ψk and V ψk in (6) involves the conditional expectation E k (cid:104)
ˆ ψ k (cid:105) knowing x k , y k and z k − 1 .
This can be also expressed using Ψ as follows:
[Equation]
where we used the tower property for conditional expectations in the ﬁrst line, then the fact that the expectation of ψ k conditionally on x k , y k and z k is simply Ψ( x k , y k , z k ) .
Finally, for the last line, we use the independence of the noise and the linearity of Ψ w.r.t.
the last variable.
In all what follows, we write ¯ z k = E
[ z k | x k , y k , z k − 1 ] which is the same object as deﬁned in Proposition 7.
We then treat E ψk and V ψk separately.
Bounding E ψk .
Using Propositions 6 and 7 we directly get the desired inequality:
[Equation]
Bound on V ψk .
We decompose V ψk into a sum of three terms W k , W (cid:48) k and W (cid:48)(cid:48)
k given by:
[Equation]
where we used that ˜ ξ N +1 ,k is independent from z k and D f to get the last term.
Hence, using
Assumption 4 to bound the ﬁrst term of the above relation, we get V ψk ≤ σ 2 f + W (cid:48) k
+ 2 W (cid:48)(cid:48) k .
Thus, it remains to control each of W (cid:48)
k and W (cid:48)(cid:48) k .
Bound on W (cid:48)(cid:48) k .
Using that D f is independent from ˜ ξ n,k , we can apply Proposition 14 to write:
[Equation]
t (cid:33) ∂ y ( ˆ f − f )( x k , y k , D f ) (cid:35) where we used the simplifying notion ( ˆ f − f )( x k , y k , D f ) =
ˆ f ( x k , y k , D f ) − f ( x k , y k ) .
Using Assumption 3 to bound (cid:107) ∂
xy g ( x k , y k ) (cid:107) op by L (cid:48)
g , Assumption 1 to upper-bound (cid:13)(cid:13)(cid:13)(cid:16)(cid:80)
Nt
=
( I − β k A ) N − t (cid:17)(cid:13)(cid:13)(cid:13) op by (cid:16)(cid:80)
Nt =1 (1 − β k µ g ) N − t (cid:17) we get | W (cid:48)(cid:48) k
| ≤ β k L (cid:48)
g
N − 1 (cid:88) t
= (1 − β k µ g ) t E (cid:104)(cid:12)(cid:12)(cid:12) ∂
x ( ˆ f − f )( x k , y k , D f ) (cid:62) ∂
y ( ˆ f − f )( x k , y k , D f ) (cid:12)(cid:12)(cid:12)(cid:105) ≤ L (cid:48)
g µ − 1 g E (cid:104)(cid:12)(cid:12) (cid:12) ∂
x ( ˆ f − f )( x k , y k , D f ) (cid:62) ∂
y ( ˆ f − f )( x k , y k , D f ) (cid:12)(cid:12) (cid:12) (cid:105) ≤ L (cid:48)
g µ − 1 g E (cid:104)(cid:13)(cid:13)(cid:13)
∂
x (
[Table]
f − f )( x k , y k , D f ) 2 (cid:13)(cid:13)(cid:13)(cid:105)
[Table]
≤
L (cid:48) g µ − 1 g σ 2 f where we used that (cid:80) N − 1 t =
(1 − β k µ g ) ≤ 1 β k µ g for the second line, Cauchy-Schwarz inequality to get the third line and Assumption 4 to get the last line.
Bound on W (cid:48)
k Using that ˜ ξ N +1 ,k is independent from z k , we write:
[Table]
[Equation]
(i) follows from Assumptions 3 and 5, (ii) uses that (cid:107) z k (cid:107) [Equation] ≤ [Equation]
z
[Equation]
(cid:17) , (iii) uses that (cid:107)
z (cid:63) ( x k , y k ) (cid:107) ≤ Bµ − 1 g by Proposition 6.
Finally (iv) follows by application of Proposition 7.
We further have by deﬁnition of R zk that:
[Equation]
Combining the inequalities on W (cid:48) k , W (cid:48)(cid:48) k and (29), we get that V ψk ≤ w 2 x + σ 2 x Π k E
zk , with w 2 x and σ 2 x given by (16).
C G ENERAL ANALYSIS OF A M IGO C. A NALYSIS OF THE OUTER - LOOP
Proof of Proposition 2.
We treat both cases µ ≥ 0 and µ < 0 separately.
For simplicity we denote by E k the conditional expectation knowing the iterates x k , y k and z k − 1 and write ψ k = E k (cid:104)
ˆ ψ k (cid:105) .
Case µ ≥ 0 .
Recall that E xk is given by:
[Equation]
For simplicity deﬁne (cid:15) k
= uδ
k
+ (1 − u ) , e k
=
(1 − u )
( L ( x k ) − L (cid:63) )
+ η k 2 (cid:107) x
k − x (cid:63) (cid:107) 2 and e (cid:48)
k
= uδ k ( L ( x k ) − L (cid:63) ) .
It is then easy to see that E [ e k ] is equal to the l.h.s of (10), i.e. E [ e k ] =
E x k .
We will start by bounding the difference between two successive iterates of e k :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(i) follows from the update expression x k = x
k − 1 − γ k ˆ ψ k − 1 , (ii) follows from the convexity of L and (iii) follows by L -smoothness of L .
Taking the expectation conditionally on the randomness at iteration k − 1 and using that E k − 1 (cid:104)
ˆ ψ k − 1 (cid:105) =
ψ k − 1 , we therefore get
[Equation]
[-e k − 1 + γ k s k (cid:18) E k − 1 (cid:104)(cid:13)(cid:13)(cid:13) ˆ ψ k − 1 − ψ k − 1 (cid:13)(cid:13)(cid:13)(cid:105) 
 + (cid:107) ψ k − 1 − ∇L ( x k − 1 ) (cid:107) 2 (cid:19) − δ k ( ψ k − 1 − ∇L ( x k − 1 ))-][Table]
where (i) follows from δ k ≤ (cid:15) k since by construction δ k ≤ 1 .
Taking the expectation w.r.t.
all the randomness and applying Cauchy-Schwarz inequality to the last term yields the following inequality:
[Equation]
Since L is convex, we have the inequality: (cid:107) x k − 1 − γ k ∇L ( x k − 1 ) − x (cid:63) (cid:107) 2 ≤ (cid:107) x k − 1 − x (cid:63) (cid:107) 2 .
Hence, we can deduce that:
[Equation]
where we used that η k is non-increasing by construction.
Combining the above inequality with (30) yields:
[Equation]
Case µ < 0 .
Recall that for µ < 0 , we set E xk = 1 L E (cid:104) (cid:107)∇L ( x k ) (cid:107) 2 (cid:105) .
Using that L is L -smooth, we have that:
[Equation]
Taking the expectation w
[Equation]
r
[Equation]
t. all randomness in the algorithm in the above inequality
[Equation]
we get:
[Equation]
where we used that 1 − Lγ k 2 ≥ 12 and 0 ≤ 1 − Lγ k ≤ 1 to get the last inequality.
Using the deﬁnition of F k yields an inequality of the form:
[Equation]
Hence
[Equation]
in both cases µ ≥ 0 and µ < 0 we get an inequality of the of the same form, but with different expressions for F k and s k .
We get the desired result using Young’s inequality, to upper-bound the last term in the r.h.s.
of the above inequality.
More precisely, we use that for any 0 < ρ k < 1 :
[Equation]
C. I NNER - LEVEL ERROR BOUND
In this section we prove Proposition 5 which controls the evolutions of the warm-start errors E yk and
E zk .
As a ﬁrst step, in Proposition 12, we provide a result controlling the mean squared error between two successive iterates x
k − 1 , x k and y k − 1 , y k which will be used in the proof of Proposition 5.
Proposition 12 (Control of the increments of x k and y k ) .
Consider ζ k , φ k and ˜ R yk as deﬁned in
Proposition 8 for some ﬁxed 0 ≤ v ≤ 1 .
Then, the following holds:
[Equation]
Proof.
Proof of Proposition 12 We prove each inequality separately.
Increments of x k .
By the update equation, we have that x k = x k − 1 − γ k ˆ ψ k − 1 , hence we only need to control E (cid:20) (cid:13)(cid:13)(cid:13) ˆ ψ k − 1 (cid:13)(cid:13)(cid:13) 2 (cid:21) .
We have the following:
[Equation]
In the case ( µ < 0) , we have E xk − 1 = 12 L E (cid:104) (cid:107)∇L ( x k − 1 ) (cid:107) 2 (cid:105) , hence by setting ζ k :=
L , we get the desired inequality.
In the convex case ( µ ≥ 0) , since L is L -smooth, we have that: [Equation] u
[Equation]
provided that u < 1 .
We also have that ( L ( x k − 1 ) − L (cid:63) )
≤ L 2 (cid:107) x k − 1 − x (cid:63) (cid:107) 2 ≤ Lη − 1 k − 1 E xk − 1 which yields (cid:107)∇L ( x k − 1 ) (cid:107) 2 ≤ 2 L 2 η − 1 k − 1 E xk − 1 .
Hence, we can set ζ k = 2 L min (cid:0) (1 − u ) − 1 , Lη − 1 k − 1 (cid:1) .
Increments of y k .
Denoting by D tg a batch of samples at time iteration t of algorithm A k and using the update equation of y t we get the following inequality by application of the triangular inequality:
[Equation]
where we applied Proposition 7 for every 0 <t ≤ T − 1 to get the second line with Λ t,k :=(1 − α k µ g ) t .
This directly implies the following bound:
[Equation]
On the other hand
[Equation]
using a triangular inequality and applying Proposition
[Equation]
we also have that:
[Equation]
The result follows by combining (31) and (32) using coefﬁcients 1 − v and v .
C. P ROOF OF P ROPOSITION
Proof of Proposition 5 .
We will control each of E yk and E
z k separately.
Upper-bound on E yk .
Let r k be a non-increasing sequences between 0 and 1 .
The following holds:
[Equation]
(i) follows by Young’s inequality, (ii) uses Proposition 7 to bound the ﬁrst term and that (1+ r − 1 k ) ≤ 2 r − 1 k for the second term, (iii) uses that y (cid:63) is L y -Lipschitz by Proposition 6 and (iv) uses the update equation x k = x k − 1 − γ k ˆ ψ k − 1 .
Upper-bound on E zk .
Similarly, for a non-increasing sequence 0 < θ k ≤ 1 , we have that:
[Equation]
34a
[Equation]
(i) follows by Young’s inequality, (ii) uses Proposition 7 to bound the ﬁrst term and that (1+ θ − 1 k ) ≤ 2 θ − 1 k for the second term, (iii) uses that z (cid:63) ( x, y ) is L z -Lipschitz in x and y by Proposition 6 and, ﬁnally, (iv) uses the update equation x k = x k − 1 − γ k ˆ ψ k − 1 for the term E (cid:104) (cid:107) x k − x k − 1 (cid:107) 2 (cid:105) and
Proposition 12 to control the increments E (cid:104) (cid:107) y k − y k − 1 (cid:107) 2 (cid:105) .
In order to express the upper-bound on E zk in terms of E yk − 1 instead of E yk , we substitute E yk in (34a) by its upper-bound in (33a) and use that (1 + r k ) ≤ 2 to write:
[Equation]
We can then express (33a) and (35) jointly in matrix form as follows:
[Equation]
where the P k is a 2 × 2 matrix and U k and V k are 2 -dimensional vectors given by (17).
The desired result follows directly by substituting E (cid:20)(cid:13)(cid:13)(cid:13)
ˆ ψ k − 1 (cid:13)(cid:13)(cid:13) 2 (cid:21) by its upper-bound from Proposition 12 in the above inequality.
C. G ENERAL ERROR BOUND
Proof of Proposition 9.
First note that, by assumption, we have that δ k r − 1 k ≤ δ k − 1 r − 1 k − 1 and δ k θ − 1 k ≤ δ k − 1 θ − 1 k − 1 .
Moreover, since α k and β k are non-increasing, we also have that Λ k − 1 ≤ Λ k and Π k − 1 ≤ Π k .
This implies the following inequalities which will be used in the rest of the proof:
[Equation]
Now
[Equation]
let ρ k be a non-increasing sequence with 0 < ρ k < 1 .
By Proposition [Equation] it follows that E xk satisﬁes the inequality:
[Equation]
On the other hand
[Equation]
by Proposition 5 we know that E yk and E
zk satisfy:
[Equation]
where the P k , U k and V k are deﬁned in (17)
[Equation]
For conciseness, deﬁne S k and E Ik to be:
[Equation]
By (36), we directly have that:
[Equation]
where the inequality in (39) holds component-wise.
Therefore, multiplying (38) by S k and using (39) yields:
[Equation]
Furthermore, by Proposition 4 we can bound E ψk − 1 and V ψk − 1 as follows:
[Equation]
where we used (36) a second time to replace Λ k − 1 ( a k − 1 ) − 1 and Π k − 1 ( b k − 1 ) − 1 by Λ k ( a k ) − 1 and Π k ( b k ) − 1 .
By summing both inequalities (37) and (40) and substituting all terms E ψk − 1 and V ψk −by their upper-bounds we obtain an inequality of the form:
[Equation]
where A xk , A yk , A zk are the components of the vector
A k deﬁned in (18) and V totk is the variance term also deﬁned in (18).
The desired inequality follows by upper-bounding A xk , A yk , A zk by their maximum value (cid:107)
A k (cid:107) ∞ .
D C ONTROLLING THE PRECISION OF THE INNER - LEVEL ALGORITHMS .
In this section, we prove Proposition 10.
To achieve this, we ﬁrst provide general conditions on Λ k and Π k for controlling the rate (cid:107) A k (cid:107) ∞ and which hold regardless of the choice of step-sizes.
This is achieved in Proposition 13 of Appendix D.1.
Then we prove Proposition 10 in Appendix D.2.
D. C ONTROLLING Π k AND Λ k .
We introduce the following quantities:
[Equation]
Proposition 13.
Let ρ k be a non-increasing sequence of positive numbers smaller than 1 .
Consider Λ k and Π k so that:
[Equation]
Then, the following inequalities holds:
[Equation]
where A k and V totk are deﬁned in (18) of Proposition 9 and ˜ V totk is deﬁned as:
[Equation]
Proof.
We ﬁrst prove that u Ik ≤ u + k ≤ 1 and p + k ≤ 1 under (42a) and (42b) with u + k , p + k given by:
[Equation]
A direct calculation shows u + k ≤ 1 whenever (42a) holds.
Moreover, recall that u I k
= a k U (1) k
+ b k U (2) k with U (1) k and U (2)
k being the components of the vector U k deﬁned in (17).
Thus by direct substitution, we get the following expression for u Ik :
[Equation]
Therefore, (42b) sufﬁces to ensure that u Ik ≤ u + k .
Finally, (42b) implies directly that p + k ≤ 1 .
We will control each component A xk , A yk and A zk of the vector A k separately.
Controlling A xk .
Recalling the expression of A xk , the ﬁrst component of A k in (18), it holds that:
[Equation]
(i) holds since u I k ≤ u + k while (ii) follows from (42a) which ensures that 2 ζ k γ k u + k ≤ 1 2 ρ k δ k .
Controlling A yk .
Recall the expression of second component of A yk in (18), we have:
[Equation]
(i) holds since p + k :=
16 L 2 z Π sk θ − 2 k φ k ≤ 1 and u Ik ≤ 1 while (ii) is a consequence of (42a).
Controlling A zk .
Similarly, recalling the expression of the third component of A k we get that: A
[Equation]
where (i) uses that u
Ik ≤ 1 and (ii) follows from (42b).
Controlling V totk .
Recalling the expression of V totk from (18), we have that:
[Equation]
where we use 16 L 2 z φ k θ − 2 k Π sk ≤ 1 and u Ik ≤ 1 for the ﬁrst line and u Ik ≤ u + k for the last line.
D.
C ONTROLLING THE NUMBER OF INNER - LEVEL ITERATIONS
We provide now a proof of Proposition 10 which is a consequence of Proposition 13.
Proof of Proposition 10.
We ﬁrst provide conditions on the number of iterations T and N of algorithms
A k and B k to control the rate (cid:107) A k (cid:107) ∞ and then provide an upper-bound on V totk .
Conditions on T and N .
We consider the setting with constant step-size γ k = γ , α k
= α and β k = β
and choose r k = θ
k
= 1 and δ k
= δ 0 for some 0 <δ 0 < 1 .
We also take v =1 so that φ k =2 and ˜ R yk
= R yk .
By direct substitution of the parameters r k , θ k , φ k , γ k , δ k , ρ k and ζ k , in the expressions of D (1) k ,
D (2) k , D (3) k , D (4) k , D (5) k and D (6) k deﬁned in (41a) to (41f), we verify that:
[Equation]
Hence, we can ensure the conditions of Proposition 13 hold by choosing T and N so that:
[Equation]
This is achieved by for the following choice:
[Equation]
Hence, for such choice, we are guaranteed by Proposition 13 that (cid:107) A k (cid:107) ∞ ≤ 1 − (1 − ρ k ) δ k .
Bound on the variance V totk .
By choosing T and N as in (43), we know that Λ k and Π k satisfy (42), so that the variance term V totk is upper-bounded by ˜ V totk .
Moreover, by direct substitution of the sequences appearing in the expression of ˜ V totk by their values, we get:
[Equation]
Furthermore, by deﬁnition of R yk − 1 and R zk − 1 , we have that:
[Equation]
Moreover, recall that ˜ R yk
=
R yk since we chose v =1 .
Thus ˜ R yk ≤ 2 µ − 1 g ασ 2 g .
This implies that:
[Equation]
By choosing T and N as in (43)
[Equation]
the following conditions hold:
[Equation]
By applying these inequalities in (44)
[Equation]
we get:
[Equation]
where we used that 2 µ − 3 g (cid:16) σ 2 g xy + ( L (cid:48) g ) 2 (cid:17)(cid:16) B 2 βσ 2 g yy + 3 µ g σ 2 f (cid:17) ≤ w 2 x by deﬁnition of w 2 x in (16a).
Therefore, we have shown that V totk ≤ γδ 0 W 2 , with W 2 given by (20).
E S TOCHASTIC L INEAR DYNAMICAL SYSTEM WITH CORRELATED NOISE
Let A be a positive deﬁnite matrix in R d × R d satisfying 0 < µ g ≥ | σ i ( A ) | ≤ L g and b a vector in
R d .
We denote by z (cid:63) =
− A − 1 b .
Consider A
m be a sequence of i.i.d.
positive symmetric matrices in R d × R d such that E
[ A m ] =
A , and ˆ b a random vector in R d such that E (cid:104)
ˆ b (cid:105)
=
b , with A m and ˆ b being mutually independent.
Deﬁne Σ
A = E (cid:104)
( A n − A ) (cid:62) ( A n − A ) (cid:105) and denote by σ A and L A the largest singular values of Σ A and A − 1 Σ A A − 1 .
Let β be such that β ≤ 1 L g .
Finally let σ 2 c be an upper-bound on E (cid:20)(cid:13)(cid:13)(cid:13)
ˆ b − b (cid:13)(cid:13)(cid:13) 2 (cid:21) .
Let z and z (cid:48) be two vectors in R d and deﬁne the iterates z n and ¯ z n such
that z 0 =
z and ¯ z
= z (cid:48) and using the recursion:
[Equation]
Hence, from the deﬁnition of z n and ¯ z n we directly have that:
[Equation]
− βA
[Equation]
The next proposition computes the bias E (cid:104)
z n − ¯ z n (cid:12)(cid:12)(cid:12) ˆ b (cid:105) .
Proposition 14.
The following identities hold:
[Equation]
Proof.
The proof is a consequence of A n and ˆ b being i.i.d.
and unbiased estimates of A and b .
The next proposition controls the mean squared errors E (cid:104) (cid:107) z n − z (cid:63) (cid:107) 2 (cid:105) and E (cid:104) (cid:107) z n − ¯ z n (cid:107) 2 (cid:105) .
Proposition 15.
Deﬁne n (cid:63) ( n, β ) = min( n, 1 βµ g ) .
Let β such that:
[Equation]
Then, the following inequalities holds:
[Equation]
Moreover, if z (cid:48) =
z , then we have:
[Equation]
Proof.
It is straightforward to see that:
[Equation]
Now
[Equation]
let’s control E (cid:104) (cid:107) z n − ¯ z n (cid:107) 2 (cid:105) .
The following identity holds by deﬁnition of z n and ¯ z n : E
[Equation]
where Σ A = E (cid:104)
( A n − A ) (cid:62) ( A n − A ) (cid:105) and D n = (cid:80) n − 1 t =
( I − βA ) t .
By simple calculation we can upper-bound the last term by:
[Equation]
Moreover, provided that β ≤ 1 L g (1+ L A ) , where L A is the highest eigenvalue of A − 1 Σ A A − 1 , then we have the following:
[Equation]
Unrolling the recursion
[Equation]
it follows that:
[Equation]
In particular, if z (cid:48) =
z (cid:63) , then ¯ z n = z (cid:63) and we get:
[Equation]
To get the last inequality [Equation] we simply choose z (cid:48)
=
z and recall that:
[Equation]
Using the above in in (46) yields:
[Equation]
Moreover, by Lemma 3 we know that nβ 2 µ 2 g (1 − βµ g ) n − 1 ≤ (1 − βµ g 2 ) n − 1 and since βµ g ≤ 1 , we have that (1 − βµ g 2 ) − 1 ≤ 2 so that (1 − βµ g 2 ) n − 1 ≤ 2(1 − βµ g 2 ) n .
Hence, we can write:
[Equation]
Lemma 2.
Let A and Σ A be symmetric positive matrix in R d × R d with σ 2 A its largest singular value of Σ A and 0 < µ g ≤ σ i ( A )
≤ L g .
Let β be a positive number such that:
[Equation]
Then the following holds:
[Equation]
Proof.
First note that β ≤ 1 L g , so that I − βA is positive.
Now, we observe that (cid:13)(cid:13)
( I − βA ) 2 + β 2 Σ A (cid:13)(cid:13) op ≤ (1 − βµ g ) 2 + β 2 σ 2 A which holds since I − βA is positive.
And since β ≤ µ [Equation] µ 2 g + σ 2 A , we further have (1 − βµ g ) 2 + β 2 σ 2 A ≤ 1 − βµ g , which yields the desired result.
Lemma 3.
Let 0 ≤ b < 1 and n ≥ 1 , then the following inequality holds:
[Equation]
Proof.
We consider the function h ( n, b ) deﬁned by:
[Equation]
We need to show that h ( n, b ) is non-negative for any n ≥ 1 and 0 ≤ b < 1 .
For this purpose, we ﬁx b and consider the variations of h ( n, b ) in n :
[Equation]
∂ n h ( n, b ) is non-negative for n ≥ n (cid:63)
= log (cid:16) 1 − b 2 1 − b (cid:17) − 1 and non-positive for all n ≤ n (cid:63) .
Hence, h ( n, b ) achieves its minimum value in n (cid:63) over the (0 , + ∞ ) .
We distinguish two case depending on whether n (cid:63) is greater of smaller than 1 .
Case n (cid:63) ≤ 1 .
In this case n (cid:55)→ h ( n, b ) is increasing on the interval [1 , + ∞ ) since ∂ n h ( n, b ) ≥for n ≥ n (cid:63) .
Hence, h ( n, b ) ≥ h (1 , b ) for all n ≥ 1 .
Moreover, since h (1 , b ) = − log( b 2 ) ≥
the result follows directly.
Case n (cid:63) > 1 .
In this case we still have h ( n, b ) ≥ h ( n (cid:63) , b ) for all n ≥ 1 , since n (cid:63) achieves the minimum value of h .
Thus we only need to show that h ( n (cid:63) , b ) ≥ 0 .
Using the expression of n (cid:63) , we have:
[Equation]
Since n (cid:63) > 1 , the ﬁrst term 1 − 1 n (cid:63) is non-negative, thus we only need to show that n (cid:63) b 2 ≤ 1 so that the last term is also non-negative.
It is easy to see that n (cid:63) b 2 ≤ 1 is equivalent to having ˜ h ( b ) ≥ 0 , where we deﬁne the function ˜
h ( b ) as:
[Equation]
We can analyze the variations of ˜ b be computing its derivative which is given by:
[Equation]
Hence, we have the following equivalence:
[Equation]
This is always true for 0 ≤ b < 1 since b (1 − b ) ≤ 14 so that 2 b (1 − b )(2 − b ) ≤ 2 − b 2 ≤ 1 .
Thus we have shown that ˜ h is increasing over [0 , 1) so that ˜ h ( b ) ≥ ˜ h (0)
= 0 .
As discussed above, this is equivalent to having n (cid:63) b 2 ≤ 1 , so that h ( n (cid:63) , b ) ≥ 0 which concludes the proof.
F E XPERIMENTS
F. D ETAILS OF THE SYNTHETIC EXAMPLE
We choose the functions f
and g to be of the form: f ( x, y ) :
= 12 x (cid:62) A f x + y (cid:62) C f
and g ( x, y ) :
= 12 y (cid:62) A g y + y (cid:62) B g x where A f and A g are symmetric deﬁnite positive matrices of size d x × d x and d y × d y , B g is a d y ×
d x matrix and C f is a d y vector with d x = 2000 and d y = 1000 .
We generate the parameters of the problem so that the smoothness constants L and L g are ﬁxed to 1 , κ L =10 and κ g taking values in { 10 i , i ∈ { 0 , .., 7 }} .
We then solve each problem using different methods and perform a grid-search on the number of iterations T and M of algorithms A k and B k .
We ﬁx the step-sizes to γ k =1 /L and α k = β k =
/L
g
and perform a grid-search on the number of iterations T and M of algorithms
A k and B k from { 10 i , i ∈ 0 , 1 , 2 , 3 } .
For AID methods without warm-start in B k , we consider an additional setting where M increases logarithmically with k , as suggested in Ji et al.
(2021), with M = (cid:98) 10 3 log( k ) (cid:99) .
Similarly, for (ITD) and (Reverse), we additionally use an increasing T of the same form.
F. E XPERIMENTAL DETAILS FOR L OGISTIC REGRESSION
The inner-level and outer-level cost functions for such task take the following form:
[Equation]
For the default setting, we use the well-chosen parameters reported in Grazzi et al.
; Ji et al.
(2021) where α k = γ k =100 , β k =0 .
5 , and T = N =10 .
For the grid-search setting, we select the best performing parameters T , M and β k from a grid { 10 , 20 } × { 5 , 10 } × { 0 .
5 , 10 } , while the batchsize (chosen to be the same for all steps of the algorithms) varies from 10 ∗ { 0 . 1 , 1 , 2 , 4 } .
We also compared with VRBO (Yang et al., 2021) using the implementation available online and noticed instabilities for large values of T and N , as reported by the authors, but also a drop in performance compared to stocBiO for smaller T and N due to inexact estimates of the gradient.
[Figure] [Figure]
κ g ∈ { 10 4 , 10 5 , 10 7 } .
For each method we consider T ∈ { 1 , 10 , 10 2 , 10 3 } .
Lightest colors correspond to smaller values of T .
F. D ATASET DISTILLATION
Dataset distillation (Wang et al., 2018; Lorraine et al., 2020) consists in learning a small synthetic dataset such that a model trained on this dataset achieves a small error on the training set.
Speciﬁcally, we consider a classiﬁcation problem of C classes using a linear model and a training dataset
D tr where each training point ξ ∈ D tr is a d -dimensional vector with a class c ξ ∈ { 1 , ..., C } .
The linear model is represented by a matrix
y ∈ R c × d multiplying a data point yξ and providing the logits of each class.
The dataset distillation can be cas as a bilevel problem of the form:
[Equation]
where λ ∈ R d is a vector of hyper-parameter for regularizing the inner-level problem which we found beneﬁcial to add.
Experimental setup.
We perform the distillation task on MNIST dataset.
We set the stepsizes α k = β k =0 .
1 and T = N =10 .
We perform a grid-search on the outer-level step-size γ k ∈{ 0 . 01 , 0 . 001 , 0 . 0001 } and run the algorithms for k = 10000 iterations.
[Figure]
[Figure]