A framework for bilevel optimization that enables stochastic and global variance reduction algorithms
Abstract
Bilevel optimization, the problem of minimizing a value function which involves the arg-minimum of another function, appears in many areas of machine learning.
In a large scale empirical risk minimization setting where the number of samples is huge, it is crucial to develop stochastic methods, which only use a few samples at a time to progress.
However, computing the gradient of the value function involves solving a linear system, which makes it difficult to derive unbiased stochastic estimates.
To overcome this problem we introduce a novel framework, in which the solution of the inner problem, the solution of the linear system, and the main variable evolve at the same time.
These directions are written as a sum, making it straightforward to derive unbiased estimates.
The simplicity of our approach allows us to develop global variance reduction algorithms, where the dynamics of all variables is subject to variance reduction.
We demonstrate that SABA, an adaptation of the celebrated SAGA algorithm in our framework, has O ( 1 T ) convergence rate, and that it achieves linear convergence under Polyak-Łojasciewicz assumption.
This is the first stochastic algorithm for bilevel optimization that verifies either of these properties.
Numerical experiments validate the usefulness of our method.
Introduction
Bilevel optimization is attracting more and more attention in the machine learning community thanks to its wide range of applications.
Typical examples are hyperparameters selection [5, 38, 17, 6], data augmentation [11, 42], implicit deep learning [3] or neural architecture search [33].
Bilevel optimization aims at minimizing a function whose value depends on the result of another optimization problem:
[Equation]
such that z
[Equation]
where F and G are two real valued functions defined on R p × R d .
G is called the inner function , F is the outer function and h is the value function .
Similarly, z is the inner variable and x is the outer variable .
In most cases, the function z ∗ can only be approximated by an optimization algorithm, which makes bilevel optimization problems challenging.
Under appropriate hypotheses, the function h is differentiable, and the chain rule and implicit function theorem give for any x ∈ R d
[Equation]
where v ∗ ( x ) ∈ R p is the solution of a linear system
[Equation]
In the light of (2) and (3), it turns out that the derivation of the gradient of h at each iteration is cumbersome because it involves two subproblems: the resolution of the inner problem to find an approximation of z ∗ ( x ) and the resolution of a linear system to find an approximation of v ∗ ( x ) .
It makes the practical implementation of first order methods like gradient descent for (1) challenging.
[Figure]
As is the case in many machine learning problems, we suppose in this paper that F and G are empirical means:
[Equation]
This structure suggests the use of stochastic methods to solve (1).
For single-level problems (that is, classical optimization problems where one function should be minimized), using Stochastic Gradient Descent (SGD; [41, 7]) and variants is natural because individual gradients are straightforward unbiased estimators of the gradient.
In the bilevel framework, we want to develop algorithms that make progress on problem (1) by using only a few functions F j and G i at a time.
However, since ∇ h involves the inverse of the Hessian of G , building such stochastic algorithms is quite challenging, one of the difficulties being that there is no straightforward unbiased estimator of ∇ h .
Still, in settings where m or n are large, where computing even a single evaluation of F or G is extremely expensive, stochastic methods are the only scalable algorithms.
Variance reduction [27, 13, 43, 15, 12] is a popular technique to obtain fast stochastic algorithms.
In a single-level setting, these methods build an approximation of the gradient of the objective function using only stochastic gradients.
Contrary to SGD, the variance of the approximation goes to 0 as the algorithm progresses, allowing for faster convergence.
For instance, the SAGA method [13] achieves linear convergence if the objective function satisfies a Polyak-Łojasciewicz inequality, and O ( 1 T ) convergence rate on smooth non-convex functions [40].
The extension of these methods to bilevel optimization is a natural idea to develop faster algorithms.
However, this idea is hard to implement because it is hard to derive unbiased estimators of ∇ h , let alone variance reduction ones.
Contributions.
We introduce a novel framework for bilevel optimization in Section 2, where the inner variable, the solution of the linear system (3) and the outer variable evolve jointly.
The evolution directions are written as sums of derivatives of F j and G i , which allows us to derive simple unbiased stochastic estimators.
In this framework, we propose SOBA, an extension of
SGD (Section 2.1), and SABA (Section 2.2), an extension of the variance reduction algorithm
SAGA [13].
In Section 3 we analyse the convergence of our methods.
SOBA is shown to achieve inf t ≤ T E [ ∥∇ h ( x t ) ∥ 2 ]
=
O (log( T ) T − 1 2 ) with decreasing step sizes.
We prove that SABA with fixed step sizes achieves 1 T (cid:80)
Tt =1 E [ ∥∇ h ( x t ) ∥ 2 ]
= O ( 1 T ) .
SABA is therefore, to the best of our knowledge, the first stochastic bilevel algorithm that matches the convergence rate of gradient descent on h .
We also prove that SABA achieves linear convergence under the assumption that h satisfies a Polyak-Łojasciewicz inequality.
To the best of our knowledge, SABA is also the first stochastic bilevel algorithm to feature such a property.
Importantly, these rates match the rates of the single level counterparts of each algorithm in non-convex setting (SGD for SOBA and SAGA for SABA).
Finally, in Section 4, we provide an extensive benchmark of many stochastic bilevel methods on hyperparameters selection and data hyper-cleaning, and illustrate the usefulness of our approach.
Related work.
The bilevel optimization problem has a strong history in the optimization community, taking root in game theory [45].
Gradient-based algorithms to solve (1) can be mainly classified in two different categories depending on how ∇ h is computed, by automatic or implicit differentiation .
Since the solution of the inner problem z ∗ ( x ) is approximated by the output of an iterative algorithm, it is possible to use automatic differentiation [46, 31] to approximate ∇ h ( x ) .
It consists in differentiating the different steps of the inner optimization algorithm – see [4] for a review – and has been applied successfully to several bilevel problems arising in machine learning [14, 16].
One of the main drawbacks of this approach is that it requires to store in memory each iterate of the inner optimization algorithm, although this problem can sometimes be overcome using invertible optimization algorithms [34] or truncated backpropagation [44].
The use of the implicit function theorem to obtain (2) and (3) is known as implicit differentiation [5].
While the cost of computing exactly (2) can be prohibitive for large scale problems, Pedregosa [38] showed that we can still converge to a stationary point of the problem by using approximate solutions of the inner problem and linear system (3), if the approximation error goes to 0 sufficiently quickly.
The complexity of approximate implicit differentiation has been studied in [20].
Ramzi et al.
[39] propose to reuse the computations done in the forward pass to approximate the solution of the linear system (3) when the inner problem is solved thanks to a quasi-Newton method.
In the last few years, several works have proposed different strategies to solve (1) in a stochastic fashion.
A first set of methods relies on two nested loops : one inner loop to solve the inner problem with a stochastic method, and one outer loop to update the outer variable with an approximate gradient direction.
In [19, 26, 9] the authors use several SGD iterations for the inner problem and then use stochastic Neumann approximations to get an estimate solution of the linear system, which provides them with an approximation of ∇ h used to update x .
The analysis of this kind of method was refined by Chen et al.
[9], allowing to achieve the same convergence rates as those of SGD.
The convergence of the hypergradient when using stochastic solvers for the inner problem and the linear system has been studied in [21].
Arbel and Mairal [2] replace the Neumann approximation by SGD steps to estimate (3).
Other authors have proposed single loop algorithms, alternating steps in the inner and the outer problem.
Hong et al.
[24] propose to perform Neumann approximations of the inverse Hessian and use a single SGD step for the inner problem.
It was refined in [23] and [47] where the optimization procedure uses a momentum acceleration.
Other variations around this idea include [25, 28, 10, 22, 30].
We refer to Table 1 in appendix for a detailed comparison of these methods.
Notation.
The set of integers between 1 and n (included) is denoted [ n ] .
For f : R p × R d → R , we denote ∇ i f
( z, x ) its gradient w.r.t.
the i th variable.
The Hessian of f with respect to the first variable is denoted ∇ 211 f ( z, x ) ∈ R p × p , and the cross-derivatives matrix is ∇ 221 f ( z, x ) ∈ R d × p .
If v is a vector, ∥ v ∥ is its Euclidean norm.
If M is a matrix, ∥ M ∥ is its spectral norm.
A function is said to be L -smooth, for L > 0 , if it is differentiable, and its gradient is L -Lipschitz.
Proposed framework
Input: initializations z 0 ∈ R p , x 0 ∈ R d , v 0 ∈ R p , number of iterations T , step size sequences ( ρ t ) t<T and ( γ t ) t<T .
for t = 0 , . . .
, T − 1 do
Update z : z t +1 =
z t − ρ t D tz ,
Update v : v t +1 = v t − ρ t D tv ,
Update x : x t +1 = x
t − γ t D tx , where D tz , D tv and D tx are unbiased estimators of D z ( z t , v t , x t ) , D v ( z t , v t , v t ) and D x ( z t , v t , x t ) .
end for
In this section, we introduce our framework in which the solution of the inner problem, the solution of the linear system (3) and the outer variable all evolve at the same time, following directions that are written as a sum of derivatives of F j and G i .
We define
[Equation]
These directions are motivated by the fact that we have ∇ h ( x ) =
D x
( z ∗ ( x ) , v ∗ ( x ) , x ) , with z ∗ ( x ) the minimizer of G ( · , x ) and v ∗ ( x ) the solution of ∇ 211 G ( z ∗ ( x ) , x ) v =
−∇ 1 F ( z ∗ ( x ) , x ) .
When x is fixed, we approximate z ∗ by doing a gradient descent on G , following the direction
− D z
( z, v, x ) .
Finally, when z and x are fixed, we find v ∗ by following the direction − D v ( z, v, x ) , which corresponds to a gradient descent on v (cid:55)→ 12 ⟨∇ 211 G ( z, x ) v, v ⟩ + ⟨∇ 1 F ( z, x ) , v ⟩ .
The rest of the paper is devoted to the study of the global dynamics where the three variables z, v and x evolve at the same time, following stochastic approximations of D z , D v and D x .
The next proposition motivates the choice of these directions.
Proposition 2.1.
Assume that for all x ∈ R d , G ( · , x ) is strongly convex.
If ( z, v, x ) is a zero of ( D z , D v , D x ) , then z
= z ∗ ( x ) , v = v ∗ ( x ) and ∇ h ( x ) = 0 .
We also note that the computation of these directions does not require to compute the matrices ∇ 211 G ( z, x ) and ∇ 221 G ( z, x ) : we only need to compute their product with a vector, which can be computed at a cost similar to that of computing a gradient.
The framework we propose is summarized in Algorithm 1.
It consists in following a joint update rule in ( z, v, x ) that follows directions D tz , D tv and D tx that are unbiased estimators of D z , D v , D x .
The first and most important remark is that whereas ∇ h cannot be written as a sum over samples, the directions D z , D v and D x involve only simple sums, since their expressions are “linear” in F and G :
[Equation]
It is therefore straightforward to derive unbiased estimators of these directions.
In [30], the authors considered one particular case of our framework, where each direction is estimated by using the
STORM variance reduction technique (see [12]).
Taking a step back by proposing the framework summarized in Algorithm 1 opens the way to potential new algorithms that implement other techniques that exist in stochastic single level optimization.
In what follows, we study two of them.
2.1 First example: the SOBA algorithm
The simplest unbiased estimator is obtained by replacing each mean by one of its terms chosen uniformly at random, akin to what is done in classical single-level SGD.
We call the resulting algorithm SOBA (StOchastic Bilevel Algorithm).
To do so, we choose two independent random indices i ∈ [ n ] and j ∈ [ m ] uniformly and estimate each term coming from G using G i and each term coming from F using F j .
This gives the unbiased SOBA directions
[Equation]
This provides us with a first algorithm, SOBA, where we plug Equations (10a) to (10c) in Algorithm 1.
We defer its analysis to the next section.
Importantly, we use different step sizes for the update in ( z, v ) and for the update in x .
We use the same step size in z and in v since the inner problem and the linear system have similar conditioning, which is that of ∇ 211 G ( z t , x t ) .
The need for a different step size for the outer and inner problem is clear: both problems can have a different conditioning.
An important remark for SOBA is that all the stochastic directions used are computed at the same point z t , v t and x t with the same indices ( i, j ) .
The update of z , v and x can thus be performed in parallel instead of sequentially, benefiting from hardware parallelism.
Moreover, this enables to share the computations between the different directions.
This is the case in hyperparameters selection where G i ( z, x ) =
ℓ i ( ⟨ z, d i ⟩ ) + x 2 ∥ z ∥ 2 , with d i a training sample, and ℓ i that measures how good is the prediction ⟨ z, d i ⟩ .
In this setting, we have ∇ 1 G i ( z, x ) =
ℓ ′ i ( ⟨ z, d i ⟩ ) d
i + xz and ∇ 211 G i ( z, x ) v = ℓ ′′ i ( ⟨ z, d i ⟩ ) ⟨ v, d i ⟩ d i .
The prediction ⟨ z, d i ⟩ can thus be computed only once to obtain both quantities.
For more complicated models, where automatic differentiation is used to compute the different derivatives and Jacobian-vector products, we can store the computational graph only once to compute at the same time ∇ 1 G i ( z, x ) , ∇ 2 11 G i ( z, x ) v and ∇ 2 21 G i ( z, x ) v , requiring only one backward pass, thanks to the R technique [37].
Finally, like all single loop bilevel algorithms, our method updates at the same time the inner and outer variable, avoiding unnecessary optimization of the inner problem when x is far from the optimum.
2.2 Global variance reduction with the SABA algorithm
In classical optimization, SGD fails to reach optimal rates because of the variance of the gradient estimator.
Variance reduction algorithms aim at reducing this variance, in order to follow directions that are closer to the true gradient, and to achieve superior practical and theoretical convergence.
In our framework, since the directions D z , D v and D x are all written as sums of derivatives of F j and
G i , it is easy to adapt most classical variance reduction algorithms.
We focus on the celebrated SAGA algorithm [13].
The extension we propose is called SABA (Stochastic Average Bilevel Algorithm).
The general idea is to replace each sum in the directions D by a sum over a memory, updating only one term at each iteration.
To help the exposition, we denote y =
( z, x, v ) the vector of joint variables.
Since we have sums over i and over j , we have two memories for each variable: w ti for i ∈ [ n ] and ˜ w tj for j ∈ [ m ] , which keep track of the previous values of the variable y .
At each iteration t , we draw two random independent indices i ∈ [ n ] and j ∈ [ m ] uniformly and update the memories.
To do so, we put w t
+1 i =
y t and w t
+1 i ′ =
w ti ′ for
i ′
̸
= i , and ˜ w t +1 j = y t and ˜ w t +1 j ′ =
˜ w tj ′ for j ′ ̸ = j .
Each sum in the directions D is then approximated using SAGA-like rules: given n functions ϕ
i ′ for i ′ ∈ [ n ] , we define S [ ϕ, w ] ti = ϕ
i ( w t +
i
[Equation]
This is an unbiased estimators of the average of the ϕ ’s since E i (cid:104) S
[ ϕ, w ] ti (cid:105) = 1 n (cid:80) ni
=1 ϕ
[Equation]
y
[Equation]
With a slight abuse of notation, we call ∇
Gv the sequence of functions ( y (cid:55)→ ∇ 211 G i ( z, x ) v )
i ∈
[ n ] and ∇ 221 Gv the sequence of functions ( y (cid:55)→ ∇ 221 G i ( z, x ) v ) i ∈ [ n ] .
We define the SABA directions as
[Equation]
These estimators are unbiased estimators of the directions
D z , D v and D x .
The SABA algorithm corresponds to Algorithm 1 where we use Equations (11a) to (11c) as update directions.
When taking a step size γ t = 0 in the outer problem, hereby stopping progress in x , we recover the iterations of the SAGA algorithm on the inner problem.
In practice, the sum in S is computed by doing a rolling average (see Appendix B for precision), and the quantities ϕ
i ( w ti ) are stored rather than recomputed: the cost of computing the SABA directions is the same as that of SGD.
It requires an additional memory for the five quantities, of total size n × p + ( n + m ) ×
( p + d ) floats that can be reduced by using larger batch sizes.
Indeed, if b in and b out are respectively the inner and the outer batch sizes, the memory load is reduced to n b × p + ( n b + m b ) × ( p × d ) with n b = ⌈ nb inn ⌉ and m b = ⌈ mb out ⌉ which are smaller than the number of samples.
This memory load can also be reduced in specific cases, for instance when G and F correspond to linear models, where the individual gradients and Hessian-vector products are proportional to the samples.
In this case, we only store the proportionality ratio, reducing the memory load to 3 n + 2 m floats.
Like for SOBA, the computations of the new quantities ϕ i ( w t +1 i ) are done in parallel, thus benefiting from hardware acceleration and shared computations.
Despite this memory load, using SAGA-like variance reduction instead of
STORM as done in [30, 47, 28] has the advantage to bring the variance of the estimate directions to zero, enabling faster O ( 1 T ) convergence.
In the next section, we show that SABA is fast.
It essentially has the same properties as SAGA: despite being stochastic, it converges with fixed step sizes, and reaches the same rate of convergence as gradient descent on h .
Theoretical analysis
In this section, we provide convergence rates of SOBA and SABA under some classical assumptions.
Note that, unlike most of the stochastic bilevel optimization papers, we work in finite sample setting rather than the more general expectation setting.
Actually, SABA does not make any sense for functions that don’t have a finite sum structure.
However, we stress that SOBA could be studied in a more general setting to obtain the same bounds as here.
Also, the finite sum setting is still interesting since doing empirical risk minimization is very common in practice in machine learning.
The proofs and the constants in big-
O are deferred in Appendix C.
3.1 Background and assumptions
We start by stating some regularity assumptions on the functions F and G .
Assumption 3.1.
The function F is twice differentiable.
The derivatives ∇ F and ∇ 2 F are Lipschitz continuous in ( z, x ) with respective Lipschitz constants L F 1 and L F 2 .
Note that the above assumption is typically verified in the machine learning context, e.g., when F is the ordinary least squares (OLS) loss or the logistic loss.
Assumption 3.2.
The function G is three times continuously differentiable on R p × R d .
For any x ∈ R d , G ( · , x ) is µ G -strongly convex.
The derivatives ∇ G , ∇ 2 G and ∇ 3 G are Lipschitz continuous in ( z, x ) with respective Lipschitz constants L G 1 , L G 2 and L G 3 .
Strong convexity and smoothness with respect to z of G are verified when G is a regularized leastsquares/logistic regression with a full rank design matrix, when the data is not separable for the logistic regression.
Moreover, the strong convexity ensures the existence and uniqueness of the inner optimization problem for any x ∈ R d .
Assumption 3.3.
There exists C F > 0 such that for any x we have ∥∇ 1 F ( z ∗ ( x ) , x ) ∥ ≤ C F .
This assumption, combined with the strong convexity of G ( · , x ) , shows boundedness of v ∗ .
This assumption holds, for instance, in the case of hyperparameters selection for a Ridge regression problem.
Note that in Assumptions 3.1 and 3.2, we assume more regularity of F and G than in stochastic bilevel optimization literature (see for instance [19, 24, 26, 2]).
It is necessary to get the smoothness of v ∗ which will allow to adapt the proof of Chen et al.
[9] and get tight convergence rates.
The following lemma gives us some smoothness properties of the considered directions that will be useful to derive convergence rates of our methods.
Lemma 3.4.
Under the Assumptions 3.1 to 3.3, there exist constants L z , L v and L x such that
[Equation]
In first order optimization, a fundamental assumption on the objective function is the smoothness assumption.
In the case of vanilla gradient descent applied to a function f , it allows to get a convergence rate of ∥∇ f ( x t ) ∥ 2 in O (1 /T ) , i.e. convergence to a stationary point [36].
The following lemma proved by Ghadimi and Wang [19, Lemma 2.2] ensures the smoothness of h .
Lemma 3.5.
Under the Assumptions 3.1 to 3.3, the function h is L h -smooth for some L h > 0 .
The constant L h is specified in Appendix C.3.
As usual with the analysis of stochastic methods, we define the expected norms of the directions V tz =
E
[ ∥ D tz ∥ 2 ] , V tv =
E [ ∥ D tv ∥ 2 ] and V tx =
E
[ ∥ D tx ∥ 2 ] , where the expectation is taken over the past.
Thanks to variance-bias decomposition, they are the sum of the variance of the stochastic direction and the squared-norm of the unbiased direction.
For
SOBA, we use classical bounds on variances like those found for instance in [24]:
Assumption 3.6.
There exists
B z ,
B v and
B x such that for all t
[Equation]
where E t denotes the expectation conditionally to ( z t , v t , x t ) .
For SOBA and SABA, we need to bound the expected norm of D tx .
For SABA, this assumption allows to get a the same sample complexity as SAGA for single level problems.
Assumption 3.7.
There exists B x such that for all t , E t [ ∥ D tx ∥ 2 ] ≤ B 2 x .
Assumptions 3.6 and 3.7 are verified for instance, if all the G i and ∇ 1 G i have at most quadratic growth, and if F has bounded gradients.
They are also verified if the iterates remain in a compact set.
Note that we do not assume that G has bounded gradients, as this would contradict its strong-convexity.
Finally, for the analysis of SABA, we need regularity on each G i and F j :
Assumption 3.8.
For all i ∈ [ n ] and j ∈ [ m ] , the functions ∇ G i , ∇ F j , ∇ 211 G i and ∇ 221 G i are
Lipschitz continuous in ( z, x ) .
3.2 Fundamental descent lemmas
Our analysis for SOBA and SABA is based on the control of both δ tz = E
[ ∥ z t − z ∗ ( x t ) ∥ 2 ] and δ tv =
E [ ∥ v t − v ∗ ( x t ) ∥ 2 ] , Strong convexity of G and smoothness of z ∗ ( x ) and v ∗ ( x ) allow to obtain the following lemma by adapting the proof of Chen et al.
In what follows, we drop the dependency of the step sizes ρ and γ in t for clarity.
Lemma 3.9.
Assume that γ 2 ≤ min (cid:16) µ G L 2 ∗ 4 B 2 x L 2 zx , µ G L 2 ∗ 8 B 2 x L 2 vx (cid:17) ρ .
We have:
[Equation]
where β zx = β vx = 3 L 2 ∗ , β zx = 8 L 2 ∗ µ G , β vx =
L 2 ∗ µ G , L ∗ is the maximum between the Lipschitz constants of z ∗ and v ∗ (see Lemma C.1), β vz =
µ 3 G ( L F 1 µ G + L G 2 ) 2 , L zx and L vx are respectively the smoothness constants of z ∗ and v ∗ .
We insist that this result is obtained in general for Algorithm 1 with arbitrary unbiased directions.
We can therefore invoke this lemma for the analysis of both SOBA and SABA.
We use the smoothness of h to get the following lemma, which is similar to [9, Lemma 1].
Lemma 3.10.
Let h t =
E
[ h ( x t )] and g t = E [ ∥∇ h ( x t ) ∥ 2 ] .
We have
[Equation]
If z t = z ∗ ( x t ) , v t
=
v ∗ ( x t ) , that is δ z , δ v both cancel and D x ( z t , v t , x t )
= ∇ h
( x t ) , we get an inequality reminiscent of the smoothness inequality for SGD on h .
3.3 Analysis of SOBA
The analysis of SOBA is based on Lemmas 3.5 and 3.9.
We have the following theorem, with fixed step sizes depending on the number of iterations:
Theorem 1 (Convergence of SOBA, fixed step size) .
Fix an iteration T > 1 and assume that
Assumptions 3.1 to 3.7 hold.
We consider fixed steps ρ t = ρ √ T and γ t = ξρ t with ρ and ξ precised in the appendix.
Let ( x t ) t ≥ 1 the sequence of outer iterates for SOBA.
Then,
[Equation]
As opposed to [24], we do not need that the ratio γρ goes to 0, which allows to get a complexity (that is, the number of call to oracles to have an ϵ -stationary solution) in O ( ϵ − 2 ) better than the ˜ O
( ϵ − 52 ) they have.
Also, note that this rate is the same as the one of SGD for non-convex and smooth objective [18, 8].
We obtain a similar rate using decreasing step sizes:
Theorem 2 (Convergence of SOBA, decreasing step size) .
Assume that Assumptions 3.1 to 3.7 hold.
We consider steps ρ t = ρt − 12 and γ t = ξρ .
Let x t the sequence of outer iterates for SOBA.
Then,
[Equation]
As for SGD, SOBA suffers from the need of decreasing step sizes to get actual convergence because of the variance of the estimation on each directions.
On the other hand, the analysis of SABA leverages the dynamic of all three variables, resulting in fast convergence with fixed step sizes.
3.4 SABA: a stochastic method with optimal rates
In what follows, we denote N = n + m the total number of samples.
The following theorem shows O ( N 23 T − 1 ) convergence for the SABA algorithm in the general case where we only assume smoothness of h .
Our analysis of SABA is inspired by the analysis of single-level SAGA by Reddi et al.
Theorem 3 (Convergence of SABA, smooth case) .
Assume that Assumptions 3.1 to 3.3 and 3.7 to 3.8 hold.
We suppose ρ = ρ ′ N − 23 and γ = ξρ , where ρ ′ and ξ depend only on F and G and are specified in appendix.
Let x t the iterates of SABA.
Then,
[Equation]
To prove the theorem, the idea is to control the distance from the memory to the current variables.
We define S t = 1 n (cid:80) ni
=1 ∥
y t − w ti ∥ 2 + 1 m (cid:80) mj
=1 ∥
y t − ˜ w tj ∥ 2 .
In appendix, we show that we can find scalars ϕ s , ϕ z , ϕ v > 0 such that the quantity L t = h t
+ ϕ s S t + ϕ z δ tz + ϕ v δ t v satisfies L t +1 ≤ L t − γ 2 g t .
Summing these inequalities for t = 1 . . .
T and using the fact that L t is lower bounded demonstrates the theorem.
Note that the step sizes are constant with respect to the time, but they scale with N − 23 .
As a consequence, the sample complexity is O ( N 23 ϵ − 1 ) which is analogous of the one of SAGA for non-convex single level problems [40].
This is better than the sample complexity of Algorithm 1 with full batch directions, which is O
( Nϵ − 1 ) .
Hence, with SABA, we get the best of both worlds: the stochasticity makes the scaling in N of the sample complexity goes from N in full batch mode
to
N 23 for SABA, and the variance reduction makes the scaling in ϵ goes from ϵ − 2 for SOBA to ϵ − 1 for
SABA.
Our experiments in Section 4 confirm this gain.
Furthermore, if we assume that h satisfies a Polyak-Łojasiewicz (PL) inequality, we recover linear convergence.
Recall that h has the PL property if there exists µ h > 0 such that for all x ∈ R d , 12 ∥∇ h ( x ) ∥ 2 ≥ µ h ( h ( x ) − h ∗ ) with h ∗ the minimum of h .
Theorem 4 (Convergence of SABA, PL case) .
Assume that h satisfies the PL inequality and that
Assumptions 3.1 to 3.3 and 3.7 to 3.8 hold.
We suppose ρ = ρ ′ N − 23 and γ = ξρ ′ N − 1 , where ρ ′ and ξ depend only on F and G and are specified in appendix.
Let x t the iterates of SABA and c ′ ≜ min (cid:0) µ h , 1 16 P ′ (cid:1) with P ′ specified in the appendix.
Then,
[Equation]
where C 0 is a constant specified in appendix that depends on the initialization of z, v, x and memory.
The proof is similar to that of the previous theorem: we find coefficients ϕ s , ϕ z , ϕ v such that
L t = h t + ϕ s S t + ϕ z δ tz + ϕ v δ tv satisfies the inequality L t +1 ≤ (1 − c ′ γ )
L t , which is then unrolled.
Note that in the case where we initialize z and v with z
= z ∗ ( x 0 ) , v 0 = v ∗ ( x 0 ) , and the memories w
i = w 0 , ˜ w 0 j = w 0 for all i, j , the constant C 0 cancels and the bound simplifies
to
E [ h ( x T )] − h ∗ ≤ (1 − c ′ γ ) T ( h ( x 0 ) − h ∗ ) .
Just like classical variance reduction methods in single-level optimization, this theorem shows that our method achieves linear convergence under PL assumption on the value function.
To the best of our knowledge, our method is the first stochastic bilevel optimization method that enjoys such property.
We note that the PL hypothesis is more general than µ h -strong convexity of h – it is a necessary condition for strong convexity.
We see here the importance of global variance reduction.
Indeed, using variance reduction only on z and SGD on x would lead to sub-linear convergence in x .
This would be the case even with a perfect estimation of z ∗ ( x ) .
Similarly, using variance reduction only on x and SGD on z would lead to sub-linear convergence in z , and hence in x .
Using global variance reduction with respect to each variable as we propose here is the only way to achieve linear convergence.
We now turn to experiments, where we find that our method is also promising from a practical point of view.
Experiments
Here we compare the performances of SOBA and SABA with competitor methods on different tasks.
The different methods being compared are stocBiO [26], AmiGO [2], FSLA [30], MRBO [47], TTSA [24], BSA [19] and SUSTAIN [28].
A detailed account of the experiments is provided in Appendix B.
[Figure]
4.1 Hyperparameters selection
The first task we perform is hyperparameters selection to choose regularization parameters on ℓ 2 logistic regression.
Let us denote (( d train i , y train i ))
1 ≤ i ≤ n and (( d val i , y val i ))
1 ≤ i ≤ m the training and the validation sets.
In this case, the inner variable θ corresponds to the parameters of the model, and the outer variable λ to the regularization.
The functions F and G of the problem (1) are the logistic loss, with ℓ 2 penalty for G , that is to say F ( θ, λ )
= 1 m (cid:80) mi =1 φ ( y val i ⟨ d val i , θ ⟩ ) and
G ( θ, λ ) = 1 n (cid:80) ni =1 φ ( y train i ⟨ d train i , θ ⟩ ) + 12 (cid:80) pk =1 e λ k θ 2 k where φ ( u )
= log(1 + e − u ) .
We fit a binary classification model on the IJCNN1 1 dataset.
Here, n = 49 990 , m = 91 701 and p = 22 .
The suboptimality gap is plotted in Figure 2a for each method.
The lowest values are reached by
SABA.
Moreover, SABA is the only single-loop method that reaches a suboptimality below 10 − 3 .
SOBA reaches a quite high final value but slightly better than TTSA and FSLA.
The gap between SOBA and SABA highlights the benefits of variance reduction: it gives us a lower plateau and the fixed step sizes enable faster convergence.
4.2 Data hyper-cleaning
The second task we perform is data hyper-cleaning introduced in [16] on the MNIST 2 dataset.
The data is patitioned into a training set ( d train i , y train i ) , a validation set ( d val i , y val i ) , and a test set.
The training set contains 20000 samples, the validation set 5000 samples and the test set 10000 samples.
The targets y take values in { 0 , . .
, 9 } and the samples x are in dimension 784 .
Each sample in the training set is corrupted with probability p : a sample is corrupted when we replace its label
y
i by a random label in { 0 , . . .
Samples in the validation and test sets are not corrupted.
The goal of datacleaning is to train a multinomial logistic regression on the train set and learn a weight per training sample, that should go to 0 for corrupted samples.
This is formalized by the bilevel optimization problem (1) with F ( θ, λ ) =
1 m (cid:80) mi =1 ℓ ( θd val i , y val i ) and G ( θ, λ ) = 1 n (cid:80) ni =1 σ ( λ i ) ℓ ( θd train i , y train i )
+ C r ∥ θ ∥where ℓ is the cross entropy loss and σ is the sigmoid function.
The inner variable θ is a matrix of size 10 × 784 , and the outer variable λ is a vector in dimension n train = 20000 .
For the estimated parameters θ during optimization, we report in Figure 2b the test error, i.e. , the percent of wrong predictions on the testing data.
We use for this experiment a corruption probability p = 0 .
In general, the error decreases quickly until it reaches a final value.
We observe that our method SABA outperforms all the other methods by reaching faster its smallest error, which is smaller than the ones of the other methods.
For SOBA, it reaches a lower final error than stocBiO and BSA.
In appendix, we provide other convergence curves, and find that for higher values of p , SABA is still the fastest algorithm to reach its final accuracy.
Overall, we find that among all methods, even those that implement variance reduction (that is FSLA, MRBO, SUSTAIN, SABA), SABA is the one that demonstrates the best empirical performance.
Conclusion
In this paper, we have presented a framework for bilevel optimization that enables the straightforward development of stochastic algorithms.
The gist of our framework is that the directions in Equations (4) to (6) are all written as simple sums of samples derivatives.
We leveraged this fact to propose SOBA, an extension of SGD to our framework, and SABA, an extension of SAGA to our framework, which both achieve similar convergence rates as their single level counterparts.
Finally, we think that our framework opens a large panel of potential methods for stochastic bilevel optimization involving techniques of extrapolation, variance reduction, momentum and so on.
Acknowledgments and Disclosure of Funding
We thank Othmane Sebbouh, Zaccharie Ramzi and Benoît Malézieux for their precious comments.
The authors acknowledge the support of the ANER RAGA BFC.
SV acknowledges the support of the
ANR GraVa ANR-18-CE40-0005.
This research was supported by DATAIA convergence institute as part of the " Programme d’Investissement d’Avenir ", (ANR-17-CONV-0003) operated by Inria.
A Extensive comparison between stochastic methods for bilevel optimization
We provide here tables summarizing other methods in stochastic bilevel optimization.
They are grouped between methods that are based on two nested loops and methods that use only one loop.
In the following tables, the inner iterations are referred with the variable k and the outer iterations are referred with the variable t (or T for the total number of iterations).
In the literature, there are three main ways to perform Hessian inversion.
The HIA, first proposed in [19], and SHIA, proposed in [26], procedures used for Hessian inversion are precised in Algorithmand 3.
These methods are based on Neumann approximation of the inverse of a matrix.
SGD for
Hessian inversion refers to Stochastic Gradient Descent on v (cid:55)→ 12 ⟨∇ 211 G ( z, x ) v, v ⟩−⟨∇ 1 F ( z, x ) , v ⟩ .
The complexity refers to the number of call to the oracles to get an ϵ -stationary solution.
In these complexities, the notation ˜
O hide polynomial factors in log ϵ − 1 .
Algorithm 2 Hessian Inverse Approximation (HIA)
Input: variables z ∈ R p , x ∈ R d , gradient ∇ 1 F ( z, x ) ∈ R p , maximum number of iterations b , a parameter η .
Set v 0 = ∇ 1 F ( z, x )
Choose p ∈ { 0 , . .
, b − 1 } randomly.
for k = 1 , . . .
, p do
Sample i ∈ [ n ]
Update v : v k +
=
( I − η ∇ 211 G ( z, x )) v k end for
Return: bηv p +
Algorithm
Summed Hessian Inverse Approximation (SHIA)
Input: variables z ∈ R p , x ∈ R d , gradient ∇ 1 F ( z, x ) ∈ R p , maximum number of iterations b , a parameter η .
Set v 0 = ∇ 1 F ( z, x )
Set s 0 = vfor k = 0 . . .
, b − 1 do
Sample i ∈ [ n ]
Update v : v k +
=
( I − η ∇ 211 G ( z, x )) v k
Update s : s k +
=
s k
+ v k + end for
Return: ηs b
The momentum column refers to the use of STORM [12] momentum in the inner loop or the outer loop.
This momentum can be applied to either the inner or the implicit gradient estimate.
If we consider the current estimate y t = ( z t , v t , x t ) and the previous estimate y t − 1 = ( z t − 1 , v t − 1 , x t − 1 ) , and we apply STORM to the quantity ϕ ( y t ) with the memory ˆ ϕ t , the momentum update rule reads
[Equation]
Note that this update requires to evaluate the quantity ϕ twice per iteration, once in y t and once in y t − 1 .
The memory is need to store the previous estimates y t − 1 as well as the running estimate of the gradient ˆ ϕ .
M e t hod ( T w o - l oop s )
H e ss i a n i nv e r
s
i on
I nn e r l oop
M o m e n t u m
L R i n
L R o u t
C o m p l e x it y
B S A [ 19 ]
H I
A
S GD on i nn e r
N o
O ( k − 1 )
O ( T − 1 /)
O ( ϵ − 3 ) s t
o c
B i
O [ 26 ]
S H
I
A
S GD on i nn e r
N o
C on s t a n t
C on s t a n t ˜ O
( ϵ − 2 )
V
RB
O [ 47 ]
S H
I
A
SP
I D
E R on i nn e r
Y e s ( SP
I D E
R )
C on s t a n t C on s t a n t ˜ O
( ϵ
[Table]
A m I
GO [ 2 ]
S GD
S GD on i nn e r
N o
C on s t a n t C on s t a n t
O ( ϵ − 2 )
M e t hod ( O n e - l oop )
H e ss i a n i nv e r
s
i on
I nn e r s t e p
M o m e n t u m
L R i n
L R o u t
C
o m p l
[Table]
x
[Table]
y
TT S
A [ 24 ]
H I
A
S GD
N o
O ( T − 2 /)
O ( T
[Table] [Table]
O (
[Table]
−
[Table] [Table]
S M
B [ 23 ]
H I
A
S GD w it h
m o m e n t u m
Y e s
C on s t a n t C on s t a n t ˜ O
( ϵ − 4 )
M
RB
O [ 47 ]
S H
I
A
S GD w it h
S T O
R M
Y e s ( S T
O R
M )
O ( t − 1 / 3 ) O ( t − 1 / 3 ) ˜
O ( ϵ − 3 /)
S T A
B LE [ 10 ]
D i r ec t
S GD
N o
O ( T − 1 /) O ( T − 1 /)
O ( ϵ − 2 )
S U S
T A
I
N [ 28 ]
H I
A
S GD w it h
S T O
R M
Y e s ( S T
O R
M )
O ( t − 1 / 3 ) O ( t − 1 / 3 )
O ( ϵ − 3 / 2 )
S V
RB [ 22 ]
D i r ec t + m o m e nt u
m
S GD w it h
m o m e n t u m
Y e s
O ( t − 1 / 3 ) O ( t − 1 / 3 ) ˜
O ( ϵ − 3 )
S B F
W [ 1 ]
H I
A
S GD
N o
O ( t − 1 / 2 )
O ( T − 3 /)
˜ O
( ϵ − 4 )
FS L
A [ 30 ]
S GD w it h
S T O
R M
S GD w it h
S T O
R M
Y e s ( S T
O R
M )
O ( t − 1 / 2 )
O ( T − 1 /)
O ( ϵ − 2 )
S O
B A
S GD s t e p
S GD
N o
O ( t − 1 / 2 ) O ( t − 1 / 2 )
O ( ϵ − 1 / 2 )
S A
B A
S
AGA s t e p
S
AGA
N o
C on s t a n t C on s t a n t
O (( n + m ) 2 / 3 ϵ − 1 )
T a b l e 1 : C
o m p a r
i s on o f t
h e s t
o c h a s ti c b il e v e l o p ti m i za ti on s o l v e r s i n t h e lit
e
r a t u r e .
T
h e c o m p l e x i t y r e p r e s e n t s t
h e
nu m b e r
o
f
o r ac l e ca ll s n ece ss a r y t
o a tt a i n a n ϵ acc u
r a t e s t a ti on a r y po i n t .
B Details on experiments
We provide here additional informations on the experiments.
B.1 Generalities
All the experiments are performed with Python , using the package Benchopt [35] and Numba [29] for fast implementation of stochastic methods.
For each problem, we use oracles for a function given function f that ( f ( z, x ) , ∇ 1 f ( z, x ) , ∇ 211 f ( z, x ) v, ∇ 221 f ( z, x ) v ) avoiding duplicate computation of intermediate results for these quantities.
We find that using mini-batches instead of individual samples to compute the stochastic estimates allowed for much faster computations, thanks to hardware acceleration and vectorization of the computations.
We use continuous batches to avoid random memory access that slow down the computations.
Concretely, if i b is the index of the current batch and B is the batch-size, the indices of the corresponding samples are those in the set { i b × B, . .
, ( i b + 1) × B − 1 } .
By doing so, the samples in a same batch are contiguous in memory, which facilitates the access.
We use a batch-size of 64 in all experiments.
For the methods involving an inner loop (stocBiO, BSA, AmIGO), we perform 10 inner steps at each outer iteration as proposed in the papers which introduced these methods.
For the approximate
Hessian vector product, we perform 10 steps per outer iteration for each methods using HIA (BSA,
TTSA, SUSTAIN), SHIA (MRBO, stocBiO) or SGD (AmIGO) for the inversion of the linear system.
For the step sizes, they all have the form ρ t = α/t a and γ t = β/t b .
For the pair of exponents ( a, b ) , we choose the theoretical one from the original papers, that is (1 / 2 , 1 / 2) for BSA and FSLA, (1 / 3 , 1 / 3) for MRBO and SUSTAIN, (0 , 0) for SABA, AmIGO and stocBiO, (2 / 5 , 3 / 5) for TTSA and SOBA.
For ( α, β ) , we perform a grid search (the grid is precised in the subsection dedicated to each experiment) and we keep for each method, the pair ( α, β ) that gives the lowest value of h (for the hyperparameters) or the lowest test accuracy (for the data cleaning task) in median over 10 runs for each possible pair.
When we use HIA or SHIA for the Hessian inversion, we set η = α since the
Hessian inversion problem has the same conditioning as the inner optimization problem.
For the STORM’s momentum parameter in MRBO and SUSTAIN, we take 0 .
5 /t 2 / 3 .
For SABA, we have to maintain the estimate S [ ϕ, w ] it
= ϕ
i ( w t +1 i ) − ϕ
i ( w ti ) + 1 n (cid:80) ni ′ =
1 ϕ i ′ ( w ti ′ ) of 1 n (cid:80) ni
=
ϕ i ( y t ) (see Section 2.2 for the notations).
The sum inside S is maintain by performing a rolling mean on the past gradients computed.
More precisely, A t = 1 n (cid:80) ni ′
=1 ϕ i ′ ( w ti ′ ) .
To get
A t +1 , instead of computing the summing all the gradients stored, which has O ( n ) computational complexity, we do A t
+
=
A t + 1 n ( ϕ i ( w t +1 i ) − ϕ i ( w ti )) , which is equivalent mathematically but has
O (1) computational complexity.
B.2 Hyperparameter selection on a toy problem
The Figure 1 corresponds to the methods SABA et SOBA applied to an hyperparameter selection problem for a Ridge regression.
We generate 1000 samples x 1 , . . .
, x 1000 ∈ R 1 0 for N (0 , I 1 0) .
We generate a parameter β ∼ N (0 , I 10 ) and do y =
( X ⊙ W ) β
+ ϵ where ϵ ∼ N (0 , 0 .
01 I 10 ) and the entries of W have the form W i,j = 1 + u j v i,j with v
i,j ∼ U ([0 , 1]) and u j ∼ U ([0 , 1]) if 1 ≤ j ≤or u
j ∼ U
[0 , 10]) if 6 ≤ j ≤ 10 .
Then we use 750 pairs ( x train i , y train i ) 1 ≤ i ≤ 750 as training samples and the remaining pairs ( x val i , y val i ) 1 ≤ i ≤ 250 as validation samples.
Finally, we solve (1) with
[Equation]
and
[Equation]
with n train = 750 and n val = 250 .
B.3 Hyperparameters selection on IJCNN
In this experiment, we select the parameters regularization for a multiregularized logistic regression model precised in Equations (12) and (13) where we have one hyperparameter per feature
[Equation]
[Equation]
Note that the parametrization in e λ of the penalty instead of λ can be surprising at first glance, but it is classical in the bilevel optimization literature [38, 26, 21] because it avoids positivity constraints on λ .
In order to choose the select proper parameters ( α, β ) for each algorithm, we perform a grid search.
We search α in a set of 9 values between 2 − 5 and 2 3 spaced on a log scale.
For β , we choose r in a set of 7 values between 10 − 2 and 10 spaced on a logarithmic scale and we set β = αr .
For this experiments, we use Just-In-Time (JIT) compilation thanks to the package Numba [29], to decrease the python overhead in the iteration loop.
To evaluate the value function h , we use L-BFGS [32] to solve compute z ∗ ( x t ) and then evaluate the function h ( x t )
=
F ( z ∗ ( x t ) , x t ) .
[Figure]
B.4 Data hyper-cleaning
For the regularization parameter C r , we choose C r = 0 .
2 after a manual search in order to get the best final test accuracy.
In this experiment, the selection of the good pair ( α, β ) is also performed by grid search.
The parameter α is picked in a set of 11 numbers between 10 − 3 and 100 spaced on a logarithmic scale.
For β , we choose r in a set of 11 values between 10 − 5 and 1 spaced on a logarithmic scale and we set β = αr .
Note that in this case, we could not use JIT from Numba since at the moment of the experiment, the softmax function coming from Scipy was not compatible with Numba.
We report in Figure B.2 some additional convergence curves with different corruption probabilities p ∈ { 0 . 5 , 0 .
9 } (the figure in the main text corresponds to p = 0 . 5 ).
SABA is always the fastest algorithm to reach its final accuracy.
B.5 Additional experiment: Hyperparameter selection on the covtype dataset
We also perform an additional experiment which consists in selecting the best regularization parameter for a ℓ 2 -regularized multinomial logistic regression problem on the covtype dataset 4 .
This dataset
[Figure]
contains 581 , 012 samples with p = 54 features and there are C = 7 classes.
We used n = 371 ,train samples, m = 92 , 962 samples and n test = 116 , 203 test samples.
We fit a multiclass logistic regression on this dataset, with one hyperparameter per class.
This means that, if ( d train i , y train i ) i ∈
[ n ] and ( d val , y val ) i ∈ [ m ] are respectively the training samples and the validation samples, we solve the
Problem (1) with
F ( θ, λ ) = 1 m m (cid:88) i =
ℓ ( θd val i , y val i ) and
G ( θ, λ ) = 1 n n (cid:88) i = ℓ ( θd train i ,
y train i )
+
C (cid:88) c
= e λ c p (cid:88) i
= θ 2 i,c where θ ∈ R p × C and λ ∈ R C .
As for the other experiments, we performed and grid search over 63 pairs ( α, β ) to set the step sizes.
The parameter α is chosen among values between 2 − 5 and 2 3 spaced in log scale.
For β , we choose it in a set of values between 10 − 2 and 10 spaced in log scale.
We used a batch size of 64.
The experiment took 525 CPU hours.
We show in Figure B.3 the error on the test samples with respect to the running time and the number of gradients/Hessian-vector products sampled.
We observe that SABA and SOBA achieve the best performances.
The initial gap between the first and the second plot for SABA is due to the overhead of the initialization of the memory.
This gap can be reduced by increasing the batch size.
[Figure]
C Proofs C.1 Proof of Proposition 2.
Proof.
Let ( z, v, x ) a zero of ( D z , D v , D x ) .
For D z , this means that ∇ 1 G ( z, x ) = 0 .
Since
G ( · , x ) is strongly convex, z is the minimizer of G ( · , x ) , i.e. z = z ∗ ( x ) .
The fact that ( z, v, x ) is a zero of D v implies that ∇ 211 G ( z, x ) v = −∇ 1 F ( z, x ) .
Replacing z by its value, we get v = − (cid:2) ∇ 211 G ( z ∗ ( x ) , x ) (cid:3) − 1 ∇ 1 F ( z ∗ ( x ) , x ) which is v ∗ ( x ) by definition.
Putting all together and using the expression of ∇ h ( x ) given by (2), we get
[Equation]
On the other hand, D x ( z, v, x ) = 0 so ∇ h ( x ) = 0 .
C.
2 Proof of Lemma 3.
Proof.
Let ( z, v, x ) ∈ R p × R p × R d .
Using the fact that ∇ 1 G ( z ∗ ( x ) , x ) =
and the L G 1 -smoothness of G ( · , x ) , we have
[Equation]
For D v , since ∇ 211 G ( z ∗ ( x ) , x ) v ∗ ( x ) = −∇ 1 F ( z ∗ ( x ) , x ) , we write ∥
[Equation]
For the first term, we use the Lipschitz continuity of ∇ 211 G :
[Equation]
Then, since G in µ G -strongly convex w.r.t.
z , ∇ 1 F ( z ∗ ( · ) , · ) is bounded and v ∗ ( x ) = − [ ∇ 211 G ( z ∗ ( x ) , x )] − 1 ∇ 1 F ( z ∗ ( x ) , x ) , we have
[Equation]
For the second term, we use the L G 1 -smoothness of G ( · , x ) and for the third term, we use the
L F 1 -smoothness of F
and we finally get
[Equation] [Equation]
Then, taking L v = √ 2 max (cid:16) L G 2 C F
[Equation]
G + L F , L G 1 (cid:17) , we get
[Equation]
For D x ( z, v, x ) − ∇ h
( x ) we start by writing
[Equation]
We bound the first term using the fact that ∇ 2 F is L F 1 -Lipschitz continuous.
For the second term, the fact that ∇ 221 G is bounded thanks to the Lipschitz continuity of ∇ 1 G ( z, · ) .
For the third term, we use that ∇ 221 G ( · , x ) is L G 2 -Lipschitz continuous and the same derivation as Equation (16).
We finally get
[Equation]
Taking L x = √ 2 max (cid:16) L F 1 + C F L G 2 µ G , L G 1 (cid:17) yields
[Equation]
C.3 Smoothness constant of h From
Ghadimi and Wang [19, Lemma 2.2], we get the Lemma 3.10 which states the L h -smoothness of h with
[Equation]
C.4 Lemmas on the regularity of z ∗ and v ∗
We start by showing the Lipschitz continuity of z ∗ and v ∗ .
Lemma C.1.
There exists a constant L ∗ > 0 such that for any x 1 , x 2 ∈ R
d
we have
[Equation]
Proof.
Let x ∈
R d .
The
Jacobian of z ∗ is given by d z ∗ ( x ) = − [ ∇ 211 G ( z ∗ ( x ) , x )] − 1 ∇ 21 , 2 G ( z ∗ ( x ) , x ) .
Thanks to the µ G -strong convexity of G and the fact that ∇ 221 G is bounded, we have ∥ d z ∗ ( x ) ∥ ≤ L G 1 µ G .
Thus, z ∗ is Lipschitz continuous.
For ∥ v ∗ ( x 1 ) − v ∗ ( x 2 ) ∥ , we start from the definition v ∗ :
[Equation]
[Equation]
For the first term, we use that for any invertible matrix A and B we have A − 1 − b − 1 =
A − 1 ( B −
A ) B − 1 to get
[Equation]
And then, since ∇ 1 F ( z ∗ ( · ) , · ) is bounded:
[Equation]
For the second term
[Equation]
the strong convexity of G ( ·
[Equation]
x ) and the fact that ∇ 1 F is Lipschitz continuous lead to
[Equation]
Then we get
[Equation]
We conclude by setting
[Equation]
In what follows, we denote by E t [ · ] the expectation conditionally on z t , v t and x t .
We have the smoothness property of z ∗ provided in [9, Lemma 2].
Lemma C.2.
Under the Assumptions 3.1, 3.2 and 3.3, the function z ∗ : R d → R p is L zx -smooth with
[Equation]
We establish the same result for v ∗ .
To this, we need more regularity on G and F .
Lemma C.3.
The function v ∗ : R d → R p is differentiable and its differential is defined for any x, ϵ ∈ R d by:
[Equation]
where for any z, α ∈ R p and x ∈ R d
[Equation]
( ∇ 3111 G ( z, x ) | α ) ∈
R p × p is defined by
[Equation]
and for any β ∈ R d
[Equation]
( ∇ 3112 G ( z, x ) | β ) ∈
R p × p is defined by
[Equation]
Moreover, d v ∗ is L vx -Lipschitz continuous.
Proof.
Let x
[Equation]
ϵ ∈ R d .
Using the differentiability of ∇ 211 G , ∇ 1 F and of the matrix inversion
[Equation]
we have v
[Equation]
( x )
[Equation]
x
[Equation]
×
[Equation] [Equation]
−
[Equation]
] ×
[Equation]
×
[Equation]
z
[Equation]
] −
[Equation]
× ∇ 1 F ( z ∗ ( x ) , x ) + o
[Equation]
∥ ) that proves (31).
Now, let x, y, ϵ ∈ R d with ∥ ϵ ∥ = 1 .
Let us denote
A ( x, ϵ ) = − [ ∇ 21 G ( z ∗ ( x ) , x )] − 1 [( ∇ 3111 G ( z ∗ ( x ) , x ) | d z ∗ ( x ) .ϵ )
+( ∇ 3112 G ( z ∗ ( x ) , x ) | ϵ )]
[ ∇ 21 G ( z ∗ ( x )
[Equation]
x )] −and
[Equation]
so that d v ∗ ( x ) .ϵ
= [ ∇ 211 G ( z ∗ ( x ) , x )] − 1 B ( x, ϵ )
+ A ( x, ϵ ) ∇ 1 F ( z ∗ ( x ) , x ) .
We have (
[Equation] [Equation]
[Equation] [Equation]
We can now bound each term using the regularity assumptions on G and F :
[Equation]
For the second term:
[Equation]
For the third term
[Equation]
we have:
[Equation]
And finally, for the forth term:
[Equation]
Thus v ∗ is L vx -smooth with
[Equation]
C.5 Proof of Lemma 3.
We now provide the proof of Lemma 3.9.
Proof.
Inequality for δ z .
We start by expanding the square:
[Equation]
We study each member, using the unbiasedness of D tz and the µ G − strong convexity of G ( · , x t ) :
[Equation]
Taking the total expectation yields
[Equation]
The second member is bounded using Lipschitz continuity of z ∗ :
[Equation]
For the remaining scalar product
[Equation]
we have
[Equation] [Equation]
The second term can be bounded using Cauchy-Schwarz inequality, the Lipschitz-continuity of z ∗ and Young inequality:
[Equation]
[Equation]
For − 2 ⟨ z t − z ∗ ( x t ) , z ∗ ( x t +1 ) − z ∗ ( x t ) ⟩ , we follow the proof of [9] which consists in making appear the "unbiased part of z ∗ ( x t +1 − z ∗ ( x t ) by a linear approximation.
More precisely, we have
[Equation]
z
[Equation] [Equation]
[Equation]
For A , we use the unbiasedness of D tx , Cauchy-Schwarz inequality, the Lipschitz continuity of z ∗ (Lemma C.1) and the identity ab ≤ ηa 2 + b 2 η for any η > 0 :
[Equation] [Equation]
[Equation] [Equation]
We take η = ρµ G 4 and we get
[Equation]
[Equation]
For B , we use Cauchy-Schwarz inequality, the smoothness of z ∗ (Lemma C.2), Young inequality and the boundedness of E t [ ∥ D tx ∥ 2 ] to get
[Equation]
We take ν =
L zx L 2 ∗ and we get
[Equation]
[Equation]
Now, using γ 2 ≤ ρµ G L 2 ∗ B 2 x L 2 zx , we end up with δ t +1 z
[Equation]
[Equation]
with β zx = 3 L 2 ∗ and β zx = 8 L 2 ∗ µ G .
Inequality for δ v .
We proceed in a similar way for v : δ t +1 v ≤
[Equation] [Equation]
For the first term
[Equation]
we have
E
[Equation] [Equation]
Now, using that D v ( z ∗ ( x t )
[Equation]
v ∗ ( x t ) , x t ) = 0 : ⟨ D v (
[Equation]
t , v t , x t
[Equation]
, v t −
[Equation]
t
[Equation] [Equation]
[Equation] [Equation]
t
[Equation] [Equation]
[Equation]
(75) where ω =
L F 1 + L G 2 C F µ G .
We then use ω ∥
z t − z ∗ ( x t ) ∥∥ v t − v ∗ ( x t ) ∥ ≤ 12 c ∥ v t − v ∗ ( x t ) ∥ 2 + ω 2 2 c ∥
z t − z ∗ ( x t ) ∥ 2 with c = µ G to get −⟨ D v ( z t , v t , x t ) , v t − v ∗ ( x t ) ⟩ ≤ − 1 2 µ G δ t v + ω 2 2 µ G δ t z .
We get the overall inequality by taking the total expectation
[Equation]
We also use Lipschitz on v ∗ to bound the other term
[Equation]
As previously
[Equation]
the scalar product is bounded by:
[Equation]
We do similar manipulations pour v ∗ , thanks to Lemma C.3.
We have as for z from Lemma C.1 for any η > 0 :
[Equation]
We take η = ρµ G 8 and we get
[Equation]
Then smoothness of v ∗ for any η > 0 gives us
[Equation]
With ν =
L vx L 2 ∗ we get − E [
[Equation]
With the assumption γ 2 ⩽ ρµ G L 2 ∗ 8 L 2 vx B 2 x , we get δ t +1 v
[Equation] [Equation]
And finally we have δ
[Equation]
with β vz = ω 2 2 µ G , β vx = 3 L 2 ∗ and β vx = 16 L 2 ∗ γ 2 µ G .
C.6 Proof of Lemma 3.
Proof.
We use smoothness of h to get
E
[Equation]
where the last inequality comes from the identity ⟨ a, b ⟩ = 1 2 ( ∥ a ∥ 2 + ∥ b ∥ 2 − ∥ a − b ∥ ) 2 .
We take the total expectation and use the previous Lemma 3.4 to get
[Equation]
C.7 Proof of Theorem
This section is devoted to the proof of Theorem 1 that we recall here.
Theorem 1 (Convergence of SOBA, fixed step size) .
Fix an iteration T > 1 and assume that
Assumptions 3.1 to 3.7 hold.
We consider fixed steps ρ t = ρ √ T and γ t = ξρ t with ρ and ξ precised in the appendix.
Let ( x t ) t ≥ 1 the sequence of outer iterates for SOBA.
Then,
[Equation]
The values of the differents constants are
[Equation]
and ξ 2 = µ G 4 min (cid:20) min (cid:18)
[Equation]
zx
[Equation]
Before, one has to adapt our descent lemmas to the case of SOBA.
Lemma C.4.
Assume that the step sizes ρ and γ verify ρ ≤ min (cid:16) µ G 16 L 2 z B 2 z
[Equation]
µ G 32 L 2 v B 2 v , β vz L 2 v B 2 v (cid:17) and γ 2 ≤ min (cid:16) ρµ
G L 2 ∗ 4 B 2 x L 2 zx , ρµ G L 2 ∗ 8 B 2 x L 2 vx (cid:17) .
Then it holds δ
[Equation]
z
[Equation]
Proof.
From Assumption 3.6 and Lemma 3.4, we have
[Equation]
Plugging this into Equation (69) and using V tx ≤ B 2 x yields
[Equation]
Since by assumption ρ ≤
[Equation]
z B 2 z , we have
[Equation]
For δ t v , Assumption 3.3 and Lemma 3.4 provide us
[Equation]
Since the assumptions of Lemma 3.9 are verified, we can plug the previous inequality into Equation (85) to get δ t +1 v
[Equation]
which can be simplified using ρ ≤ min (cid:16) µ G 32 L 2 v B 2 v ,
β vz L 2 v B 2 v (cid:17) to get finally
[Equation]
We can now prove Theorem 1.
Proof.
Consider the Lyapunov function L t = h t + ϕ z
δ tz + ϕ v δ tv .
Using the Equations (88), (69) and (85)
[Equation]
we can bound L t +1 − L t :
[Equation]
Let ϕ ′
z =
ϕ z γρ and ϕ ′ v = ϕ v γρ
[Equation]
so that:
[Equation]
In order to get a decrease, ϕ ′ z , ϕ ′ v , ρ and γ must verify
[Equation]
Let us take ϕ ′
z = 1 8 β zx and ϕ ′ v = min (cid:16) 1 8 β vx , µ G ϕ ′
z 32 β vz (cid:17) .
We have
[Equation]
and
[Equation]
If we impose L 2 x 2 γ + ϕ ′
z µ G
[Equation]
≤ ϕ ′
z µ G 8 ρ 2 γ , this combined with the third condition in Equation (97) gives the condition L 2 x 2 γ 2 ≤ min ( ϕ ′ v , ϕ ′ z ) µ G 16 ρ 2 .
We also have the conditions coming from the assumptions of C.4, that is
[Equation] [Equation]
and γ 2 ≤ min (cid:16) 1 L 2 zx , 1 L 2 vx (cid:17) µ G L 2 ∗ 4 B 2 x ρ .
Let us take ρ = ρ √ T with γ = ξρ where ξ is defined as
[Equation] [Equation]
From now, we have
[Equation] [Equation]
Summing and telescoping yields
[Equation]
and so
[Equation]
C.8 Proof of Theorem
Proof.
In the decreasing step size case, we take ρ t = ρ √ t and γ t = ξρ t where ρ is defined in
Equation (98) and ξ is defined in Equation (99).
We recall the integral majorization:
[Equation]
With such definition of ρ t and γ t , Equation (100) is still valid for any t ⩾ 1 .
The only difference is that the step sizes decrease with t .
Hence, by summing and rearranging in Equation (100), we get
T (cid:88)
[Equation]
The left-hand-side in Equation (104) can be lower bounded by
T (cid:88)
[Equation] [Equation]
Also we have
T
[Equation]
Plugging Equations (105) and (106) into Equation (104) and rearranging give
[Equation]
[Equation]
that is to say
[Equation]
C.9 Proof of Theorem
In this section, we prove Theorem 3 that we recall here
Theorem 3 (Convergence of SABA, smooth case) .
Assume that Assumptions 3.1 to 3.3 and 3.7 to 3.8 hold.
We suppose ρ = ρ ′ N − 2 3 and γ = ξρ , where ρ ′ and ξ depend only on F and G and are specified in appendix.
Let x t the iterates of SABA.
Then,
[Equation]
The constants ρ ′ and ξ are given by
[Equation]
and
[Equation]
where
[Equation]
and
[Equation]
C.9.1 Control of distance from memory to iterates
We can view our method has having two “parallel” memories for each variable ( z ti , v ti , x ti ) for i ∈ 1[ n ] corresponding to calls in G and ( z ′ tj , v ′ tj , x ′ tj ) for j ∈ [ m ] corresponding to calls to F .
At each iteration, we sample i at random uniformly and do ( z t +1 i ,
v t +1 i , x t +1 i ) =
( z t , v t , x t ) and ( z t +1 i ′ , v t +1 i ′ ,
x t +1 i ′ )
=
( z ti ′ , v ti ′ , x ti ′ ) for i ′
̸
= i , and similarly for the other memory.
In what follows, we focus on controlling the error between the iterates and the memories.
We define to make things simpler
[Equation]
and similarly E ′ tx , E ′ tv and E ′ tx .
Lemma C.5.
We have the following inequalities:
[Equation]
Proof.
We provide the detailed proof for E tz .
The approach for the five others is similar.
Let i ∈ [ n ] .
Taking the expectation of ∥ z t +1 − z t +1 i ∥ 2 conditionaly to z t , v t , x t yields
[Equation]
Then, using the fact that E t [ D tz ( z t , v t , x t )]
= D z ( z t , v t , x t ) , we have
[Equation]
We then upper-bound crudely the scalar product by Cauchy-Schwarz and Young inequalities with parameter β :
[Equation]
As a consequence, by taking the total expectation and summing for all i ∈ [ n ] , we find
E t +1 z
[Equation]
Finally, we take β = 12 nρ to obtain
[Equation]
C.9.2 Bounds on the variances
The following lemma gives us upper-bounds for E [ ∥ D tz ( z t , v t , x t ) ∥ 2 ] , E [ ∥ D tv ( z t , v t , x t ) ∥ 2 ] , and E [ ∥ D tx ( z t , v t , x t ) ∥ 2 ] .
Lemma C.6.
For SABA, there are constants L ′ z , L ′ v , L ′ x > 0 such that
[Equation]
and
[Equation]
Proof.
For SABA, if we consider i sampled from [ n ] at iteration t , we have
[Equation]
Hence we get
[Equation]
The second term is the variance of ∇ 1 G i
( z t , x t ) − ∇ 1 G i
( z ti , x ti ) , which is therefore upper-bounded by
[Equation]
where the inequality comes from the Lipschitz continuity of each ∇ 1 G i with L ′ z = max i ∈ [ n ] L G i 1 .
Then, by plugging (114) into (113) and taking the total expectation
[Equation]
we get
[Equation]
Things are quite similar for the other variables, albeit a bit more difficult.
In v
[Equation]
it holds
[Equation]
Here, we see that we need to control the variance of ∇ 1 F j ( z t , x t ) − ∇ 1 F j ( z ′ t j , x ′ t j ) + ∇ 211 G i ( z t , x t ) v t − ∇ 211 G i ( z ti , x ti ) v ti .
Since i and j are independent, this is a sum of two independent random variables, hence its variance is the sum of the variances, which is upper-bounded by
[Equation]
For E t [ ∥∇ 1 F j ( z t , x t ) − ∇ 1 F j ( z ′ tj , x ′ tj ) ∥ 2 ] we use the lipschitz continuity of the ∇ 1 F j :
[Equation]
[Equation]
The control of E t [ ∥∇ 211 G i ( z t , x t ) v t −∇ 211 G i
( z ti , x ti ) v ti ∥ 2 ] is a bit harder without assuming the boundness of v beforehand.
But, we can bypass the difficulty by introducing ∇ 211 G i ( z ∗ ( x t ) , x t ) v ∗ ( x t ) :
E t [ ∥∇
[Equation]
i
[Equation]
z
[Equation]
ti
[Equation] [Equation]
Let
[Equation]
=
[Equation]
and
[Equation]
Taking the total expectation and putting all together yields
[Equation]
In x we have similarly
[Equation]
We now form S t =
E tz + E tx + E tv + E ′ tz + E ′ tv + E ′ tx , and letting Γ = min( 1 m , 1 n ) .
Note that by definition, each quantity E tz is smaller than S t .
We will therefore use the cruder bounds on E [ ∥ D tz ∥ 2 ] , E [ ∥ D tv ∥ 2 ] and E [ ∥ D tx ∥ 2 ] as follows thanks to
Lemma 3.4 and Lemma C.
[Equation] [Equation]
and
[Equation]
We have the following lemma
Lemma C.7.
If 4 ρ 2 ( L ′ z + L ′ v ) + 4 γ 2 L ′ x ≤ Γ2 and 4 L ′′ x γ 2 ≤ ρ
( L 2 v + 4 L ′′ v ) , it holds
[Equation]
for some L s , β sz , P > 0 .
Proof.
It holds following eq.
(112) (and omitting the dependencies in ( z t , v t , x t ) in the direction for simplicity)
[Equation]
Using the previous bounds (115), (123) and (124), we get
[Equation]
Next, using 4 ρ 2 ( L ′ z + L ′ v ) + 4 γ 2 L ′ x ≤ Γ2 and letting P =
(2( m + n ) + 4) we get
[Equation]
To finish, we use Lemma 3.4 to get
S
[Equation]
(cid:18)
[Equation]
Then, using that 4 L ′′ x γ 2 ≤ ρ 2 ( L 2 v + 4 L ′′ v ) , we get the bound, letting L sz =
L 2 z + L 2 v + 4 L ′′ v and
L sv =
L 2 v + 4 L ′′ v :
S
[Equation]
≤ (cid:18)
[Equation]
with β sz = 2 PL sz , β sv = 2 PL sv
C.9.3 Putting
it all together
Recall that we denote g t =
E [ ∥∇ h ( x t ) ∥ 2 ] and h t =
E [ h ( x t )] .
In the following lemma, we adapt
Lemma 3.9 and Lemma 3.10 to the SABA algorithm.
Lemma C.8.
If
[Equation]
and γ ≤ min
[Equation]
then it holds
[Equation] [Equation]
and
[Equation] [Equation]
Proof.
We start from Lemma 3.9 and plug the bounds of Equations (125) and (126).
[Equation]
Since ρ ≤ µ G 64 L 2 z and γ 2 ≤
[Equation] [Equation]
zx
[Equation]
x , we have
[Equation]
The condition γ 2 ≤
L ′
[Equation]
2 gives us
[Equation]
With ρ ≤ β zx 2 β zx , we get
[Equation]
We can plug Equations (132), (133) and (134) into Equation (131) and we end up with
[Equation]
t
[Equation]
1 z
[Equation]
The proof for δ tv is quite similar.
From Lemma 3.9, Equations (126) and (127).
δ t +1 v
[Equation] [Equation]
Using ρ ≤
[Equation]
and γ 2 ≤
[Equation]
, we get
[Equation] [Equation]
With γ 2 ≤
[Equation] [Equation]
vx and ρ ≤ β
[Equation] [Equation]
) , we have
[Equation]
The condition γ 2 ≤
L ′ v 2 L ′ x
β vx ρ 2 yields
[Equation]
With ρ ≤ β vx 2 β vx we get
[Equation]
As a consequence of Equations (135), (137), (138), (139) and (140), we have
[Equation]
For the inequality on h t , we start from Equations (88) and (127)
[Equation]
Assuming γ ≤ min (cid:16) 14 L h ,
[Equation]
(cid:17) leads
[Equation]
We are now ready to prove Theorem 3.
Proof.
We consider the Lyapunov function
[Equation]
143) for some constants ϕ s , ϕ z and ϕ v .
We have
[Equation]
To get a decrease, ϕ z , ϕ v and ϕ s , ρ and γ must be such that:
[Equation]
In order to take into account the scaling of the quantities with respect to N = n + m , we take ρ = ρ ′ N n ρ , γ = γ ′ N n γ , ϕ z = ϕ ′
z N n z , ϕ v = ϕ ′ v N n v and ϕ s = ϕ ′
s N n s .
Since Γ = O ( N − 1 ) ,
P = O ( N ) , β sz = O ( N ) and β sv
= O ( N ) , we also define Γ ′ =
Γ N , P ′ =
PN − 1 , β ′ sz = β sz N −and β ′
sv N − 1 .
Now, the previous Equations read (after slight simplifications):
[Equation]
In order to ensure that the exponents on N are lower in the left-hand-side than those on the righthand-side
[Equation]
we take n z = n v = 0 , n ρ = n γ = − 23 and n s = − 13 .
The Equations become
[Equation]
We can replace the penultimate equation by the stronger
[Equation]
so that we can simplify all the equations by dropping the dependencies in N :
[Equation]
Let us take ϕ ′
s = 1 , ϕ ′
z = ϕ ′′
z ρ ′ γ ′ and ϕ ′ v = ϕ ′′ v
[Equation]
γ ′ with ϕ ′′ z
=
[Equation]
and
[Equation]
The equations become
[Equation]
The condition γ ′ ≤ 18 P ′ ensures that the first equation is verified.
With γ ′ ≤ min (cid:16)(cid:113) ϕ ′′ z µ G 32 L 2 x ρ ′ , ϕ ′′
z µ G 32 β
[Equation]
sz (cid:17) , the second equations is verified.
With γ ′ ≤ min (cid:16)(cid:113) ϕ
[Equation]
x ρ
[Equation]
(cid:17) , the third is verified.
With γ ′ ≤ (cid:113) Γ ′ 6 L h L ′ x , the last can be simplified:
[Equation]
Let us write γ ′ =
ξρ ′ .
If we want that equation does no contradict the previous upper bound on γ ′ involving ρ ′ and the conditions of Lemma C.8, that is
[Equation]
L ′ z 2 L ′ x β zx , (cid:115)
L ′ v 2 L ′ x
β vx (cid:33) (cid:124) (cid:123)(cid:122) (cid:125)
Kρ ′ γ ′ ≤ min (cid:32)(cid:114) µ G 64 β zx L ′′ x , (cid:114) µ G 128 β vx L ′′ x , (cid:115) β
vz 4 L ′′ x
β vx (cid:33) (cid:124) (cid:123)(cid:122) (cid:125) K(cid:112) ρ ′ γ ′ ≤ (cid:115) ϕ
′′ v µ G 384 ϕ
z L ′′ x (cid:124) (cid:123)(cid:122) (cid:125) K1 √ ρ ′ γ ′ ≤ min (cid:32)4 L h ,
L 2 x 2 L h L ′′ x , (cid:115) Γ ′ 6 L h L ′ x , 8 P ′ , ϕ ′′
z µ G 32 β ′ sz , ϕ ′′ v µ G 48 β ′ sv (cid:33) (cid:124) (cid:123)(cid:122) (cid:125)
Kγ ′ ≥ 15( ϕ ′′
z L ′ z + ϕ ′′ v L ′ v ) Γ ′ (cid:124) (cid:123)(cid:122) (cid:125)
Kρξ must verify ξ ≤ Kξ ≤ K
[Equation]
( ρ ′ ) −ξ ≤ K 3 ( ρ ′ ) −ξ ≤ K 4 ( ρ ′ ) −ξ ≥ K 5 ( ρ ′ )which is possible if ρ ′ satisfies ρ ′ ≤ min (cid:32)(cid:114) K
K 5 , (cid:18) K
K(cid:19) −, (cid:18) K K(cid:19) −, (cid:18) K
K(cid:19) − 2 (cid:33) .
Let us take
[Equation]
β
[Equation]
β zx
[Equation]
µ G
[Equation]
L
[Equation]
v
[Equation]
L
[Equation]
β vz
[Equation]
L 2 v
[Equation]
L
[Equation]
v )
[Equation]
β vx 2 β vx
[Equation] [Equation]
and
[Equation]
Finally, we have
[Equation]
and therefore, summing and telescoping yields
[Equation]
Since with respect to N we have
[Equation]
we end up with
T
[Equation]
(cid:88)
[Equation]
C.10 Proof of Theorem
We are now going to prove Theorem 4 that we recall here:
Theorem 4 (Convergence of SABA, PL case) .
Assume that h satisfies the PL inequality and that
Assumptions 3.1 to 3.3 and 3.7 to 3.8 hold.
We suppose ρ = ρ ′ N − 23 and γ = ξρ ′ N − 1 , where ρ ′ and ξ depend only on F and G and are specified in appendix.
Let x t the iterates of SABA and c ′ ≜ min (cid:0) µ h , 1 16 P ′ (cid:1) with P ′ specified in the appendix.
Then,
[Equation]
where C 0 is a constant specified in appendix that depends on the initialization of z, v, x and memory.
Here, we have ρ
[Equation]
min
[Equation]
and
[Equation]
where P ′ =
PN − 1 , Γ ′ =
Γ N ,
[Equation]
and
[Equation]
Proof.
For simplicity
[Equation]
we assume that h ∗ = 0 and so for any x ∈ R d the PL inequality reads:
[Equation]
Then, eq.
(130) gives
[Equation]
We take L t the Lyapunov function given in Equation (143).
We find
[Equation]
We now try to find linear convergence, hence we add to this c L t to get
[Equation]
Hence
[Equation]
the set of inequations for decrease becomes
[Equation]
We see that it is more convenient to write c = γc ′ .
As previously
[Equation]
we write γ = γ ′ N n γ , ρ = ρ ′ N n ρ , ϕ
[Equation]
The equations read:
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
In order to ensure that the exponents on N are lower in the left-hand-side than those on the righthand-side
[Equation]
we take n z = n v = 0 , n ρ = − 23 , n γ = − 1 and n s = − 13 .
The Equations become
[Equation]
Now we have to find ρ ′ , γ ′ , ϕ ′ z , ϕ ′ v and ϕ ′
s that verifies the following conditions (which are a bit stronger than thoose in the previous Equations):
[Equation]
As previously, we take ϕ ′ s =
and we denote ϕ ′
z = ϕ ′′
z ρ ′ γ ′ with ϕ ′′ z = 1 32 β zx and ϕ ′
z = ϕ ′′
z ρ ′ γ ′ with ϕ ′′ v = min (cid:16) 1 32 β vx , ϕ
[Equation]
z µ G 128 β vz (cid:17) , the equations become
[Equation]
Since c ′ ≤ 116 and γ ′ ≤ 1 16 P ′ , the second equation is verified.
With γ ′ ≤ min (cid:16)(cid:113) ϕ ′′ z µ G 48 L 2 x ρ ′ , ϕ
[Equation]
z
[Equation]
G 48 β sv (cid:17) and c ′ ≤ µ G ρ ′ 48 γ ′
the third is verified.
The conditions γ ′ ≤ min (cid:16)(cid:113) ϕ ′′ v µ G 64 L 2 x ρ ′ , (cid:113)
[Equation]
, ϕ
[Equation] [Equation]
sv (cid:17) and c ′ ≤ µ G ρ ′ 64 γ ′ ensure that the forth is verified.
With γ ′ ≤ (cid:113) Γ ′
L h L ′ x and c ′ ≤ Γ ′ 8 γ ′ , the fifth is simplified in
[Equation]
As in the proof of Theorem 3, let us denote γ ′ = ξρ ′ .
To verify this equation and the previous bounds on γ ′ and c ′ , we need
[Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
So
[Equation]
ξ
[Equation]
ρ ′ and c ′ must verify
[Equation]
which is possible if
[Equation]
So let us take c ′ =
min (cid:0) µ h , 116 , 1 16 P ′ (cid:1) =
min (cid:0) µ h , 1 16 P ′ (cid:1) ,
ρ ′ =
min   (cid:115) K ′
K 5 , (cid:18) K
K(cid:19), (cid:18) K K(cid:19), (cid:18) K ′
K(cid:19), µ G 64 L 2 z , β zx 2 β zx , µ G 128( L 2 v + L ′′ v ) , β vz 8( L 2 v + L ′′ v ) , β vx 2 β vx 
 and
[Equation]
We have
[Equation]
therefore, unrolling yields
[Equation]
D Convergence rates with weaker regularity assumptions
To get our rates, we need stronger assumptions than in the stochastic bilevel optimization literature [19, 24, 26, 2].
In this section, we shortly present the convergence rates we can expect if we replace
Assumptions 3.1 and 3.2 by Assumptions D.1 and D.2.
Assumption D.1.
The function F is differentiable.
The gradient ∇ F is Lipschitz continuous in ( z, x ) with Lipschitz constants L F 1 .
Assumption D.2.
The function G is twice continuously differentiable on R p × R d .
For any x ∈ R d ,
G ( · , x ) is µ G -strongly convex.
The derivatives ∇ G are ∇ 2 G are Lipschitz continuous in ( z, x ) with respective Lipschitz constants L G 1 and L G 2 .
With these assumptions, we are not ensured that v ∗ is smooth, and so the descent lemmas take the form of Lemma D.3.
Lemma D.3.
Assume that ρ ≤ 2 µ G .
We have:
[Equation]
where L ∗ is the maximum between the Lipschitz constants of z ∗ and v ∗ (see Lemma C.1) and β vz = 1 µ 3 G ( L F µ G + L G 2 ) 2 .
Proof.
Inequality for δ z .
Instead of expanding the square as done in the proof of Lemma 3.9 in Equation (45), we use Young’s inequality for some a >
[Equation]
Treating E [ ∥ z t +1 − z ∗ ( x t ) ∥ 2 ] and E [ ∥ z ∗ ( x t +1 ) − z ∗ ( x t ) ∥ 2 ] as done in the proof of Lemma 3.9 leads to
[Equation]
In order to keep a decrease in δ z , we might want to use a = 12 ρµ G , which gives the bound
[Equation] [Equation]
with β zx = 4 L 2 ∗ µ G .
Indeed, this gives (1 + 12 ρµ G )(1 − ρµ G ) ≤ 1 − 12 ρµ G .
We have a ≤ 1 since ρ ≤ 2 µ G , so (1 + a ) ρ 2 ≤ 2 ρ 2 .
Finally, we also have 1 + a − 1 ≤ 2 a − 1 = 4 ρµ G .
Inequality for δ v .
As for δ z , the difference with the proof of Lemma 3.9 is that we use we use
Young’s inequality for some b > 0 to get
[Equation]
t
[Equation]
v
[Equation]
The remaining part of the proof is similar to the proof of Lemma 3.9.
The main difference with Lemma 3.9 is that we have O ( γ 2 ρ ) in factor of V tx instead of O ( γ 2 ) .
As a consequence, we need that the ratio γρ goes to zero to get convergence, as in [24].
This prevent us in getting rates that match rates of single level algorithms.
Hence, for SOBA, we have to choose γ
=
O ( T − 35 ) and ρ =
O ( T − 25 ) and we end up with a convergence rate in O ( T − 25 ) .
For SABA, we get a O (( n + m ) ϵ − 1 ) sample complexity, which is actually the sample complexity of SOBA used with full batch estimated directions.