Will Bilevel Optimizers Benefit from Loops
Anonymous Author(s)
AffiliationAddress email
Abstract
Bilevel optimization has arisen as a powerful tool for solving a variety of machinelearning problems.
Two current popular bilevel optimizers AID-BiO and ITD-BiOnaturally involve solving one or two sub-problems, and consequently, whetherwe solve these problems with loops (that take many iterations) or without loops(that take only a few iterations) can significantly affect the overall computationalefficiency.
Existing studies in the literature cover only some of those implementation choices, and the complexity bounds available are not refined enough toenable rigorous comparison among different implementations.
In this paper, wefirst establish unified convergence analysis for both AID-BiO and ITD-BiO that areapplicable to all implementation choices of loops.
We then specialize our results tocharacterize the computational complexity for all implementations, which enablean explicit comparison among them.
Our result indicates that for AID-BiO, theloop for estimating the optimal point of the inner function is beneficial for overallefficiency, although it causes higher complexity for each update step, and the loopfor approximating the outer-level Hessian-inverse-vector product reduces the gradient complexity.
For ITD-BiO, the two loops always coexist, and our convergenceupper and lower bounds show that such loops are necessary to guarantee a vanishing convergence error, whereas the no-loop scheme suffers from an unavoidablenon-vanishing convergence error.
Our numerical experiments further corroborateour theoretical results.
Introduction
Bilevel optimization has attracted significant attention recently due to its popularity in a variety ofmachine learning applications including meta-learning [8, 1, 31, 15], hyperparameter optimization [8,32, 4], reinforcement learning [19, 14] and signal processing [20, 6].
In this paper, we consider thebilevel optimization problem that takes the following formulation.
[Equation]
where the outer- and inner-level functions f
and g are both jointly continuously differentiable.
Wefocus on the setting where the lower-level function g is strongly convex with respect to (w.r.t.)
ywith the condition number κ = Lµ (where L and µ are gradient Lipschitzness and strong convexitycoefficients defined respectively in Assumptions 1 and 3 in Section 3), and the outer-level objectivefunction Φ( x ) is
possibly nonconvex w.r.t.
x .
Such types of geometries arise in many applicationsincluding meta-learning (which uses the last layer of neural networks as adaptation parameters),hyperparameter optimization (e.g., data hyper-cleaning and regularized logistic regression) andlearning in communication networks (e.g., network utility maximization).
A variety of algorithms have been proposed to solve the bilevel optimization problem in eq.
Forexample, [13, 33, 29] proposed constraint-based approaches by replacing the inner-level problem [Table] [Table]
strong-convexity constant of inner-level function g ( x, · ) .
For the last two columns, ’N/A’ means that the complexities to achieve an ϵ -accuracy are not measurable due to the nonvanishing convergence error.
[Table]
with its optimality conditions as constraints.
In comparison, gradient-based bilevel algorithms havereceived intensive attention recently due to the effectiveness and simplicity, which include two popularapproaches via approximate implicit differentiation (AID) [3, 30, 10, 17] and iterative differentiation(ITD)
Readers can refer to Appendix A for an expanded list of related work.
Consider the AID-based bilevel approach (which we call AID-BiO).
Its base iteration loop updatesthe variable x until convergence.
Within such a base loop, it needs to solve two sub-problems:finding a nearly optimal solution of the inner-level function via N iterations, and approximatingthe outer-level Hessian-inverse-vector product via Q iterations.
If Q and N are chosen to be large,then the corresponding iterations form additional loops of iterations within the base loop, whichwe respectively call as Q -loop and N -loop.
Thus, AID-BiO can have four popular implementationsdepending on different choices of N and Q : N -loop (with large N = κ ln κ and small Q = O (1) ),
N - Q -loop (with large N = Θ( κ ln κ ) and large Q = Θ
( κ ln κ ) ), Q -loop (with N = 1 and Q =Θ( κ ln κ ) ), and No-loop (with N = 1 and Q = O (1) ).
Note that No-loop refers to no additionalloops within the base loop, and can be understood as conventional single-(base)-loop algorithms.
These implementations can significantly affect the efficiency of AID-BiO. Generally, large Q (i.e., a
Q -loop) provides a good approximation of the Hessian-inverse-vector product for the hypergradientcomputation, and large N (i.e., a N -loop) finds an accurate optimal point of the inner function.
Hence, an algorithm with N -loop and Q -loop require fewer base-loop steps to converge, but eachsuch base-loop step requires more computations due to these loops.
On the other hand, small Qand/or N avoid computations of loops in each base-loop step, but can cause the algorithm to convergewith many more base-loop steps.
An intriguing question here is which implementation is overallmost efficient and whether AID-BiO benefits from having N -loop and/or Q-loop.
Existing theoreticalstudies on AID-BiO are far from answering this question.
The studies [9, 17] on deterministic AID-BiO focused only on the N - Q -loop scheme.
A few studies analyzed the stochastic AID-BiO,such as [23] on No-loop, and [14, 18] on Q -loop.
Those studies were not refined enough to capturethe computational differences among different implementations, and further those studies collectivelydid not cover all the four implementations either.
• The first contribution of this paper lies in the development of a unified convergence theory forAID-BiO, which is applicable to all choices of N and Q .
We further specialize our generaltheorems to provide the computational complexity for all of the above four implementations(as summarized in Table 1).
Comparison among them suggests that AID-BiO does benefitfrom both N -loop and Q -loop.
This is in contrast to minimax optimization (a special caseof bilevel optimization), where it is shown in [24, 37] that (No-loop) gradient descent ascent(GDA) with N = 1 often outperforms ( N -loop) GDA with N = κ ln κ (here N denotes thenumber of ascent iterations for each descent iteration).
To explain the reason, the gradientw.r.t.
x in bilevel optimization involves additional second-order derivatives (that do not existin minimax optimization), which are more sensitive to the accuracy of the optimal point of theinner function.
Therefore, a large N finds such a more accurate solution, and is hence morebeneficial for bilevel optimization than minimax optimization.
Differently from AID BiO
, the
ITD-based bilevel approach (which we call as ITD-BiO) constructs  the outer-level hypergradient estimation via
backpropagation along the N loop iteration path
, and
Q = N always holds.
Thus,
ITD-BiO has only two implementation choices: N - N -loop (with largeN = κ
ln κ and No-loop (with small N = O (
Here, N - N -loop and No-loop also refer toadditional loops for solving sub-problems within the ITD- BiO’s base loop of updating the variablex . The only convergence
rate analysis on ITD BiO was provided in [ but only for N - N -loop,which does not suggest how N - N -loop compares with No-loop.
It is still an open question whetherITD-BiO benefits from N -loops.•
The second contribution of this paper lies in the development of a unified convergence theoryfor ITD-BiO, which is applicable to all values of N .
We then specialize our general theorem to provide the computational complexity for both of the above implementations (as summarizedin Table 2).
We further develop a convergence lower bound, which suggests that N - N -loop isnecessary to guarantee a vanishing convergence error, whereas the no-loop scheme suffersfrom an unavoidable non-vanishing convergence error.
The technical contribution of this paper is two-fold.
For AID methods, most existing studies including[17] solve the linear system with large Q = Θ( κ log κ ) so that the upper-level Hessian-inverse-vectorproduct approximation error can vanish.
In contrast, we allow arbitrary (possibly small) Q , andhence this upper-level error can be large and nondecreasing, posing a key challenge to guaranteeconvergence.
We come up with a novel idea to prove the convergence by showing that this error,not by itself but jointly with the inner-loop error, admits an (approximately) iteratively decreasingproperty, which bounds the hypergradient error and yields convergence.
The analysis contains newdevelopments to handle the coupling between this error and the inner-loop error, which is criticalin our proof.
For ITD methods, unlike existing studies including [17], we remove the boundednessassumption on y ∗ ( x ) via a novel error analysis over the entire execution rather than a single iteration.
Our analysis tools are general and can be extended to stochastic and acceleration bilevel optimizers.
Algorithms
2.1 AID-based Bilevel Optimization Algorithm
As shown in Algorithm 1, we present the general AID-based bilevel optimizer (which we refer to
AID-BiO for short).
At each iteration k of the base loop, AID-BiO first executes N steps of gradientdecent (GD) over the inner function g ( x, y ) to find an approximation point y Nk , where N can bechosen either at a constant level or as large as N = κ ln κ (which forms an N -loop of iterations).
Moreover, to accelerate the practical training and achieve a stronger performance guarantee, AID-BiOoften adopts a warm-start strategy by setting the initialization y
k of each N -loop to be the outputy
Nk − 1 of the preceding N -loop rather than a random start.
To update the outer variable, AID-BiO adopts the gradient descent, by approximating the true gradient∇
Φ( x k ) of the outer function w.r.t.
x (called hypergradient) that takes the following form:
[Equation]
where v ∗ k is the solution of the linear system ∇ 2 y g ( x k , y ∗ ( x k )) v = ∇ y f
( x k , y ∗ ( x k )) .
To approximatethe above true hypergradient, AID-BiO first solves v
Qk as an approximate solution to a linear system∇ 2 y g ( x k , y Nk ) v = ∇ y f
( x k , y Nk ) using Q steps of GD iterations starting from v 0 k .
Here, Q can also bechosen either at a constant level or as large as Q = κ ln κµ (which forms a Q -loop of iterations).
1: Input: Stepsizes α, β > 0 , initializations x 0 , y 0 , v 0 .
2: for k = 0 , 1 , 2 , ..., K do 3:
Set y
k
= y Nk − 1 if k > 0 and y 0 otherwise (warm start initialization) 4: for t
= 1 , ...., N do 5:
Update y tk = y t − 1 k − α ∇ y g
( x k , y t − 1 k ) 6: end for 7:
Hypergradient estimation via:
Set v 0 k = v
Qk −
if k > 0 and v 0 otherwise (warm start initalization) .
Solve v Qk from ∇ 2 y g ( x k , y Nk ) v = ∇ y f
( x k , y Nk ) via Q steps of iterative algorithms starting from v 0 k
Compute (cid:98) ∇ Φ( x k ) =
∇ x f ( x k , y Nk ) − ∇ x ∇ y g
( x k , y Nk ) v Qk 8:
Update x k +
=
x
k − β (cid:98) ∇ Φ( x k ) 9: end for
Note that a warm start is also adopted here by setting v 0 k = v
Qk − 1 , which is critical to achievethe convergence guarantee for small Q .
If Q is large enough, e.g., at an order of κ ln κµ , a zeroinitialization with v 0 k =
suffices to solve the linear system well.
Then, AID-BiO constructs ahypergradient estimator (cid:98) ∇ Φ( x k ) given by(AID-based hypergradient estimate:)
[Equation]
[Equation]
Note that the execution of AID-BiO involves only Hessian-vector products in solving the linearsystem and Jacobian-vector product ∇ x ∇ y g ( x k , y Nk ) v Qk which are more computationally tractablethan the calculation of second-order derivatives.
It is clear that different choices of N and Q lead to four implementations within the base loop of
AID-BiO: N -loop (with large N = κ ln κ and small Q =
O
(1) ), N - Q -loop (with large N = κ ln κand Q
= κ ln κ ), Q -loop (with small N = 1 and large Q = κ ln κ ) and No-loop (with small N = 1and Q = O (1) ).
In Section 4, we will establish a unified convergence theory for AID-BiO applicableto all its implementations in order to formally compare their computational efficiency.
2.2 ITD-Based Bilevel Optimization Algorithm
As shown in Algorithm 2, the ITD-based bilevel optimizer (which we refer to as ITD-BiO)updates the inner variable y similarly to AID-BiO, and obtains the N -step output y Nk of GDwith a warm-start initialization.
ITD-BiO differentiates from AID-BiO mainly in its estimation of the hypergradient.
Without leveraging the implicit gradient formulation, ITD-BiO computes a direct derivative ∂f ( x k ,y Nk ) ∂x k via automatic differentiation for hypergradient approximation.
Since y Nk has a dependence on x k through the N -loop iterative GD updates, the execution of
ITD-BiO takes the backpropagation over the entire N -loop trajectory.
To elaborate, it can beshown via the chain rule that the hypergradient estimate ∂f ( x k ,y Nk ) ∂x k takes the following formof ∂f ( x k ,y Nk ) ∂x k = ∇ x f ( x k , y Nk ) − α (cid:80) N − 1 t =0 ∇ x ∇ y g ( x k , y tk ) (cid:81)
N − 1 j = t
{++
+}
( I − α ∇ 2 y g ( x k , y jk )) ∇ y f
( x k , y Nk ) .
Asshown in this equation, the differentiation does not compute the second-order derivatives directlybut compute more tractable and economical Hessian-vector products ∇ 2 y g ( x k , y j − 1 k ) v j , j = 1 , ..., N(similarly for Jacobian-vector products), where each v j is obtained recursively via v j − 1 =( I − α ∇ 2 y g ( x m , y jm )) v j with v N = ∇ y f
( x m , y Nm ) .
Clearly, the implementation of ITD-BiO implies that N = Q always holds.
Hence, ITD-BiO takesonly two possible architectures within its base loop: N - N -loop (with large N = κ ln κϵ ) and No-loop(with small N = 1 ).
In Section 5, we will establish a unified convergence theory for ITD-BiOapplicable to both of its implementations in order to formally compare their computational efficiency.
Definitions and Assumptions
This paper focuses on the following types of objective functions. Assumption 1. The inner-level function g ( x, y ) is µ -strongly-convex w.r.t.
y [-
-].
1: Input: Stepsize 0 , initializations x y 0 .
2: do 3: Set do 5:
− α
6: end for 7:
Compute (cid:98)
= y k via backpropagation w k 8:
Update x k +
=
x
k − β
Φ x k ) 9: end for
Since the objective function Φ( x ) in eq.
(1) is possibly nonconvex, algorithms are expected to find anϵ -accurate stationary point defined as follows.
Definition 1.
We say ¯ x is an ϵ -accurate stationary point for the bilevel optimization problem given ineq.
(1) if ∥∇ Φ(¯ x ) ∥ 2 ≤ ϵ , where ¯ x is the output of an algorithm.
In order to compare the performance of different bilevel algorithms, we adopt the following metricsof computational complexity.
Definition 2.
Let Gc ( ϵ ) be the number of gradient evaluations, and MV ( ϵ ) be the total number of
Jacobian- and Hession-vector product evaluations to achieve an ϵ -accurate stationary point of thebilevel optimization problem in eq.
Let z = ( x, y ) .
We take the following standard assumptions, as also widely adopted by [9, 15].
Assumption 2.
Gradients ∇ f ( z ) and ∇ g ( z ) are L -Lipschitz, i.e., for any z, z ′ ,
[Equation]
As shown in eq.
(2), the gradient of the objective function Φ( x ) involves the second-order derivatives∇ x ∇ y g
( z ) and ∇ 2 y g ( z ) .
The following assumption imposes the Lipschitz conditions on such higherorder derivatives, as also made in [9].
Assumption 3.
Suppose the derivatives ∇ x ∇ y g ( z ) and ∇ 2 y g ( z ) are ρ -Lipschitz, i.e., for any z, z ′
[Equation]
To guarantee the boundedness the hypergradient estimation error, existing works [9, 15, 10] assumethat the gradient ∇ f ( z ) is bounded for all z = ( x, y ) .
Instead, we make a weaker boundednessassumption on the gradients ∇ y f ( x, y ∗ ( x )) .
Assumption 4.
There exists a constant M such that for any x , ∥∇ y f ( x, y ∗ ( x ))
∥ ≤ M .
For the case where the total objective function Φ( · ) has some benign structures, e.g., convexity orstrong convexity, Assumption 4 can be removed by an induction analysis that all iterates are boundedas in [16].
Assumption 4 can also be removed by projecting x onto a bounded constraint set X .
Convergence Analysis of AID-BiO
As we describe in Section 2.1, AID-BiO can have four possible implementations depending onwhether N and Q are chosen to be large enough to form an N -loop and/or Q -loop.
In this section, wewill provide the convergence analysis and characterize the overall computational complexity for all ofthe four implementations, which will provide the general guidance on which algorithmic architectureis computationally most efficient.
4.1 Convergence Rate and Computational Complexity
In this subsection, we develop two unified theorems for AID-BiO, both of which are applicable toall the regimes of N and Q .
We then specialize these theorems to provide the complexity bounds(as corollaries) for the four implementations of AID-BiO. It turns out that the first theorem providestighter complexity bounds for the implementations with small Q = Θ(1) , and the second theorem provides tighter complexity bounds for the implementations with large Q = κ ln κϵ .
Our presentationof those corollaries below will thus focus only on the tighter bounds.
The following theorem providesour first unified convergence analysis for AID-BiO.
Theorem 1.
Suppose Assumptions 1, 2, 3 and 4 hold.
Choose parameters α, η and λ such that(1 + λ
[Equation]
1 −
[Equation]
N
[Equation]
r
[Equation]
[Equation]
ηµ
[Equation] L [Equation] ≤ 1 − ηµ , where r =
C [Equation]
[Equation] + L ) 2 with C Q = Q ( [Equation] [Equation]
Let
[Equation]
be the smoothness parameter of Φ( · ) . Let (cid:101) w := (1 − ηµ ) ηµ 3 λrL 2 (cid:0) 1 +
[Equation]
M [Equation] 2 (cid:1) L
[Equation] 2 + (cid:0) 1 + 1 ηµ (cid:1)(cid:0) L 2 + ρ [Equation] M
[Equation]
(cid:1)(cid:0) 16(1 − ηµ
[Equation]
4(1 − ηµ ) ηµ 3 λL(cid:1)
L 2 µ 2 .
Choose the outer stepsize β such that β = min (cid:8) 1 12 L Φ , (cid:112) ηµ 18 L 2 (cid:101) w (cid:9) .
Then,
K
[Equation]
Theorem 1 also elaborates the precise requirements on the stepsizes α , η and β and the auxiliaryparameter λ , which take complicated forms.
In the following, by further specifying these parameters,we characterize the complexities for AID-BiO in more explicit forms.
We focus on the implementations with Q = Θ(1) (for which Theorem 1 specializes to tighter bound than Theorem 2 below),which includes the N -loop scheme (with N = Θ( κ ln κ ) ) and the No-loop scheme (with N = 1 ).
Corollary 1 ( N -loop) .
Consider N -loop AID-BiO with N = Θ( κ ln κ ) and Q = Θ(1) , whereκ =
Lµ denotes the condition number of the inner problem.
Under the same setting of Theorem 1,choose η = 1 L , α = 1 L , and λ = 1 .
Then, we have 1 K (cid:80) K − 1 k =0 ∥∇ Φ( x k ) ∥
= O (cid:0) κ 4 K
+ κ 3 K (cid:1) , and thecomplexity to achieve an ϵ -accurate stationary point is Gc ( ϵ ) = (cid:101)
O ( κ 5 ϵ − 1 ) , MV ( ϵ )
= (cid:101) O (cid:0) κ 4 ϵ − 1 (cid:1) .
Corollary 2 (No-loop) .
Consider No-loop AID-BiO with N = 1 and Q = Θ(1) .
Under the samesetting of Theorem 1, choose parameters α = 1 L , λ
= αµ 2 and η = min { 1128 αµ 2 Q 2 L 2 , α 4 , 1 µQ } .
Then,1 K (cid:80) K − 1 k =0 ∥∇ Φ( x k ) ∥ 2 = O (cid:0) κ 6 K
+ κ 5 K (cid:1) , and the complexity is Gc ( ϵ ) = (cid:101)
O
( κ 6 ϵ − 1 ) , MV ( ϵ ) =
(cid:101) O ( κ 6 ϵ − 1 ) .
The analysis of Theorem 1 can be further improved for the large Q regime, which guarantees asufficiently small outer-level approximation error, and helps to relax the requirement on the stepsize η .
Such an adaptation yields the following alternative unified convergence characterization for AID-BiO,which is applicable for all Q and N , but specializes to tighter complexity bounds than Theorem 1 inthe large Q regime.
For simplicity, we set the initialization v 0 k = 0 in Algorithm 1.
Theorem 2.
Suppose Assumptions 1, 2, 3 and 4 hold.
Define τ = (1 − αµ ) N
(1 + λ + {+6(
+}
+ λ − 1 )( L 2 +ρ 2 M 2 µ − 2 + 2 L 2 C 2 Q (cid:1) L 2 β 2 µ − 2 ) , w = 6(1 − αµ ) N ( L 2 + ρ 2 M 2 µ − 2 + 2 L 2 C 2 Q )(1 + λ − 1 ) L 2 µ − 2 , where
C Q is a positive constant defined as in Theorem 1. Choose parameters α, β such that τ < 1 and
[Equation]
+ [Equation] (cid:0) 12 + βL [Equation]
(cid:1) 11 − τ ≤ 14 hold. Then, the output of AID-BiO satisfies
[Equation]
where δ [-
-]0 = 3 (cid:0) L 2 + ρ 2 M 2 µ 2 + 2 L 2 C 2 Q (cid:1) (1 − αµ ) N ∥
y ∗ 0 − y
∥ 2 is the initial distance.
We next specialize Theorem 2 to obtain the complexity for two implementations of AID-BiO with
Q = Θ( κ ln κ ) : N - Q -loop (with N =
Θ( κ ln κ ) ) and Q -loop (with N = 1 ), as shown in the followingtwo corollaries.
For each case, we need to set the parameters λ, η and α in Theorem 2 properly.
Corollary 3 ( N - Q -loop) .
Consider N - Q -loop AID-BiO with N = Θ( κ ln κ ) and Q = Θ( κ ln κϵ ) .
Under the same setting of Theorem 2, choose η = α = 1 L , λ = 1 and β
= Θ( κ − 3 ) .
Then,1 K (cid:80) K − 1 k =0 ∥∇ Φ( x k ) ∥ 2 = O (cid:0) κ 3 K + ϵ (cid:1) , and the complexity is Gc ( ϵ )
= (cid:101)
O
( κ 4 ϵ − 1 ) , MV ( ϵ ) =
(cid:101)
O ( κ 4 ϵ − 1 ) .
Corollary 4 ( Q -loop) .
Consider Q -loop AID-BiO with N = 1 and Q = Θ
( κ ln κϵ ) .
Under the samesetting of Theorem 2, choose α = η = 1 L , λ
= αµ 2 and β
= Θ( κ − 4 ) .
Then, 1 K (cid:80) K − 1 k =0 ∥∇ Φ( x k ) ∥ 2 =
O (cid:0) κ 5 K
+ κ 4 K + ϵ (cid:1) , and the complexity is Gc ( ϵ ) = (cid:101) O ( κ 5 ϵ − 1 ) , MV ( ϵ ) = (cid:101) O ( κ 6 ϵ − 1 ) .
4.2 Comparison among Four Implementations
Impact of N -loop ( N = 1 vs N = κ ln κ ).
We fix Q , and compare how the choice of N affectsthe computational complexity.
First, let Q = Θ(1) , and compare the results between the two implementations N -loop with Θ( κ ln κ ) (Corollary 1) and No-loop with N = 1 (Corollary 2).
Clearly,the N -loop scheme significantly improves the convergence rate of the No-loop scheme from O ( κ 6 K ) to
O ( κ 4 K ) , and improves the matrix-vector and gradient complexities from (cid:101) O
( κ 6 ϵ − 1 ) and (cid:101) O
( κ 6 ϵ − 1 ) to(cid:101)
O ( κ 4 ϵ − 1 ) and (cid:101) O
( κ 5 ϵ − 1 ) , respectively.
To explain intuitively, the hypergradient estimation involvesa coupled error η ∥
y Nk − y ∗ ( x k ) ∥ induced from solving the linear system ∇ 2 y g ( x k , y Nk ) v = ∇ y f
( x k , y Nk )with stepsize η .
Therefore, a smaller inner-level approximation error ∥
y Nk − y ∗ ( x k ) ∥ allows a moreaggressive stepsize η , and hence yields a faster convergence rate as well as a lower total complexity,as also demonstrated in our experiments.
It is worth noting that such a comparison is generallydifferent from that in minimax optimization [24, 37], where alternative (i.e., No-loop) gradientdescent ascent (GDA) with N = 1 outperforms (N-loop) GDA with N = κ ln κ , where N denotesthe number of ascent iterations for each descent iteration.
To explain the reason, in constrast tominimax optimization, the gradient w.r.t.
x in bilevel optimization involves additional second-orderderivatives, which are more sensitive to the inner-level approximation error.
Therefore, a larger Nis more beneficial for bilevel optimization than minimax optimization.
Similarly, we can also fix
Q = Θ( κ ln κ ) , the N - Q -loop scheme with N = κ ln κ (Corollary 3) significantly outperforms the
Q -loop scheme with N = 1 (Corollary 4) in terms of the convergence rate and complexity.
Impact of Q -loop ( Q = 1 vs Q = Θ( κ ln κϵ ) ).
We fix N , and characterize the impact of the choice of
Q on the complexity.
For N = 1 , comparing No-loop with Q = Θ(1) in Corollary 2 and Q -loop with
Q = Θ( κ ln κ ) in Corollary 4 shows that both choices of Q yield the same matrix-vector complexity(cid:101)
O ( κ 6 ϵ − 1 ) , but Q -loop with a larger Q improves the gradient complexity of No-loop with Q = Θ(1)from (cid:101) O
( κ 6 ϵ − 1 ) to (cid:101) O
( κ 5 ϵ − 1 ) .
A similar phenomenon can be observed for N = Θ( κ ln κ ) based onthe comparision between N - Q -loop in Corollary 3 and N -loop in Corollary 1.
In deep learning.
Also note that in the setting where the matrix-vector complexity dominates thegradient complexity, e.g., in deep learning, such two choices of Q do not affect the total computationalcomplexity.
However, a smaller Q can help reduce the per-iteration load on the computational resourceand memory, and hence is preferred in practical applications with large models.
Comparison among four implementations.
By comparing the complexity results in Corollaries 1, 2,3 and 4, it can be seen that N - Q -loop and N -loop (both with a large N = Θ( κ ln κ ) ) achieve thebest matrix-vector complexity (cid:101) O ( κ 4 ϵ − 1 ) , whereas Q -loop and No-loop (both with a smaller N = 1 )require higher matrix-vector complexity of (cid:101) O
( κ 6 ϵ − 1 ) .
Also note that N - Q -loop has the lowestgradient complexity.
This suggests that the introduction of the inner loop with large N can help toreduce the total computational complexity.
Convergence Analysis of ITD-BiO
In this section, we first provide a unified theory for ITD-BiO, which is applicable for all choices of
N , and then specialize the convergence theory to characterize the computational complexity for thetwo implementations of ITD-BiO: No loop and N - N -loop.
We also provide a convergence lowerbound to justify the necessity of choosing large N to achieve a vanishing convergence error.
Thefollowing theorem characterizes the convergence rate of ITD-BiO for all choices of N .
Theorem 3.
Suppose Assumptions 1, 2, 3 and 4 hold.
Define w = (cid:0) 1 + 2 αµ (cid:1) L 2 µ 2 (1 − αµ ) N λ N +4 M 2 w 2 N L 2 µand τ = N 2 (1 − αµ ) N + w 2 N + λ N (1 − αµ ) N , where λ N and w N are given by
[Equation]
N =
[Equation]
M 2 w 2 N
[Equation]
(1 − 14 αµ
[Equation] [Equation] αLN
[Equation]
Chooseparameters such that β 2 ≤ 1 − 14 αµ 2 w , α ≤ 12 L and βL Φ + 8 αµ (cid:16) 12 + βL Φ (cid:17) wβ 2 < 14 , where L Φ = L + 2 L 2 + ρM 2 µ [Equation]
+
[Equation] M µdenotes the smoothness parameter of Φ( · ) .
Then, we have
[Equation]
where ∆ Φ =
Φ( x 0 ) − min x Φ( x ) and ∆ y = ∥
y 0 − y ∗ ( x 0 ) ∥ 2 .
In Theorem 3, the upper bound on the convergence rate for ITD-BiO contains a convergent term
O ( 1 K ) (which converges to zero sublinearly with K ) and an error term O (cid:0) M 2 (1 − αµ ) 2 N αµ(cid:1) (which is independent of K , and possibly non-vanishing if N is chosen to be small).
To show that such apossibly non-vanishing error term (when N is chosen to be small) fundamentally exists, we nextprovide the following lower bound on the convergence rate of ITD-BiO.
Theorem 4 ( Lower Bound).
Consider the ITD-BiO algorithm in Algorithm 2 with α ≤ 1 L , β ≤ 1 L Φand N ≤ O (1) , where L Φ is the smoothness parameter of Φ( x ) .
There exist objective functionsf ( x, y ) and g ( x, y ) that satisfy Assumptions 1, 2, 3 and 4 such that for all iterates x K (where K ≥ 1 )generated by ITD-BiO in Algorithm 2, ∥∇ Φ( x K ) ∥ 2 ≥ Θ (cid:0) L 2 M 2 µ 2 (cid:0) 1 − αµ (cid:1) 2 N (cid:1) .
Clearly, the error term in the upper bound given in Theorem 3 matches the lower bound given in
Theorem 4 in terms of M 2 L 2 µ 2 (1 − αµ ) 2 N , and there is still a gap on the order of αµ , which requiresfuture efforts to address.
Theorem 3 and Theorem 4 together indicate that in order to achieve anϵ -accurate stationary point, N has to be chosen as large as N = Θ( κ log κϵ ) .
This corresponds tothe N - N -loop implementation of ITD-BiO, where large N achieves a highly accurate hypergradientestimation in each step.
Another No-loop implementation chooses a small constant-level N =Θ(1) to achieve an efficient execution per step, where a large N can cause large memory usageand computation cost.
Following from Theorem 3 and Theorem 4, such No-loop implementationnecessarily suffers from a non-vanishing error.
In the following corollaries, we further specialize Theorem 3 to obtain the complexity analysis for
ITD-BiO under the two aforementioned implementations of ITD-BiO.
Corollary 5 ( N - N -loop) .
Consider N - N -loop ITD-BiO with N = Θ( κ ln κϵ ) .
Under the same setting of Theorem 3, choose β = min (cid:110)(cid:112) αµ 40 w , (cid:113) 1 − αµ 4 2 w , 1 8 L Φ (cid:111) , α = 12 L .
Then, 1 K (cid:80) K − 1 k =0 ∥∇ Φ( x k ) ∥ 2 =
O (cid:0) κ 3 K + ϵ (cid:1) , and the complexity is Gc ( ϵ )
= (cid:101)
O
( κ 4 ϵ − 1 ) , MV ( ϵ ) =
(cid:101)
O ( κ 4 ϵ − 1 ) .
Corollary 5 shows that for a large N = Θ( κ ln κϵ ) , we can guarantee that ITD-BiO convergesto an ϵ -accurate stationary point, and the gradient and matrix-vector product complexities aregiven by (cid:101) O
( κ 4 ϵ − 1 ) .
We note that [17] also analyzed the ITD-BiO with N = Θ
( κ ln κϵ ) , andprovided the same complexities as our results in Corollary 5.
In comparison, our analysis has severaldifferences.
First, [17] assumed that the minimizer y ∗ ( x k ) at the k th iteration is bounded, whereasour analysis does not impose this assumption.
Second, [17] involved an additional error term
[Equation]
, which can be very large (or even unbounded) under standard
Assumptions 1, 2, 3 and 4.
We next characterize the convergence for the small N = Θ(1) .
Corollary 6 (No-loop) .
Consider No-loop ITD-BiO with N = Θ(1) .
Under the same setting of
Theorem 3, choose stepsizes α = 1 2 NL and β = min (cid:8)(cid:112)
αµ 40 w , (cid:113) 1 − αµ 4 2 w , 18 L Φ (cid:9) .
Then, we have1 K (cid:80)
[Equation]
(cid:0)
[Equation]
Corollary 6 indicates that for the constant-level N = Θ(1) , the convergence bound contains anon-vanishing error
O ( M 2 L 2 αµ 3 ) .
As shown in the convergence lower bound in Theorem 4, understandard Assumptions 1, 2, 3 and 4, such an error is unavoidable.
Comparison between the abovetwo corollaries suggests that for ITD-BiO, the N - N -loop is necessary to guarantee a vanishingconvergence error, whereas No-loop necessarily suffers from a non-vanishing convergence error.
Discussion on the setting with small response Jacobian.
Our results in Theorem 3 and Theorem 4apply to the general functions whose first- and second-order derivatives are Lipschitz continuous, i.e.,under Assumptions 2 and 3. Here, we further discuss the extension of our results to another settingwhere the response Jacobian is extremely small. This setting occurs in some deep learning applications [5, 15], where the response Jacobian ∂y ∗ ( x ) ∂x (which is estimated by ∂y Nk ( x ) ∂x with a large N ) canbe order-of-magnitude smaller than network gradients. Based on eq.
{+(60) and eq. (62) in the appendix,it can be shown that the convergence error is proportional to the quantity 1 K (cid:80) K − 1 k =
 ∥ ∂y ∗ ( x k ) ∂x k ∥ 2 , andhence the constant-level N = Θ(1) can still achieve a small error in this setting.+}
Experiments on AID-BiO. We first conduct experiments to verify our theoretical results in Corollaries 1, 2, 3 and 4 on AID-BiO with different implementations.
We consider the following hyperparam- eter optimization problem.min λ L D val ( λ ) =
|D val | (cid:88) ξ ∈D val L ( w ∗ ; ξ ) , s.t. w ∗ =
arg min w |D tr | (cid:88) ξ ∈D tr (cid:16) L ( w ; ξ ) + λ 2 ∥ w ∥ 22 (cid:17) , where D tr and D val stand for training and validation datasets, L ( w ; ξ ) denotes the loss functioninduced by the model parameter w and sample ξ , and λ > 0 denotes the regularization parameter.
The goal is to find a good hyperparameter λ to minimize the validation loss evaluated at the optimalmodel parameters for the regularized empirical risk minimization problem.
[Figure]
From Figure 1, we can make the following observations.
First, the learning curves with N = 20 aresignificantly better than those with N = 1 , indicating that running multiple steps of gradient descentin the inner loop (i.e., N > 1 ) is crucial for fast convergence.
This observation is consistent with ourcomplexity result that N -loop is better than No-loop, and N - Q -loop is better than Q -loop, as shownin Table 1.
The reason is that a more accurate hypergradient estimation can accelerate the convergencerate and lead to a reduction on the Jacobian- and Hessian-vector computational complexity.
Second, N - Q -loop ( N = 20 , Q = 20 ) and N -loop ( N = 20 , Q = 1 ) achieve a comparable convergenceperformance, and a similar observation can be made for Q -loop ( N = 1 , Q = 20 ) and No-loop( N = 1 , Q = 1 ).
This is also consistent with the complexity result provided in Table 1, wheredifferent choices of Q do not affect the dominant matrix-vector complexity.
Experiments on ITD-BiO.
We consider a hyper-representation problem in [35], where the innerproblem is to find optimal regression parameters w and the outer procedure is to find the bestrepresentation parameters λ .
In specific, the bilevel problem takes the following form:
[Equation]
where X T ∈ R q × m and X V ∈ R p × m are synthesized training and validation data, Y T ∈
R q , Y V ∈ {+
+} R p are their response vectors, and h ( · ) is a linear transformation.
The generation of X T , X V , Y T , Y Vand the experimental setup follow from [35].
We choose N = 20 for N - N -loop ITD and N = 1 for No-loop ITD.
The results are reported with the best-tuned hyperparameters.
[Table]
Table 3 indicates that N - N -loop with N = 20 can achieve a small loss value of 0 .
after 500 totaliterations, whereas No-loop with N = 1 converges to a much larger loss value of 0 . 04 .
This is inconsistence with our theoretical results in Table 2, where N = 1 can cause a non-vanishing error.
Conclusion
In this paper, we study two popular bilevel optimizers AID-BiO and ITD-BiO, whose implementationspotentially involve additional loops of iterations within their base-loop update.
By developingunified convergence analysis for all choices of the loop parameters, we are able to provide formalcomparison among different implementations.
Our result suggests that N -loops are beneficial forbetter computational efficiency for AID-BiO and for better convergence accuracy for ITD-BiO. Thisis in contrast to conventional minimax optimization, where No-loop (i.e., single-base-loop) schemeachieves better computational efficiency.
Our analysis techniques can be useful to study other bileveloptimizers such as stochastic optimizers and variance reduced optimizers.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Table 2 shows that there exists
a gap of κ between the upper and lower bounds for ITD-based bilevel optimization,which requires future efforts to address.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This
paper develops the convergence theory for the fundamental belivel algorithms.
Our
analysis will not have any potential negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] All assumptions
are stated in Section 3.
(b) Did you include complete proofs of all theoretical results?
[Yes] Complete proofs are included in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] The experimental details are specified in Appendix B.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] See Appendix B. We ran 5 random seeds for everyexperiment.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] The details are included in
Appendix B.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] The details are
included in Appendix B.
(b) Did you mention the license of the assets?
[Yes] The details are included in Appendix B.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
Our code is based on the existing asset.
See Appendix B for details.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
The datasets we use are public.
See Appendix B for details.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
The dataset we use does not contain personally
identifiable information, nor offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]