On the Convergence Theory for Hessian-Free Bilevel Algorithms
Abstract
Bilevel optimization has arisen as a powerful tool in modern machine learning.
However, due to the nested structure of bilevel optimization, even gradient-based methods require second-order derivative approximations via Jacobian- or/and
Hessian-vector computations, which can be costly and unscalable in practice.
Recently, Hessian-free bilevel schemes have been proposed to resolve this issue, where the general idea is to use zeroth- or first-order methods to approximate the full hypergradient of the bilevel problem.
However, we empirically observe that such approximation can lead to large variance and unstable training, but estimating only the response Jacobian matrix as a partial component of the hypergradient turns out to be extremely effective.
To this end, we propose a new Hessian-free method, which adopts the zeroth-order-like method to approximate the response Jacobian matrix via taking difference between two optimization paths.
Theoretically, we provide the convergence rate analysis for the proposed algorithms, where our key challenge is to characterize the approximation and smoothness properties of the trajectory-dependent estimator, which can be of independent interest.
This is the first known convergence rate result for this type of Hessian-free bilevel algorithms.
Experimentally, we demonstrate that the proposed algorithms outperform baseline bilevel optimizers on various bilevel problems.
Particularly, in our experiment on few-shot meta-learning with ResNet-12 network over the miniImageNet dataset, we show that our algorithm outperforms baseline meta-learning algorithms, while other baseline bilevel optimizers do not solve such meta-learning problems within a comparable time frame.
Introduction
Bilevel optimization has recently arisen as a powerful tool to capture various modern machine learning problems, including meta-learning [4, 12, 49, 24, 36], hyperparamater optimization [12, 51], neural architecture search [35, 58], signal processing [10], etc.
Bilevel optimization generally takes the following mathematical form:
[Equation]
where the outer objective f : R p × R d
→ R is continuously differentiable function and the inner objective g : R p × R
d
→ R is twice differentiable.
Gradient-based methods have served as a popular tool for solving bilevel optimization problems.
Two types of approaches have been widely used: the iterative differentiation (ITD) method [7, 11, 51] and the approximate iterative differentiation (AID) method [7, 47, 42].
Due to the bilevel structure of the problem, even such gradient-based methods typically involve second-order matrix computations, [Figure] because the gradient of Φ( x ) in eq.
(1) (which is called hypergradient ) involves the optimal solution of the inner function.
Although the ITD and AID methods often adopt Jacobian- or/and Hessian-vector implementations, the computation can still be very costly in practice for high-dimensional problems with neural networks.
To overcome the computational challenge of current gradient-based methods, a variety of Hessian-free bilevel algorithms have been proposed.
For example, popular approaches such as FOMAML [9, 35] ignores the calculations of any second-order derivatives.
However, such a trick has no guaranteed performance and may suffer from inferior test performance [1, 8].
In addition, when the outer-level function f ( x, y ) depends only on y variable, e.g., in some hyperparameter optimization applications [12], it can be shown from eq.
(2) that the hypergradient ∇ Φ( x ) vanishes if we eliminate all second-order directives.
More recently, several zeroth-order methods [18] have been proposed to approximate the full hypergradient ∇ Φ( x ) .
In particular, ES-MAML [54] and HOZOG [18] use zeroth-order methods to approximate the full hypergraident based on the objective function evaluations.
However, as demonstrated in Figure 1 (also see our further experiments in Section 4), we empirically observe that such full hypergradient estimation can encounter a large variance and inferior performance, whereas a new zeroth-order-like method PZOBO with partial hypergradient estimation (as we propose below) performs significantly better.
Encouraged by such an interesting observation, we propose a simple but effective Hessian-free method named PZOBO, which stands for partial zeroth-order-like bilevel optimizer.
PZOBO uses a zeroth-order-like approach to approximate only the response Jacobian ∂y ∗ ( x ) ∂x (which is the major computational bottleneck of hypergradient) based on the difference between two gradient-based optimization paths.
The remaining terms in hypergradient can simply be calculated by their analytical forms.
In this way, PZOBO avoids the large variance of the zeroth-order method for estimating the entire hypergradient, but still enjoys its Hessian-free advantage.
We further show that PZOBO admits an easy extension to the large-scale stochastic setting by simply taking small-batch gradients without introducing any complex sub-procedure such as the Neumann Series (NS) type of construction in [23].
Experimentally, PZOBO and its stochastic version PZOBO-S achieve superior performance compared to the current baseline bilevel optimizers, and such an improvement is robust across various bilevel problems.
In particular, on the few-shot meta-learning problem over a ResNet-12 network, PZOBO outperforms the state-of-the-art optimization-based meta learners (not necessarily bilevel optimizers), while other bilevel optimizers do not scale to solve such meta-learning problems within a comparable time frame.
Furthermore, our zeroth-order-like response Jacobian estimator in PZOBO takes the difference between two optimization-based trajectories, whereas the vanilla zeroth-order method uses the function value difference at two close points for approximation.
Such difference complicates the analysis of PZOBO from three perspectives.
(a) Conventional zeroth-order analysis often requires some Lipshitzness properties (e.g., smoothness) of the objective function, whereas they may not hold for the optimization-based output of our response Jacobian estimator.
(b) It is unclear if the trajectory-based Hypergradient estimator has bounded error at each iteration, which is critical in the convergence analysis for bilevel optimization [14, 23, 22].
(c) Such characterizations are more challenging for stochastic settings because the randomness along the trajectory needs to be considered.
Theoretically, we provide the convergence rate guarantee for both PZOBO and PZOBO-S. To the best of our knowledge, this is the first known non-asymptotic performance guarantee for Hessian-free bilevel algorithms via zeroth-order-like approximation.
Technically, in contrast to the conventional analysis of the zeroth-order estimation on the smoothed blackbox function values, we develop tools to characterize the bias, variance, smoothness and boundedness of the proposed response
Jacobian estimator via a recursive analysis over the gradient-based optimization path, which can be of independent interest for zeroth-order estimation based bilevel optimization.
1.1 Related Work
Bilevel optimization.
Bilevel optimization has been studied for decades since [5].
A variety of bilevel optimization algorithms were then developed, including constraint-based approaches [21, 52, 45], approximate implicit differentiation (AID) approaches [7, 47, 16, 34, 42] and iterative differentiation (ITD) approaches [7, 44, 11, 9, 51, 49, 39].
These methods often suffer from expensive computations of second-order information (e.g., Hessian-vector products).
Hessian-free algorithms were recently proposed based on interior-point method [37] and zeroth-order approximation [18, 54].
[38] proposed an initialization auxiliary method to deal with nonconvex inner problem.
However, these methods may suffer from either a costly procedure in regularization parameter tuning [37, 38] or a large estimation variance [18, 54] in supervised learning.
This paper proposes a simpler and more efficient
Hessian-free approach via exploiting the benign structure of the hypergradient and a zeroth-order-like response Jacobian estimation.
Recently, the convergence rate has been established for gradient-based (Hessian involved) bilevel algorithms [17, 28, 49, 25].
This paper provides a new convergence analysis for the proposed zeroth-order-like bilevel approach.
Stochastic bilevel optimization.
[14, 28, 22] proposed stochastic gradient descent (SGD) type bilevel optimization algorithms by employing Neumann Series for Hessian-inverse-vector approximation.
Recent works [29, 30, 6, 19, 20, 57] then leveraged momentum-based variance reduction to further reduce the computational complexity of existing SGD-type bilevel optimizers from O ( ϵ − 2 ) to
O ( ϵ − 1 . 5 ) .
However, these methods often do not scale well in large models due to the Hessian or
Hessian inverse computations.
In this paper, we propose a stochastic Hessian-free method, which eliminates the computation of all second-order information.
Bilevel optimization applications.
Bilevel optimization has been employed in various applications such as few-shot meta-learning [53, 12, 49, 59, 24, 27], hyperparameter optimization [11, 43, 51], neural architecture search [35, 58], etc.
For example, [43] models the response function itself as a neural network (where each layer involves an affine transformation of hyperparameters) using the
Self-Tuning Networks (STNs).
An improved and more stable version of STNs was further proposed in [3], which focused on accurately approximating the response Jacobian rather than the response function itself.
This paper demonstrates the superior performance of the proposed Hessian-free bilevel optimizer with guaranteed performance in meta-learning and hyperparameter optimization.
Zeroth-order methods and applications.
Zeroth-order optimization methods have been studied for a long time.
For example, [46] proposed an effective zeroth-order gradient estimator via Gaussian smoothing, which was further extended to the stochastic setting by [13].
Such a technique has exhibited great effectiveness in various applications including meta-reinforcement learning [54], hyperparameter optimization [18], adversarial machine learning [26, 40], minimax optimization [41, 56], etc.
This paper proposes a novel Jacobian estimator for accelerating bilevel optimization based on an idea similar to zeroth-order estimation via the difference between two optimization paths.
Proposed Algorithms 2.1 Hypergradients
The key step in popular gradient-based bilevel optimizers is the estimation of the hypergradient (i.e., the gradient of the objective with respect to the outer variable x ), which takes the following form:
[Equation]
where the Jacobian matrix J ∗ ( x ) =
[Equation]
( x ) [Equation] ∈ R d × p .
Following [42], it can be seen that ∇ Φ( x ) contains two components: the direct gradient ∇ x f ( x, y ∗ ( x )) and the indirect gradient
J ∗ ( x ) ⊤ ∇ y f
( x, y ∗ ( x )) .
The direct component can be efficiently computed using the existing automatic differentiation techniques.
The indirect component, however, is computationally much more complex, because J ∗ ( x ) takes the form of J ∗ ( x ) = − (cid:2) ∇ 2 y g ( x, y ∗ ( x )) (cid:3) − 1 ∇ x ∇ y g ( x, y ∗ ( x ))
(if ∇ 2 y g ( x, y ∗ ( x )) is invertible), which contains the Hessian inverse and the second-order mixed derivative.
Some approaches mitigate the issue by designing Jacobian-vector and Hessian-vector do 3:
Set
Q 4: do 5: k − α ∇ y
6: end for 7: Q do 8:
Generate u k j = N ( I
∈ p 9: for do 10:
Update y tk,j = y t − 1 k,j − α ∇ y g (cid:0) x k + µu k,j , y t − 1 k,j (cid:1) 11: end for 12:
Compute δ j = y Nk,j − y Nk µ 13: end for 14:
Compute (cid:98) ∇ Φ( x k ) =
∇ x f ( x k , y Nk ) + 1 Q (cid:80)
Qj =1 (cid:10) δ j , ∇ y f
( x k , y Nk ) (cid:11) u k,j 15:
Update x k +
=
x
k − β (cid:98) ∇ Φ( x k ) 16: end for products [47, 11, 17] to replace second-order computations. But the computation is still costly for high-dimensional bilevel problems such as those with neural network variables. We next introduce the zeroth-order approach which is at the core for designing efficient Hessian-free bilevel optimizers. 2.2 Zeroth-Order Approximation
Zeroth-order approximation is a powerful technique to estimate the gradient of a function based on function values, when it is not feasible (such as in black-box problems) or computationally costly to evaluate the gradient.
The idea of the zeroth-order method in [46] is to approximate the gradient of a general black-box function h : R n → R using the following oracle based only on the function values
[Equation]
where u ∈ R n is a Gaussian random vector and µ > 0 is the smoothing parameter.
Such an oracle can be shown to be an unbiased estimator of the gradient of the smoothed function E u [ h ( x + µu )] .
2.3 Proposed Bilevel Optimizers
Our key idea is to exploit the analytical structure of the hypergradient in eq.
(2), where the derivatives ∇ x f ( x, y ∗ ( x )) and ∇ y f ( x, y ∗ ( x )) can be computed efficiently and accurately, and hence use the zero-order estimator similar to eq.
(3) to estimate only the Jacobian J ∗ ( x ) , which is the major term posing computational difficulty.
In this way, our estimation of the hypergradient can be much more accurate and reliable.
In particular, our Jacobian estimator contains two ingredients: (i) for a given x , apply an algorithm to solve the inner optimization problem and use the output as an approximation of y ∗ ( x ) ; for example, the output y N ( x ) of N gradient descent steps of the inner problem can serve as an estimate for y ∗ ( x ) .
Then J N ( x ) =
∂y N ( x ) ∂x serves as an estimate of J ∗ ( x ) ; and (ii) construct the zeroth-order-like Jacobian estimator ˆ J N ( x ; u
) ∈ R
d × p for J N ( x ) as
[Equation]
where u ∈ R p is a Gaussian vector with independent and identically distributed (i.i.d.) entries.
Then for any vector v ∈ R d , the Jacobian-vector product can be efficiently computed using only vector-vector dot product ˆ J N ( x ; u ) ⊤ v = ⟨ δ ( x ; u ) , v ⟩ u , where δ ( x ; u ) =
y N ( x + µu ) − y N ( x ) µ ∈ R d .
PZOBO: partial zeroth-order based bilevel optimizer.
For the bilevel problem in eq.
(1), we design an optimizer (see Algorithm 1) using the Jacobian estimator in eq.
(4), which we call as the PZOBO algorithm.
Clearly, the zeroth-order estimator is used only for estimating partial hypergradient.
At each step k of the algorithm, PZOBO runs an N -step full GD to approximate y Nk ( x k ) .
PZOBO then samples Q Gaussian vectors { u k,j ∈ N (0 , I ) , j = 1 , ..., Q } , and for each sample u k,j , runs an N -step full GD to approximate y Nk ( x k + µu k,j ) , and then computes the Jacobian estimator ˆ J N ( x ; u k,j ) as in eq.
Then the sample average over the Q estimators is used for constructing the following hypergradient estimator for updating the outer variable x .
[Equation]
In our experiments (see Section 4), we choose a small constant-level Q
= O
(1) (e.g., Q = 1 in most applications with neural nets) due to a much better performance.
Our convergence guarantee holds for this case, as shown later.
Computationally, in contrast to the existing AID and ITD bilevel optimizers [47, 12, 17] that contains the complex Hessian-
and/or
Jacobian-vector product computations
, PZOBO has only gradient computations and becomes Hessian-free, and hence is much more efficient as shown in our experiments.
PZOBO-S: stochastic PZOBO.
In machine learning applications, the loss functions f, g in eq.
(1) often take finite-sum forms over given data D n,m = { ξ i , ζ j , i = 1 , ..., n, j = 1 , ..., m } as below.
[Equation]
where the sample sizes n and m are typically very large.
For such a large-scale scenario, we design a stochastic PZOBO bilevel optimizer (see Algorithm 2 in Appendix A), which we call as PZOBO-S.
Differently from Algorithm 1, which applies GD updates to find y N ( x k ) , PZOBO-S uses N stochastic gradient descent (SGD) steps to find { Y Nk , Y Nk, 1 , ..., Y Nk,Q } to the inner problem, each with the outer variable set to be x k + µu k,j .
Note that all SGD runs follow the same batch sampling path {S 0 , ..., S N − 1 } .
The Jacobian estimator ˆ J N ( x k ; u k,j ) can then be computed as in eq.
At the outer level, PZOBO-S samples a new batch D F independently from the inner batches {S 0 , ..., S N − 1 } to evaluate the stochastic gradients ∇ x F ( x k , Y Nk ; D F ) and ∇ y F ( x k , Y Nk ; D F ) .
The hypergradient (cid:98) ∇ Φ( x k ) is then estimated as
[Equation]
Main Theoretical Results 3.1 Technical Assumptions
In this paper, we consider the following types of objective functions.
Assumption 1.
The inner function g ( x, y ) is µ g -strongly convex with respect to y and the outer function f ( x, y ) is possibly nonconvex w.r.t.
x .
For the finite-sum case, the same assumption holds for functions G ( x, y ; ζ ) and F ( x, y ; ζ )
The above assumption on f, g has also been adopted in [14, 28, 57].
In fact, many bilevel machine learning problems satisfy this assumption.
For example, in few-shot meta-learning, the task-specific parameters are likely the weights of the last classification layer so that the resulting bilevel problem has a strongly-convex inner problem [48, 37].
Assumption 2.
Let w
=
( x, y ) .
The gradient ∇ g ( w ) is L g -Lipschitz continuous, i.e., for any w 1 , w 2 , ∥∇ g ( w 1 ) − ∇ g ( w 2 )
∥
≤ L g ∥ w 1 − w 2 ∥ ; further, the derivatives ∇ 2 y g ( w ) and ∇ x ∇ y g ( w ) are ρ - and τ -Lipschitz continuous, i.e, ∥∇ 2 y g ( w 1 ) −∇
2 y g ( w 2 )
∥ F ≤ ρ ∥ w 1 − w 2 ∥ and ∥∇ x ∇ y g ( w 1 ) − ∇ x ∇ y g ( w 2 ) ∥ F ≤ τ ∥ w 1 − w 2 ∥ .
The same assumptions hold for G ( w ; ζ ) in the finite-sum case.
Assumption 3.
Let w
=
( x, y ) .
The objective f ( w ) and its gradient ∇ f ( w ) are M - and L f -Lipschitz continuous, i.e., for any w 1 , w 2 , (cid:12)(cid:12) f ( w 1 ) − f ( w 2 ) (cid:12)(cid:12) ≤ M (cid:13)(cid:13) w 1 − w 2 (cid:13)(cid:13) , (cid:13)(cid:13) ∇ f ( w 1 ) − ∇ f ( w 2 ) (cid:13)(cid:13) ≤
L f (cid:13)(cid:13) w 1 − w 2 (cid:13)(cid:13) , which hold for F ( w ; ξ ) in the finite-sum case.
Assumption 4.
For the finite-sum case, the gradient ∇ G ( w ; ζ ) has bounded variance condition, i.e., E ζ ∥∇ G ( w ; ζ ) − ∇ g ( w ) ∥ 2 ≤ σ 2 for some constant σ ≥ 0 .
3.2 Convergence Analysis for PZOBO
Differently from the standard zeroth-order analysis for a blackbox function, here we develop new techniques to analyze the zeroth-order-like Jacobian estimator that depends on the entire inner optimization trajectory, which is unique in bilevel optimization.
We first establish the following important proposition which characterizes the Lipshitzness property of the approximate Jacobian matrix J N ( x ) =
∂y N ( x ) ∂x .
Proposition 1.
Suppose that Assumptions 1 and 2 hold.
Let L J = (cid:16)
+ Lµ g (cid:17) (cid:16) τµ g + ρLµ 2 g (cid:17) , with
L = max { L f , L g } .
Then, the Jacobian J N ( x ) is Lipschitz continuous with constant L J :
[Equation]
We next show that the variance of hypergradient estimation can be bounded.
The characterization of the estimation bias can be found in Lemma 8 in the appendix.
Proposition 2.
Suppose that Assumptions 1, 2, and 3 hold.
The hypergradient estimation variance can be upper-bounded as
[Equation]
where E [ · ] is conditioned on x k and taken over the Gaussian vectors { u k,j : j = 1 , ..., Q } .
Proposition 2 upper bounds the hypergradient estimation variance, which mainly arises due to the estimation variance of the Jacobian matrix J ∗ by the proposed estimator J N , via three types of variances: (a) the approximation variance between J N and J ∗ via inner-loop gradient descent, which decreases exponentially w.r.t.
the number N of inner iterations due to the strong convexity of the inner objective; (b) the variance between our estimator and the Jacobian J µ of the smoothed output E u y N ( x k + µu ) , which decreases sublinearly w.r.t.
the batch size Q , and (c) the variance between the Jacobian J N and J µ , which can be controlled by the smoothness parameter µ .
By using the smoothness property in Proposition 1 and the upper bound in Proposition 2, we provide the following characterization of the convergence rate for PZOBO.
Theorem 1 (Convergence of PZOBO) .
Suppose that Assumptions 1, 2, and 3 hold.
Choose the inner- and outer-loop stepsizes respectively as α ≤ 1 L and β = O ( 1 √ K ) .
Further, set Q = O (1) and µ = O (cid:16)
[Equation]
[Equation]
Then, the iterates x k for k = 0 , ..., K − 1 of PZOBO satisfy
[Equation]
Theorem 1 shows that the convergence rate of PZOBO is sublinear with respect to the number K of outer iterations due to the nonconvexity of the outer objective, and linear (i.e., exponentially decay) with respect to the number N of inner iterations due to the strong convexity.
Theorem 1 also indicates the following features that ensures the efficiency of the algorithms.
(i)
The batch size Q of the Jacobian estimator can be chosen as a constant (in particular Q = 1 as in our experiments) so that the computation of the ES estimator is efficient.
(ii)
The number of inner iterations can be chosen to be small due to its exponential convergence, and hence the algorithm can run efficiently.
(iii) PZOBO requires only gradient computations to converge, and eliminates
Hessian- and Jacobian-vector products required by the existing AID and ITD based bilevel optimizers ([47, 12, 17].
Thus, PZOBO is more efficient particularly for high-dimensional problems.
3.3 Convergence Analysis for PZOBO-S
In this section, we apply the stochastic algorithm PZOBO-S to the finite-sum objective in eq.
(6) and analyze its convergence rate.
The following proposition establishes an upper bound on the estimation error of Jacobian J ∗ by J N = ∂Y N ∂x , where Y N is the output of N inner SGD updates.
Proposition 3.
Suppose that Assumptions 1, 2, and 4 hold.
Choose the inner-loop stepsize as α = 2 L + µ g , where L = max { L f , L g } .
Then, we have:
[Equation]
[Figure]
where λ , Γ , and C γ < 1 are constants (see Appendix J for their precise forms).
We next show that the variance of hypergradient estimation is bounded
[Figure]
Proposition 4.
Suppose that Assumptions 1, 2, 3, and 4 hold.
Set the inner-loop stepsize as α = 2 L + µ g where L = max { L f , L g } .
Then, we have:
[Equation]
where S and D f are the sizes of the inner and outer mini-batches, respectively.
Based on Propositions 1, 3, and 4, we characterize the convergence rate for PZOBO-S.
Theorem 2 (Convergence of PZOBO-S) .
Suppose that Assumptions 1, 2, 3, and 4 hold.
Set the innerand outer-loop stepsizes respectively as α = 2 L + µ g and β =
O ( 1 √ K ) , where L = max { L f , L g } .
Further, set Q = O (1) , D f = O (1) , and µ = O (cid:16) 1 √ Kdp 3 (cid:17) .
Then, the iterates x k , k = 0 , ..., K −of PZOBO-S satisfy:
[Equation]
Comparing to the convergence bound in Theorem 1 for the deterministic algorithm PZOBO, Theorem 2 for the stochastic algorithm PZOBO-S captures one more sublinearly decreasing error term 1 √ S due to the sampling of inner batches to estimate the objectives.
Note that the sampling of outer batches has been included into the sublinear decay term w.r.t.
the number K of outer-loop iterations.
Experiments
We validate our algorithms in four bilevel problems: shallow hyper-representation (HR) with linear/2-layer net embedding model on synthetic data, deep HR with LeNet network [32] on MNIST [Figure] dataset, few-shot meta-learning with ResNet-12 on miniImageNet dataset, and hyperparameter optimization (HO) on the 20 Newsgroup dataset.
We run all models using a single NVIDIA Tesla
P100 GPU.
All running time are in seconds.
4.1 Shallow Hyper-Representation on Synthetic Data
The hyper-representation (HR) problem [12, 17] searches for a regression (or classification) model following a two-phased optimization process.
The inner-level identifies the optimal linear regressor parameters w , and the outer level solves for the optimal embedding model (i.e., representation) parameters λ .
Mathematically, the problem can be modeled by the following bilevel optimization:
[Equation]
where X 2 ∈ R n 2 × m and X 1 ∈ R n 1 × m are matrices of synthesized training and validation data, and Y 2 ∈ R n 2 , Y 1 ∈ R n 1 are the corresponding response vectors.
In the case of shallow HR, the embedding function T ( · ; λ ) is either a linear transformation or a two-layer network.
We generate data matrices X 1 , X 2 and labels Y 1 , Y 1 following the same process in [17].
We compare our PZOBO algorithm with the baseline bilevel optimizers AID-FP, AID-CG, ITD-R, and HOZOG (see Appendix E.1 for details about the baseline algorithms and hyperparameters used).
Figure 2 show the performance comparison among the algorithms under linear and two-layer net embedding models.
It can be observed that for both cases, our proposed method PZOBO converges faster than all the other approaches, and the advantage of PZOBO becomes more significant in
Figure 2 (d), which is under a higher-dimensional model of a two-layer net.
In particular, PZOBO outperforms the existing ES-based algorithm HOZOG.
This is because HOZOG uses the ES technique to approximate the entire hypergradient, which likely incurs a large estimation error.
In contrast, our
PZOBO exploits the structure of the hypergradient and only estimate the response Jacobian so that the estimation of hypergradient is more accurate.
Such an advantage is more evident under a two-layer net model, where HOZOG does not converge as shown in Figure 2 (d).
This can be explained by the flat hypergradient norm as shown in Figure 2 (e), which indicates that the hypergradient estimator in
HOZOG fails to provide a good descent direction for the outer optimizer.
Figure 2 (c) and (f) further show that the convergence of PZOBO does not change substantially with the number N of inner GD steps, and hence tuning of N in practice is not costly.
4.2 Deep Hyper-Representation on MNIST Dataset
In order to demonstrate the advantage of our proposed algorithms in neural net models, we perform deep hyper-representation to classify MNIST images by learning an entire LeNet network.
The problem formulation is described in Appendix E.3.
Figure 3 compares the classification accuracy on the outer dataset D out between our PZOBO-S and other stochastic baseline bilevel optimizers including two AID-based stochastic algorithms AID-FP and AID-CG, and a recently proposed new stochastic bilevel optimizer stocBiO which has been demonstrated to exhibit superior performance.
Note that ITD and HOZOG are not included in the comparison, because there have not been stochastic algorithms proposed based on these approaches in the literature yet.
Our algorithm PZOBO-S converges with the fastest rate and attains the best [Figure] Algorithm [Figure]
accuracy with the lowest variance among all algorithms.
Note that PZOBO-S is the only bilevel method that is able to attain the same accuracy of 0 .
+ obtained by the state-of-the-art training of all parameters with one-phased optimization on the MNIST dataset using the same backbone network.
All other bilevel methods fail to recover such a level of performance, but instead saturate around an accuracy of 0 .
Further, Figure 3(c) indicates that the convergence of PZOBO-S does not change substantially with the number N of inner SGD steps.
This demonstrates the robustness of our method when applied to complex function geometries such as deep nets.
4.3 Few-Shot Meta-Learning over MiniImageNet
We study the few-shot image recognition problem, where classification tasks T i , i = 1 , ..., m are sampled over a distribution P T .
In particular, we consider the following commonly adopted metalearning setting (e.g., [48], where all tasks share common embedding features parameterized by ϕ , and each task T i has its task-specific parameter w i for i
= 1 , ..., m .
More specifically, we set ϕ to be the parameters of the convolutional part of a deep CNN model (e.g., ResNet-12 network) and w includes the parameters of the last classification layer.
All model parameters ( ϕ , w ) are trained following a bilevel procedure.
In the inner-loop, the base learner of each task T i fixes ϕ and minimizes its loss function over a training set S i to obtain its adapted parameters w ∗ i .
At the outer stage, the meta-learner computes the test loss for each task T i using the parameters ( ϕ , w ∗ i ) over a test set D i , and optimizes the parameters ϕ of the common embedding function by minimizing the meta-objective
L meta over all classification tasks.
The detailed problem formulation is given in Appendix E.4.
[Figure] We conduct few-shot meta-learning on the miniImageNet dataset [55] using two different backbone networks for feature extraction: ResNet-12 and CNN4 [55].
The dataset and hyperparameter details can be found in Appendix E.4.
We compare our algorithm PZOBO with four baseline methods for few-shot meta-learning MAML [9], ANIL [48], MetaOptNet [33], and ProtoNet [53].
Other bilevel optimizers are not included into comparison because they do not solve the problem within a comparable time frame.
Zeroth-order ES-MAML [54] is not included because it exhibits large variance and cannot reach a desired accuracy.
Also note that since ProtoNet and MetaOptNet are usually presented in the ResNet setting, and are not relevant in smaller scale networks, we include them into comparison only for our ResNet-12 experiment.
We run their efficient Pytorch Lightning implementations available at the learn2learn repository [2].
Figure 5(a) and (b) show that our algorithm PZOBO converges faster than the other baseline metalearning methods.
Also, Comparing Figure 5(a) and (b), the advantage of our method over the baselines MAML and ANIL becomes more significant as the size of the network increases.
Further,
Figure 5(c) shows that MetaOptNet did not reach 69% accuracy after 20 hours of training with
ResNet-12 network.
In comparison, our PZOBO is able to attain 69% in less than three hours, which is about 1.5 times less than the time taken for ProtoNet to reach the same performance level.
Both
PZOBO and ProtoNet saturate around 70% accuracy after 10 hours of training.
4.4 Hyperparameter Optimization
Hyperparameter optimization (HO) is the problem of finding the set of the best hyperparamters (either representational or regularization parameters) that yield the optimal value of some criterion of model quality (usually a validation loss on unseen data).
HO can be posed as a bilevel optimization problem in which the inner problem corresponds to finding the model parameters by minimizing a training loss (usually regularized) for the given hyperparameters and then the outer problem minimizes over the hyperparameters.
Due to space limitations, we provide a more complete description of the problem and settings in Appendix F.
It can be seen from Figure 6, our method PZOBO slightly outperforms HOZOG and converges faster than the other AID and ITD based approaches.
We note that the similar performance for PZOBO and HOZOG can be explained by the fact that in HO, the hypergradient expession in eq.
(5) contains only the second term (the first term is zero), which is very close to the approximation in HOZOG method.
However, as we have seen in the HR experiments in Figure 2, PZOBO achieves a much better performance than HOZOG.
Thus, compared to HOZOG, our PZOBO is much more stable and achieves superior performance across many bilevel problems.
Conclusion
In this paper, we propose a novel Hessian-free approach for bilevel optimization based on a zerothorder-like Jacobian estimator.
Compared to the existing such types of Hessian-free algorithms, our approach explores the analytical structure of the hypergradient, and hence leads to much more efficient and accurate hypergradient estimation.
Thus, our algorithm outperforms existing baselines in various experiments, particularly in high-dimensional problems.
We also provide the convergence guarantee and characterize the convergence rate for our proposed algorithms.
We anticipate that our approach and analysis will be useful for bilevel optimization and various machine learning applications.
Acknowledgements
The work of D. Sow and Y. Liang was supported in part by the U.S. National Science Foundation under the grant CCF-1900145 and CNS-2112471.