On the Convergence Theory for Hessian-Free Bilevel Algorithms
Anonymous Author(s)
AffiliationAddress email
Abstract
Bilevel optimization has arisen as a powerful tool in modern machine learning.
However, due to the nested structure of bilevel optimization, even gradient-basedmethods require second-order derivative approximations via Jacobian- or/and
Hessian-vector computations, which can be costly and unscalable in practice.
Recently, Hessian-free bilevel schemes have been proposed to resolve this issue,where the general idea is to use zeroth- or first-order methods to approximate thefull hypergradient of the bilevel problem.
However, we empirically observe thatsuch approximation can lead to large variance and unstable training, but estimatingonly the response Jacobian matrix as a partial component of the hypergradient turnsout to be extremely effective.
To this end, we propose a new Hessian-free method,which adopts the zeroth-order-like method to approximate the response Jacobianmatrix via taking difference between two optimization paths.
Theoretically, weprovide the convergence rate analysis for the proposed algorithms, where our keychallenge is to characterize the approximation and smoothness properties of thetrajectory-dependent estimator, which can be of independent interest.
This is thefirst known convergence rate result for this type of Hessian-free bilevel algorithms.
Experimentally, we demonstrate that the proposed algorithms outperform baselinebilevel optimizers on various bilevel problems.
Particularly, in our experiment onfew-shot meta-learning with ResNet-12 network over the miniImageNet dataset,we show that our algorithm outperforms baseline meta-learning algorithms, whileother baseline bilevel optimizers do not solve such meta-learning problems withina comparable time frame.
Introduction
Bilevel optimization has recently arisen as a powerful tool to capture various modern machine learningproblems, including meta-learning [4, 12, 49, 24, 36], hyperparamater optimization [12, 51], neuralarchitecture search [35, 58], signal processing [10], etc.
Bilevel optimization generally takes thefollowing mathematical form:
[Equation]
where the outer objective f : R p × R d
→ R is continuously differentiable function and the innerobjective g : R p × R
d
→ R is twice differentiable.
Gradient-based methods have served as a popular tool for solving bilevel optimization problems.
Twotypes of approaches have been widely used: the iterative differentiation (ITD) method [7, 11, 51] andthe approximate iterative differentiation (AID) method [7, 47, 42].
Due to the bilevel structure ofthe problem, even such gradient-based methods typically involve second-order matrix computations, because the gradient of Φ( x ) in eq.
(1) (which is called hypergradient ) involves the optimal solution ofthe inner function.
Although the ITD and AID methods often adopt Jacobian- or/and Hessian-vectorimplementations, the computation can still be very costly in practice for high-dimensional problemswith neural networks.
To overcome the computational challenge of current gradient-based methods, a variety of Hessian-freebilevel algorithms have been proposed.
For example, popular approaches such as FOMAML [9, 35]ignores the calculations of any second-order derivatives.
However, such a trick has no guaranteedperformance and may suffer from inferior test performance [1, 8].
In addition, when the outer-levelfunction f ( x, y ) depends only on y variable, e.g., in some hyperparameter optimization applications [12], it can be shown from eq.
(2) that the hypergradient ∇ Φ( x ) vanishes if we eliminate allsecond-order directives.
More recently, several zeroth-order methods [18] have been proposed to approximate the full hypergradient ∇ Φ( x ) .
In particular, ES-MAML [54] and HOZOG [18] use zerothorder methods to approximate the full hypergraident based on the objective function evaluations.
[Figure]
However, as demonstrated in Figure 1 (also seeour further experiments in Section 4), we empirically observe that such full hypergradientestimation can encounter a large variance and inferior performance, whereas a new zeroth-orderlike method PZOBO with partial hypergradientestimation (as we propose below) performs significantly better.
Encouraged by such an interesting observation,we propose a simple but effective Hessian-free method named PZOBO, which stands for partialzeroth-order-like bilevel optimizer.
PZOBO uses a zeroth-order-like approach to approximate only theresponse Jacobian ∂y ∗ ( x ) ∂x (which is the major computational bottleneck of hypergradient) based onthe difference between two gradient-based optimization paths.
The remaining terms in hypergradientcan simply be calculated by their analytical forms.
In this way, PZOBO avoids the large varianceof the zeroth-order method for estimating the entire hypergradient, but still enjoys its Hessian-freeadvantage.
We further show that PZOBO admits an easy extension to the large-scale stochasticsetting by simply taking small-batch gradients without introducing any complex sub-proceduresuch as the Neumann Series (NS) type of construction in [23].
Experimentally, PZOBO and itsstochastic version PZOBO-S achieve superior performance compared to the current baseline bileveloptimizers, and such an improvement is robust across various bilevel problems.
In particular, on thefew-shot meta-learning problem over a ResNet-12 network, PZOBO outperforms the state-of-the-artoptimization-based meta learners (not necessarily bilevel optimizers), while other bilevel optimizersdo not scale to solve such meta-learning problems within a comparable time frame.
Furthermore, our zeroth-order-like response Jacobian estimator in PZOBO takes the differencebetween two optimization-based trajectories, whereas the vanilla zeroth-order method uses thefunction value difference at two close points for approximation.
Such difference complicates theanalysis of PZOBO from three perspectives.
(a) Conventional zeroth-order analysis often requiressome Lipshitzness properties (e.g., smoothness) of the objective function, whereas they may nothold for the optimization-based output of our response Jacobian estimator.
(b) It is unclear if thetrajectory-based Hypergradient estimator has bounded error at each iteration, which is critical inthe convergence analysis for bilevel optimization [14, 23, 22].
(c) Such characterizations are morechallenging for stochastic settings because the randomness along the trajectory needs to be considered.
Theoretically, we provide the convergence rate guarantee for both PZOBO and PZOBO-S. To thebest of our knowledge, this is the first known non-asymptotic performance guarantee for Hessian-freebilevel algorithms via zeroth-order-like approximation.
Technically, in contrast to the conventionalanalysis of the zeroth-order estimation on the smoothed blackbox function values, we developtools to characterize the bias, variance, smoothness and boundedness of the proposed response
Jacobian estimator via a recursive analysis over the gradient-based optimization path, which can beof independent interest for zeroth-order estimation based bilevel optimization.
1.1 Related Work
Bilevel optimization.
Bilevel optimization has been studied for decades since [5].
A variety of bileveloptimization algorithms were then developed, including constraint-based approaches [21, 52, 45], approximate implicit differentiation (AID) approaches [7, 47, 16, 34, 42] and iterative differentiation(ITD) approaches [7, 44, 11, 9, 51, 49, 39].
These methods often suffer from expensive computationsof second-order information (e.g., Hessian-vector products).
Hessian-free algorithms were recentlyproposed based on interior-point method [37] and zeroth-order approximation [18, 54].
[38] proposedan initialization auxiliary method to deal with nonconvex inner problem.
However, these methodsmay suffer from either a costly procedure in regularization parameter tuning [37, 38] or a largeestimation variance [18, 54] in supervised learning.
This paper proposes a simpler and more efficient Hessian-free approach via exploiting the benign structure of the hypergradient and a zeroth-order-likeresponse Jacobian estimation.
Recently, the convergence rate has been established for gradient-based(Hessian involved) bilevel algorithms [17, 28, 49, 25].
This paper provides a new convergenceanalysis for the proposed zeroth-order-like bilevel approach.
Stochastic bilevel optimization.
[14, 28, 22] proposed stochastic gradient descent (SGD) type bileveloptimization algorithms by employing Neumann Series for Hessian-inverse-vector approximation.
Recent works [29, 30, 6, 19, 20, 57] then leveraged momentum-based variance reduction to furtherreduce the computational complexity of existing SGD-type bilevel optimizers from O ( ϵ − 2 ) to O ( ϵ − 1 . 5 ) .
However, these methods often do not scale well in large models due to the Hessian or
Hessian inverse computations. In this paper, we propose a stochastic Hessian-free method, whicheliminates the computation of all second-order information.
Bilevel optimization applications.
Bilevel optimization has been employed in various applicationssuch as few-shot meta-learning [53, 12, 49, 59, 24, 27], hyperparameter optimization [11, 43, 51],neural architecture search [35, 58], etc.
For example, [43] models the response function itself as aneural network (where each layer involves an affine transformation of hyperparameters) using the
Self-Tuning Networks (STNs).
An improved and more stable version of STNs was further proposedin [3], which focused on accurately approximating the response Jacobian rather than the responsefunction itself.
This paper demonstrates the superior performance of the proposed Hessian-freebilevel optimizer with guaranteed performance in meta-learning and hyperparameter optimization.
Zeroth-order methods and applications.
Zeroth-order optimization methods have been studied fora long time.
For example, [46] proposed an effective zeroth-order gradient estimator via Gaussiansmoothing, which was further extended to the stochastic setting by [13].
Such a technique hasexhibited great effectiveness in various applications including meta-reinforcement learning [54],hyperparameter optimization [18], adversarial machine learning [26, 40], minimax optimization [41,56], etc.
This paper proposes a novel Jacobian estimator for accelerating bilevel optimization basedon an idea similar to zeroth-order estimation via the difference between two optimization paths.
Proposed Algorithms
2.1 Hypergradients
The key step in popular gradient-based bilevel optimizers is the estimation of the hypergradient (i.e.,the gradient of the objective with respect to the outer variable x ), which takes the following form:
[Equation]
where the Jacobian matrix J ∗ ( x ) =
∂y ∗ ( x ) ∂x ∈ R d × p .
Following [42], it can be seen that∇
Φ( x ) contains two components: the direct gradient ∇ x f ( x, y ∗ ( x )) and the indirect gradient
J ∗ ( x ) ⊤ ∇ y f
( x, y ∗ ( x )) .
The direct component can be efficiently computed using the existing automatic differentiation techniques.
The indirect component, however, is computationally muchmore complex, because J ∗ ( x ) takes the form of J ∗ ( x ) = − (cid:2) ∇ 2 y g ( x, y ∗ ( x )) (cid:3) − 1 ∇ x ∇ y g ( x, y ∗ ( x ))
(if ∇ 2 y g ( x, y ∗ ( x )) is invertible), which contains the Hessian inverse and the second-order mixedderivative.
Some approaches mitigate the issue by designing Jacobian-vector and Hessian-vectorproducts [47, 11, 17] to replace second-order computations.
But the computation is still costly forhigh-dimensional bilevel problems such as those with neural network variables.
We next introducethe zeroth-order approach which is at the core for designing efficient Hessian-free bilevel optimizers.
2.2 Zeroth-Order Approximation
Zeroth-order approximation is a powerful technique to estimate the gradient of a function based onfunction values, when it is not feasible (such as in black-box problems) or computationally costly to
Set
Q 4: k − α ∇ y g
6: end do 8:
Generate u k j
∈ p 9: do 10:
Update y tk,j = y t − 1 k,j − α ∇ y g (cid:0) x k + µu k,j , y t − 1 k,j (cid:1) 11: for 12:
Compute δ j = y
Nk j − y Nk µ 13: for 14:
Compute (cid:98) ∇ Φ( x k ) =
∇ x f ( x k , y Nk ) + 1 Q (cid:80)
Qj =1 (cid:10) δ j , ∇ y f
( x k , y Nk ) (cid:11) u k,j 15:
Update x k +
=
x
k − β (cid:98) ∇ Φ( x k ) 16: end for evaluate the gradient.
The idea of the zeroth-order method in [46] is to approximate the gradient of ageneral black-box function h : R n → R using the following oracle based only on the function values
[Equation] [Equation]
where u ∈ R n is a Gaussian random vector and µ > 0 is the smoothing parameter.
Such an oraclecan be shown to be an unbiased estimator of the gradient of the smoothed function E u [ h ( x + µu )] .
2.3 Proposed Bilevel Optimizers
Our key idea is to exploit the analytical structure of the hypergradient in eq.
(2), where the derivatives∇ x f ( x, y ∗ ( x )) and ∇ y f ( x, y ∗ ( x )) can be computed efficiently and accurately, and hence use thezero-order estimator similar to eq.
(3) to estimate only the Jacobian J ∗ ( x ) , which is the major termposing computational difficulty.
In this way, our estimation of the hypergradient can be much moreaccurate and reliable.
In particular, our Jacobian estimator contains two ingredients: (i) for a given x ,apply an algorithm to solve the inner optimization problem and use the output as an approximation ofy ∗ ( x ) ; for example, the output y N ( x ) of N gradient descent steps of the inner problem can serve asan estimate for y ∗ ( x ) .
Then J N ( x ) =
∂y N ( x ) ∂x serves as an estimate of J ∗ ( x ) ; and (ii) construct thezeroth-order-like Jacobian estimator ˆ J N ( x ; u
) ∈ R
d × p for J N ( x ) as
[Equation]
where u ∈ R p is a Gaussian vector with independent and identically distributed (i.i.d.) entries.
Then for any vector v ∈ R d , the Jacobian-vector product can be efficiently computed using onlyvector-vector dot product ˆ J N ( x ; u ) ⊤ v = ⟨ δ ( x ; u ) , v ⟩ u , where δ ( x ; u ) =
y N ( x + µu ) − y N ( x ) µ ∈ R d .
PZOBO: partial zeroth-order based bilevel optimizer.
For the bilevel problem in eq.
(1), we designan optimizer (see Algorithm 1) using the Jacobian estimator in eq.
(4), which we call as the PZOBOalgorithm.
Clearly, the zeroth-order estimator is used only for estimating partial hypergradient.
Ateach step k of the algorithm, PZOBO runs an N -step full GD to approximate y Nk ( x k ) .
PZOBO thensamples Q Gaussian vectors { u k,j ∈ N (0 , I ) , j = 1 , ..., Q } , and for each sample u k,j , runs an N -stepfull GD to approximate y Nk
( x k + µu k,j ) , and then computes the Jacobian estimator ˆ J N ( x ; u k,j ) asin eq.
Then the sample average over the Q estimators is used for constructing the followinghypergradient estimator for updating the outer variable x .
[Equation]
In our experiments (see Section 4), we choose a small constant-level Q
= O
(1) (e.g., Q = 1 in mostapplications with neural nets) due to a much better performance.
Our convergence guarantee holdsfor this case, as shown later.
Computationally, in contrast to the existing AID and ITD bilevel optimizers [47, 12, 17] that containsthe complex Hessian-
and/or
Jacobian-vector product computations
, PZOBO has only gradient computations and becomes Hessian-free, and hence is much more efficient as shown in our experiments.
PZOBO-S: stochastic PZOBO.
In machine learning applications, the loss functions f, g in eq.
(1)often take finite-sum forms over given data D n,m = { ξ i , ζ j , i = 1 , ..., n, j = 1 , ..., m } as below.
[Equation]
where the sample sizes n and m are typically very large.
For such a large-scale scenario, we design astochastic PZOBO bilevel optimizer (see Algorithm 2 in Appendix A), which we call as PZOBO-S.
Differently from Algorithm 1, which applies GD updates to find y N ( x k ) , PZOBO-S uses N stochastic gradient descent (SGD) steps to find { Y Nk , Y Nk, 1 , ..., Y Nk,Q } to the inner problem, each with theouter variable set to be x k + µu k,j .
Note that all SGD runs follow the same batch sampling path{S 0 , ..., S N − 1 } .
The Jacobian estimator ˆ J N ( x k ; u k,j ) can then be computed as in eq.
At theouter level, PZOBO-S samples a new batch D F independently from the inner batches {S 0 , ..., S N − 1 }to evaluate the stochastic gradients ∇ x F ( x k , Y N k ; D F ) and ∇ y F ( x k , Y N k ; D F ) .
The hypergradient(cid:98) ∇ Φ( x k ) is then estimated as
[Equation]
Main Theroetical Results
3.1 Technical Assumptions
In this paper, we consider the following types of objective functions.
Assumption 1.
The inner function g ( x, y ) is µ g -strongly convex with respect to y and the outerfunction f ( x, y ) is possibly nonconvex w.r.t.
x .
For the finite-sum case, the same assumption holdsfor functions G ( x, y ; ζ ) and F ( x, y ; ζ )
The above assumption on f, g has also been adopted in [14, 28, 57].
In fact, many bilevel machinelearning problems satisfy this assumption.
For example, in few-shot meta-learning, the task-specificparameters are likely the weights of the last classification layer so that the resulting bilevel problemhas a strongly-convex inner problem [48, 37].
Assumption 2.
Let w
=
( x, y ) .
The gradient ∇ g ( w ) is
L g -Lipschitz continuous, i.e., for anyw 1 , w 2 , ∥∇ g ( w 1 ) − ∇ g ( w 2 ) ∥
≤ L g ∥ w 1 − w 2 ∥ ; further, the derivatives ∇ 2 y g ( w ) and ∇ x ∇ y g ( w )are ρ - and τ -Lipschitz continuous, i.e, ∥∇ 2 y g ( w 1 ) −∇
2 y g ( w 2 )
∥ F ≤ ρ ∥ w 1 − w 2 ∥ and ∥∇ x ∇ y g ( w 1 ) −∇ x ∇ y g ( w 2 ) ∥
F ≤ τ ∥ w 1 − w 2 ∥ .
The same assumptions hold for G ( w ; ζ ) in the finite-sum case.
Assumption 3.
Let w
=
( x, y ) .
The objective f ( w ) and its gradient ∇ f ( w ) are M - and L f -Lipschitzcontinuous, i.e., for any w 1 , w 2 , (cid:12) (cid:12) f ( w 1 ) − f ( w 2 ) (cid:12) (cid:12) ≤ M (cid:13) (cid:13) w 1 − w 2 (cid:13) (cid:13) , (cid:13) (cid:13) ∇ f ( w 1 ) − ∇ f ( w 2 ) (cid:13) (cid:13) ≤
L f (cid:13)(cid:13) w 1 − w 2 (cid:13)(cid:13) , which hold for F ( w ; ξ ) in the finite-sum case.
Assumption 4.
For the finite-sum case, the gradient ∇ G ( w ; ζ ) has bounded variance condition, i.e., 198 E ζ ∥∇ G ( w ; ζ ) − ∇ g ( w ) ∥ 2 ≤ σ 2 for some constant σ ≥ 0 .
3.2 Convergence Analysis for PZOBO
Differently from the standard zeroth-order analysis for a blackbox function, here we develop newtechniques to analyze the zeroth-order-like Jacobian estimator that depends on the entire inneroptimization trajectory, which is unique in bilevel optimization.
We first establish the followingimportant proposition which characterizes the Lipshitzness property of the approximate Jacobianmatrix J N ( x ) =
∂y N ( x ) ∂x .
Proposition 1.
Suppose that Assumptions 1 and 2 hold.
Let L J = (cid:16)
+ Lµ g (cid:17) (cid:16) τµ g + ρLµ 2 g (cid:17) , with
L = max { L f , L g } .
Then, the Jacobian J N ( x ) is Lipschitz continuous with constant L J :
[Equation]
[Table]
We next show that the variance of hypergradient estimation can be bounded.
The characterization ofthe estimation bias can be found in Lemma 8 in the appendix.
Proposition 2.
Suppose that Assumptions 1, 2, and 3 hold.
The hypergradient estimation variancecan be upper-bounded as
[Equation]
where E [ · ] is conditioned on x k and taken over the Gaussian vectors { u k,j : j = 1 , ..., Q } .
Proposition 2 upper bounds the hypergradient estimation variance, which mainly arises due to theestimation variance of the Jacobian matrix J ∗ by the proposed estimator J N , via three types ofvariances: (a) the approximation variance between J N and J ∗ via inner-loop gradient descent, whichdecreases exponentially w.r.t.
the number N of inner iterations due to the strong convexity of theinner objective; (b) the variance between our estimator and the Jacobian J µ of the smoothed output 217 E u y N ( x k + µu ) , which decreases sublinearly w.r.t.
the batch size Q , and (c) the variance betweenthe Jacobian J N and J µ , which can be controlled by the smoothness parameter µ .
By using the smoothness property in Proposition 1 and the upper bound in Proposition 2, we providethe following characterization of the convergence rate for PZOBO.
Theorem 1 (Convergence of PZOBO) .
Suppose that Assumptions 1, 2, and 3 hold.
Choose theinner- and outer-loop stepsizes respectively as α ≤ 1 L and β = O ( 1 √ K ) .
Further, set Q = O (1) andµ =
O (cid:16)
[Equation]
√
[Equation]
3 (cid:17) .
Then, the iterates x k for k = 0 , ..., K − 1 of PZOBO satisfy
[Equation]
Theorem 1 shows that the convergence rate of PZOBO is sublinear with respect to the number K ofouter iterations due to the nonconvexity of the outer objective, and linear (i.e., exponentially decay)with respect to the number N of inner iterations due to the strong convexity.
Theorem 1 also indicates the following features that ensures the efficiency of the algorithms.
(i)
The batch size Q of the Jacobian estimator can be chosen as a constant (in particular Q = 1 as inour experiments) so that the computation of the ES estimator is efficient.
(ii)
The number of inneriterations can be chosen to be small due to its exponential convergence, and hence the algorithmcan run efficiently.
(iii) PZOBO requires only gradient computations to converge, and eliminates
Hessian- and Jacobian-vector products required by the existing AID and ITD based bilevel optimizers([47, 12, 17].
Thus, PZOBO is more efficient particularly for high-dimensional problems.
3.3 Convergence Analysis for PZOBO-S
In this section, we apply the stochastic algorithm PZOBO-S to the finite-sum objective in eq.
(6) andanalyze its convergence rate.
The following proposition establishes an upper bound on the estimationerror of Jacobian J ∗ by J N = ∂Y N ∂x , where Y N is the output of N inner SGD updates.
Proposition 3.
Suppose that Assumptions 1, 2, and 4 hold.
Choose the inner-loop stepsize asα =
2 L + µ g , where L = max { L f , L g } .
Then, we have:
[Equation]
where λ , Γ , and C γ < 1 are constants (see Appendix J for their precise forms).
We next show that the variance of hypergradient estimation is bounded.
Proposition 4.
Suppose that Assumptions 1, 2, 3, and 4 hold.
Set the inner-loop stepsize as α = 2 L + µ gwhere L = max { L f , L g } .
Then, we have:
[Equation]
where S and D f are the sizes of the inner and outer mini-batches, respectively.
Based on Propositions 1, 3, and 4, we characterize the convergence rate for PZOBO-S.
Theorem 2 (Convergence of PZOBO-S) .
Suppose that Assumptions 1, 2, 3, and 4 hold.
Set the innerand outer-loop stepsizes respectively as α = 2 L + µ g and β =
O ( 1 √ K ) , where L = max { L f , L g } .
Further, set Q = O (1) , D f = O (1) , and µ = O (cid:16) 1 √ Kdp 3 (cid:17) .
Then, the iterates x k , k = 0 , ..., K − 1of PZOBO-S satisfy:
[Equation]
Comparing to the convergence bound in Theorem 1 for the deterministic algorithm PZOBO, Theorem 2 for the stochastic algorithm PZOBO-S captures one more sublinearly decreasing error term1 √ S due to the sampling of inner batches to estimate the objectives.
Note that the sampling of outerbatches has been included into the sublinear decay term w.r.t.
the number K of outer-loop iterations.
Experiments
We validate our algorithms in four bilevel problems: shallow hyper-representation (HR) withlinear/2-layer net embedding model on synthetic data, deep HR with LeNet network [32] on MNISTdataset, few-shot meta-learning with ResNet-12 on miniImageNet dataset, and hyperparameteroptimization (HO) on the 20 Newsgroup dataset.
We run all models using a single NVIDIA Tesla
P100 GPU.
All running time are in seconds.(a)
d =(b) d =
[Equation]
steps
[Equation]
[Figure]
4.1 Shallow Hyper-Representation on Synthetic Data
The hyper-representation (HR) problem [12, 17] searches for a regression (or classification) model 262 following a two-phased optimization process.
The inner-level identifies the optimal linear regressorparameters w , and the outer level solves for the optimal embedding model (i [Figure] e., representation) 264 parameters λ .
Mathematically, the problem can be modeled by the following bilevel optimization:
[Equation]
where X 2 ∈ R n 2 × m and X 1 ∈ R n 1 × m are matrices of synthesized training and validation data,and Y 2 ∈ R n 2 , Y 1 ∈ R n 1 are the corresponding response vectors.
In the case of shallow HR, theembedding function T ( · ; λ ) is either a linear transformation or a two-layer network.
We generatedata matrices X 1 , X 2 and labels Y 1 , Y 1 following the same process in [17].
We compare our PZOBO algorithm with the baseline bilevel optimizers AID-FP, AID-CG, ITD-R,and HOZOG (see Appendix E.1 for details about the baseline algorithms and hyperparameters used).
Figure 2 show the performance comparison among the algorithms under linear and two-layer netembedding models.
It can be observed that for both cases, our proposed method PZOBO convergesfaster than all the other approaches, and the advantage of PZOBO becomes more significant in
Figure 2 (d), which is under a higher-dimensional model of a two-layer net.
In particular, PZOBO outperforms the existing ES-based algorithm HOZOG.
This is because HOZOG uses the ES techniqueto approximate the entire hypergradient, which likely incurs a large estimation error.
In contrast, our
PZOBO exploits the structure of the hypergradient and only estimate the response Jacobian so that theestimation of hypergradient is more accurate.
Such an advantage is more evident under a two-layernet model, where HOZOG does not converge as shown in Figure 2 (d).
This can be explained by theflat hypergradient norm as shown in Figure 2 (e), which indicates that the hypergradient estimator in
HOZOG fails to provide a good descent direction for the outer optimizer.
Figure 2 (c) and (f) furthershow that the convergence of PZOBO does not change substantially with the number N of inner GDsteps, and hence tuning of N in practice is not costly.
4.2 Deep Hyper-Representation on MNIST Dataset
In order to demonstrate the advantage of our proposed algorithms in neural net models, we performdeep hyper-representation to classify MNIST images by learning an entire LeNet network.
Theproblem formulation is described in Appendix E.3.
[Figure]
Figure 3 compares the classification accuracy on the outer dataset D out between our PZOBO-S andother stochastic baseline bilevel optimizers including two AID-based stochastic algorithms AID-FPand AID-CG, and a recently proposed new stochastic bilevel optimizer stocBiO which has beendemonstrated to exhibit superior performance.
Note that ITD and HOZOG are not included in thecomparison, because there have not been stochastic algorithms proposed based on these approachesin the literature yet.
Our algorithm PZOBO-S converges with the fastest rate and attains the bestaccuracy with the lowest variance among all algorithms.
Note that PZOBO-S is the only bilevelmethod that is able to attain the same accuracy of 0 .
+ obtained by the state-of-the-art training of allparameters with one-phased optimization on the MNIST dataset using the same backbone network.
All other bilevel methods fail to recover such a level of performance, but instead saturate around anaccuracy of 0 .
Further, Figure 3(c) indicates that the convergence of PZOBO-S does not changesubstantially with the number N of inner SGD steps.
This demonstrates the robustness of our methodwhen applied to complex function geometries such as deep nets.
4.3 Few-Shot Meta-Learning over MiniImageNet
We study the few-shot image recognition problem, where classification tasks T i , i = 1 , ..., m aresampled over a distribution P T .
In particular, we consider the following commonly adopted metalearning setting (e.g., [48], where all tasks share common embedding features parameterized by ϕ ,and each task T i has its task-specific parameter w i for i
= 1 , ..., m .
More specifically, we set ϕ tobe the parameters of the convolutional part of a deep CNN model (e.g., ResNet-12 network) andw includes the parameters of the last classification layer.
All model parameters ( ϕ , w ) are trainedfollowing a bilevel procedure.
In the inner-loop, the base learner of each task T i fixes ϕ and minimizesits loss function over a training set S i to obtain its adapted parameters w ∗ i .
At the outer stage, themeta-learner computes the test loss for each task T i using the parameters ( ϕ , w ∗ i ) over a test set D i ,and optimizes the parameters ϕ of the common embedding function by minimizing the meta-objective
L meta over all classification tasks.
The detailed problem formulation is given in Appendix E.4.
We conduct few-shot meta-learning on the miniImageNet dataset [55] using two different backbonenetworks for feature extraction: ResNet-12 and CNN4 [55].
The dataset and hyperparameter detailscan be found in Appendix E.4.
We compare our algorithm PZOBO with four baseline methodsfor few-shot meta-learning MAML [9], ANIL [48], MetaOptNet [33], and ProtoNet [53].
Otherbilevel optimizers are not included into comparison because they do not solve the problem withina comparable time frame.
Zeroth-order ES-MAML [54] is not included because it exhibits largevariance and cannot reach a desired accuracy.
Also note that since ProtoNet and MetaOptNet are usually presented in the ResNet setting, and are not relevant in smaller scale networks, we includethem into comparison only for our ResNet-12 experiment.
We run their efficient Pytorch Lightningimplementations available at the learn2learn repository [2].
[Figure]
PZOBO [Figure]
[Figure] 20+ 20+
ProtNet
[Figure]
[Figure] [Figure] [Figure]
Figure 4(a) and (b) show that our algorithm PZOBO converges faster than the other baseline metalearning methods.
Also, Comparing Figure 4(a) and (b), the advantage of our method over thebaselines MAML and ANIL becomes more significant as the size of the network increases.
Further,
Figure 4(c) shows that MetaOptNet did not reach 69% accuracy after 20 hours of training with
ResNet-12 network.
In comparison, our PZOBO is able to attain 69% in less than three hours, whichis about 1.5 times less than the time taken for ProtoNet to reach the same performance level.
Both
PZOBO and ProtoNet saturate around 70% accuracy after 10 hours of training.
4.4 Hyperparameter Optimization
Hyperparameter optimization (HO) is the problem of finding the set of the best hyperparamters (eitherrepresentational or regularization parameters) that yield the optimal value of some criterion of modelquality (usually a validation loss on unseen data).
HO can be posed as a bilevel optimization problemin which the inner problem corresponds to finding the model parameters by minimizing a training loss(usually regularized) for the given hyperparameters and then the outer problem minimizes over thehyperparameters.
Due to space limitations, we provide a more complete description of the problemand settings in Appendix F.
It can be seen from Figure 5, our method PZOBO slightly outperforms HOZOG and converges fasterthan the other AID and ITD based approaches.
We note that the similar performance for PZOBOand HOZOG can be explained by the fact that in HO, the hypergradient expession in eq.
(5) containsonly the second term (the first term is zero), which is very close to the approximation in HOZOGmethod.
However, as we have seen in the HR experiments in Figure 2, PZOBO achieves a muchbetter performance than HOZOG.
Thus, compared to HOZOG, our PZOBO is much more stable andachieves superior performance across many bilevel problems.
Conclusion
In this paper, we propose a novel Hessian-free approach for bilevel optimization based on a zerothorder-like Jacobian estimator.
Compared to the existing such types of Hessian-free algorithms, ourapproach explores the analytical structure of the hypergradient, and hence leads to much more efficientand accurate hypergradient estimation.
Thus, our algorithm outperforms existing baselines in variousexperiments, particularly in high-dimensional problems.
We also provide the convergence guaranteeand characterize the convergence rate for our proposed algorithms.
We anticipate that our approachand analysis will be useful for bilevel optimization and various machine learning applications.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] Section 4.4 shows that our proposed PZOBO does not show much improvement over HOZOG in the hyperparameteroptimization experiment.
c)
Did you discuss any potential negative societal impacts of your work
[N/A This paper develops the convergence theory for the fundamental belivel algorithms
Ouranalysis will not have any potential negative societal impact
d) Have you
read
the
ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We state allassumptions in Section 3.1.
b) Did you include complete proofs of all theoretical results?
[Yes] We include complete
proofs in the appendix.
If
you ran experiments...
a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We specify the experimental details in Appendix E.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We ran 5 random seeds for every experiment.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We include the details in Section 4and
Appendix E.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We include the details
in Appendix E.
(b) Did you mention the license of the assets?
[Yes] We include the details in Appendix E.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
We include the source code in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] The datasets in our experiments are public.
See Appendix E for
more details.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
The dataset does not contain personally
identifiable information, nor offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]