Non-Convex Bilevel Games with Critical Point Selection Maps
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Bilevel optimization problems involve two nested objectives, where an upper-levelobjective depends on a solution to a lower-level problem.
When the latter is nonconvex, multiple critical points may be present, leading to an ambiguous deﬁnitionof the problem.
In this paper, we introduce a key ingredient for resolving this ambiguity through the concept of a selection map which allows one to choose a particular solution to the lower-level problem.
Using such maps, we deﬁne a class ofhierarchical games between two agents that resolve the ambiguity in bilevel problems.
This new class of games requires introducing new analytical tools in Morsetheory to characterize their evolution.
In particular, we study the differentiabilityof the selection, an essential property when analyzing gradient-based algorithmsfor solving these games.
We show that many existing algorithms for bilevel optimization, such as unrolled optimization, solve these games up to approximationerrors due to ﬁnite computational power.
Our analysis allows introducing a simplecorrection to these algorithms for removing the errors.
Introduction
Bilevel optimization has proven to be a major tool for solving machine learning problems that possess a nested structure such as hyper-parameter optimization [17], meta-learning [6], reinforcementlearning [23, 33] or dictionary learning [38].
Introduced in the ﬁeld of economic game theory in[49], a bilevel optimization problem can be understood as a game between a leader and a followereach of which optimizes their own objective function but where the leader can anticipate follower’sactions.
In the context of machine learning, the leader typically optimizes a hyper-parameter over avalidation loss while the follower optimizes the model parameter on a training loss [37].
Bilevel optimization introduces many challenges.
In particular, when multiple optimal solutionsare available to the follower, the leader would need to optimize a different objective dependingon the follower’s strategy to select an optimal solution.
As a result, the bilevel problem becomesambiguously deﬁned without knowing the follower’s strategy [35].
A large body of work on bilevelprograms for machine learning gets around these considerations by assuming the follower to havea unique optimal choice, a situation that typically occurs when the follower’s objective is stronglyconvex, leading to efﬁcient and scalable algorithms [1, 2, 7, 14, 20, 32, 33, 47].
However, in many machine learning applications, the strong convexity of the follower’s objective isan unrealistic assumption.
This is particularly the case in the context of deep learning, where thefollower’s objective, the training loss, can be highly non-convex in the parameters of the model andcan have regions of ﬂat optima due to symmetries and other degeneracies [15, 30].
In the literature on mathematical optimization, the ambiguity is often resolved by making an additional assumption on the follower’s strategy for choosing their optimal solution.
In particular, two problems are often considered: optimistic and pessimistic bilevel programs , see [13].
Both problemsrely on two assumptions: (i) the follower is using a strategy for selecting a solution to their problemthat is either improving or degrading the leader’s objective and (ii) the leader knows exactly whatstrategy the follower is using.
These assumptions are strong from a game-theoretical perspectiveand often unrealistic for machine learning problems such as hyper-parameter optimization.
Still,optimistic/pessimistic bilevel games are well deﬁned and early works have proposed several algorithms to solve them with strong convergence guarantees [55, 56, 57].
Yet, these algorithms are oftenill-suited to large-scale and high-dimensional problems arising in machine learning applications asthey rely on second-order optimization methods such as Newton’s method [21].
For this reason,scalable ﬁrst-order algorithms for such games have been proposed recently [34, 35].
However, many of the best-performing approaches for hyper-parameter optimization rely neitheron an optimistic nor a pessimistic formulation of the bilevel problem.
Instead, they often rely onalgorithms initially designed for bilevel problems with strongly convex lower objectives even thoughthe convexity assumption does not hold [37].
While these algorithms are seemingly solving an illdeﬁned bilevel program, due to the ambiguity in the way the follower selects their solution, theirability to provide models with good empirical performance raises the following question: Are thesealgorithms solving another class of well-deﬁned hierarchical problems that go beyond optimisticand pessimistic bilevel programs but are still relevant for machine learning?
In this work, we answer this question and introduce Bilevel Games with Selection (BGS), a classof well-deﬁned games between two agents: a leader and a follower, where the leader anticipates asolution from the follower but does not need to know the exact follower’s strategy.
We construct theanticipatory strategy using the notion of a selection , which is simply a map for selecting a particularsolution to the follower’s objective given the current state of the game.
In particular, BGS recoversthe usual bilevel program when the follower’s objective admits a unique solution.
By playing a BGS,the agents seek an equilibrium point for which each of their objectives ceases to vary.
Provided theselection is differentiable, the equilibrium point can be characterized by a ﬁrst-order optimalitycondition.
We leverage this characterization to show that popular algorithms based on Iterative
Differentiation (ITD) [5] admit ﬁxed points that are approximately equal to the BGS’s equilibriumpoints up to an error due to ﬁnite computational power.
We consider a general construction of the selection as the limit of a gradient ﬂow of the follower’sobjective.
We then show this selection to be differentiable near local minimizers provided the follower’s objective satisﬁes a relatively mild condition that we call “ parameteric Morse-Bott property ”.
This generalizes the more classical Morse-Bott property used in differential geometry [4, 16]to the case where the function depends on additional parameters and which allows us to considera general setting where the follower’s objective is non-convex and has possibly degenerate criticalpoints.
We then characterize the differential of the selection as a solution to a linear system, whichwe use to introduce a simple correctional term to algorithms based on (ITD) to ensure that the ﬁxedpoints obtained by these algorithms correspond exactly to the equilibrium points of a BGS problem.
Related Work
Unrolled optimization (ITD).
is a class of methods that approximates the solution map of lowerlevel objective by a differentiable function obtained through successive gradient updates [5].
Whenthe lower-level objective is strongly convex, [25] has shown these algorithms to solve a well-deﬁnedbilevel problem up to an error that can be controlled by increasing the computational budget forthe approximate solution map.
Our analysis suggests a simple algorithmic correction to these approaches that is applicable even when the lower-level objective is not strongly convex and which canresult in solutions to a bilevel game with a constant budget for the approximate solution map.
Approximate Implicit Differentiation (AID).
When the lower-level objective is strongly convex,the works of [18, 19, 42, 43] consider AID, a class of methods based on the Implicit Functiontheorem for approximating the solution map of the lower-level objective.
These algorithms admitﬁxed points that match the solutions to the bilevel problem [19, 23, 24, 25].
As such, they typicallyrequire a smaller computational budget [2, 25].
Recently, [8, 10, 9] extended AID to non-smoothobjectives while still requiring non-degenerate critical points. The present work is complementary tothe aforementioned works in that it extends AID to smooth objectives that have possibly degeneratecritical points. Moreover, we show that such generalized AID can be incorporated as an algorithmic correction to unrolled optimization algorithms.
Our analysis obtains such a correction without usingthe implicit function theorem which is not applicable in the non-convex possibly degenerate setting.
Optimistic and pessimistic bilevel optimization.
As discussed previously, when the lower-levelobjective is non-convex, the ambiguity of the problem can be resolved by optimizing the upperlevel over all possible lower-level solutions [53, 58].
The optimistic and pessimistic problems arisewhen either minimizing or maximizing the upper-level over the lower-level solutions.
In termsof algorithms, early works in [55, 56, 57] proposed to solve these problems using second-orderoptimization methods such as Newton’s method or a smoothing method [54].
These approaches arehard to scale to the high dimensional problems arising in machine learning.
Recent works consideredﬁrst-order methods based on unrolled optimization [35] and interior-point methods [34] for solvingoptimistic bilevel problems and provided approximation guarantees.
However, recent work in [50]points out that most practical applications to bilevel optimization rely on a formulation that goesbeyond optimistic or pessimistic formulations.
The present work departs from these approachesand instead introduces a bilevel game that is more tractable to solve.
We show that popular bilevelalgorithms, such as unrolled optimization, yield approximations of these games.
Non-Convex Bilevel Optimization with Selection
Notations.
Deﬁne X
= R p and Y =
R d for some positive integers p and d .
We consider two realvalued functions f
and g deﬁned on X ⇥ Y and assume that g is twice-continuously differentiable.
3.1 Background on Bilevel Optimization
A bilevel program is an optimization problem where an upper-level objective f deﬁned over a set
X ⇥ Y of variables ( x, y ) is optimized in the ﬁrst variable x under the constraint that the secondvariable y is optimal for a lower-level objective y 7!
g ( x, y ) depending on the upper-variable x .
When g ( x, . ) admits a unique minimizer denoted by y ?
( x ) , which is the case if y 7!
g ( x, y ) isstrongly convex, the bilevel problem is well-deﬁned and can be expressed as:
[Equation]
When g is non-convex, the set of minimizers T ( x ):= arg min y g ( x, y ) may contain more than oneelement making (BP) ambiguous.
A possible approach for resolving the ambiguity is to adopt agame-theoretical point of view, where a lower-level agent uses a particular strategy for selecting asolution in T ( x ) .
For instance, pessimistic bilevel games seek a solution of( UL ): min x 2 X F ( x ) , and ( LL ): F ( x [Equation]
[Equation]
where the lower agent chooses a minimizer of g ( ., y ) that is adversarial with respect to f and theupper agent minimizes the resulting worst-case loss F w.r.t. x .
Similarly, an optimistic bilevelgame can be obtained by replacing maximization with minimization such that both agents cooperate.
Unfortunately, while the game-theoretical point of view is highly relevant, many machine learningapplications do not rely on a pessimistic/optimistic bilevel formulation.
For instance, for hyperparameter optimization, the lower agent may have access to training data, but it should not haveaccess to the validation data processed (used in f ) by the upper agent.
Instead, a popular approachconsists of applying algorithms designed for bilevel programs that admit unique solutions for thelower problems, even though this assumption may not hold in practice [37].
In the next section, weintroduce a class of games that do not suffer from this issue and provide the ﬁrst characterization ofequilibrium points obtained by these previous popular approaches for non-convex problems.
3.2 Bilevel Games with Selection (BGS)
We introduce a new class of nested games for bilevel optimization with two agents, a leader optimizing a variable x in X for minimizing f , and a follower optimizing y in Y for minimizing g .
Theambiguity problem of (BP) is solved via a selection map :  : X ⇥ Y !
Y such that for all pair( x, y ) in X ⇥ Y , the map selects a critical point  ( x, y ) of g ( x, . ) .
The Bilevel Game with Selection(BGS) is then deﬁned as the following game between the leader and follower:
[Equation]
Before we give a more formal deﬁnition and concrete examples for  , we note that (BGS) solves theambiguity problem.
Typically,  ( x, y ) represents the critical point that is selected by an optimizationstrategy used by the follower ( e.g. , the limit of a gradient ﬂow for a gradient descent algorithm).
Deﬁnition 1 ( Selection map).
Given a continuously differentiable function g : X ⇥ Y !
R , themap  : X ⇥ Y !
Y is a selection if it satisﬁes the following properties for any pair ( x, y ) 2 X ⇥ Y :
Criticality: The element  ( x, y ) is a critical point of g ( x, . )
, i.e. @
y g ( x,  ( x, y ))
= 0 .
Self-consistency: If y is a critical point of g ( x, . )
i.e. @
y g ( x, y )
=0 , then  ( x, y ) =
y .
We will see in Section 4, under mild assumptions on g that do not require uniqueness of the minimizers, that it is always possible to deﬁne a selection  , for instance, as the limit of a continuous-timegradient ﬂow of y 7!
g ( x, y ) initialized at y .
The selection  ( x, y ) can be viewed as a rational guessfrom the leader about how the follower obtains their optimal actions knowing the current state of thegame ( x, y ) .
Criticality ensures the leader possesses a hierarchical advantage in that they know whatare the optimal choices accessible to the follower.
Self-consistency implies that the leader makes aguess that is not contradicting the current choice y of the follower.
Both properties ensure that aleader using such a selection can rationally anticipate the follower’s actions.
Connection to (BP).
When the lower-level objective y 7!
g ( x, y ) admits a unique minimizer 150 y ?
( x ) , it is easy to check that there exists a unique selection map  satisﬁes  ( x, y )=
y ?
( x ) .
Hence,(BGS) recovers the bilevel problem in (BP) as a particular case.
Connection to (pessimistic-BG) or the optimistic variant.
Key differences between (BGS) andpessimistic or optimistic games is that (i) the follower has never access to the upper function fwith (BGS), which matches practical hyper-parameter optimization applications where f relies on avalidation dataset, whereas g relies on a distinct training set; (ii) the leader in (pessimistic-BG) doesnot take into account the strategy used by the follower, whereas the leader in (BGS) makes morerational choices by guessing the strategy of the follower through the selection map  .
First-order equilibrium conditions.
The agents can play the game (BGS) by successively takingactions ( x k , y k ) to improve their own objectives x 7!
L  ( x, y k  1 ) and y 7!
g ( x k , y ) , by hoping the 160 strategy will reach an equilibrium pair ( x ⇤ , y ⇤ ) .
In the case where f , g and  are differentiable at( x ⇤ , y ⇤ ) , the equilibrium pair is characterized by a ﬁrst-order stationary condition:
[Equation]
When g is smooth and strongly convex in y , the implicit function theorem [28, Theorem 5.9] ensuresthat  is differentiable and provides an expression of @ x  ( x ? , y ? ) as a solution to a linear systemdepending only on second-order derivatives of g at the equilibrium ( x ? , y ? ) .
This allows to deviseefﬁcient algorithms using estimates of the gradient @
x L  , see, e.g. , [2].
However, such a theoremis not available when g is non-convex with possibly degenerate critical points without making verystrong assumptions on the objective [29], which are unrealistic in machine learning.
In the nextsection, we introduce instead a large class of functions
g that admit particular selection maps arisingfrom a gradient ﬂow.
We provide new analytical tools for studying the differentiability of such aselection w.r.t.
x at equilibrium points allowing us to obtain an expression for @ x
( x ? , y ? )
that willbe key for devising ﬁrst-order methods for solving (BGS), as discussed in Section 5.
Selection Based on Gradient Flows for Parameteric Morse-Bott Functions
We will consider a particular construction for a selection  ( x, y ) as the limit of a gradient ﬂow ofg ( x, . )
when initialized at y .
A fundamental question for (BGS) is the differentiability of  ( x, y )with respect to x , which requires to analyze the dynamics of the gradient ﬂow.
For general nonconvex functions with degenerate critical points, the selection might indeed not be differentiablesince a slight perturbation to the parameter x can drastically change the geometry of the set ofcritical points of g , causing the perturbed ﬂow to move away from the non-perturbed one.
We aretherefore interested in functions g
that preserve the local geometry near critical points as x varies.
In
Section 4.1, we introduce such a class of functions called parametric Morse-Bott functions, whichcovers many practical machine learning models.
We then show, in Section 4.2, that the selection constructed as a limit of a gradient ﬂow of such functions is differentiable near local minima.
4.1 Parameteric Morse-Bott Functions
We introduce parametric Morse-Bott functions, a class of parametric functions g : X ⇥ Y !
R withparameter x in X extending the more familiar notion of Morse-Bott functions (Appendix A.1, [16])to account for the effect of the parameter x on the geometry of critical points.
Deﬁnition 2 ( Parametric Morse-Bott function.).
Let g : X ⇥
Y be a real-valued twice continuousely differentiable function and deﬁne the set of augmented critical points M as follows:
[Equation]
Let ( x 0 , y 0 ) 2 M .
We say that g is Morse-Bott at y 0 w.r.t.
x 0 , if there exists an open neighbordhood
V of ( x 0 , y 0 )
s.t.
the intersection M \ V is a C 2 -connected sub-manifold of X ⇥ Y of dimension:
[Equation]
g is a parametric Morse-Bott function if for any ( x 0 , y 0 ) 2 M , g is Morse-Bott at y 0 w.r.t.
x 0 .
The functions in Deﬁnition 2 satisfy a condition that is stronger than simply satisfying the MorseBott property at any parameter value x
(Deﬁnition 3 of Appendix A.1).
Indeed, we show in Proposition 7 of Appendix A.2 that, for any x 0 2 X , the function y 7!
g ( x 0 , y ) is a Morse-Bott function,meaning that the critical set C ( x 0 ) of y 7!
g ( x 0 , . )
near a critical point y 0 is locally a C 2 connected sub-manifold of Y of dimension equal to the dimension of the null-space of the Hessian@ 2 yy g
x 0 , y 0 ) .
For conciseness, we introduce the following assumption which ensures g satisﬁes thecondition of Deﬁnition 2 as well as possesses continuous third-order derivatives.
Assumption 1 ( Parameteric Morse-Bott property).
The function g is at least three-times continuously differentiable and is a parameteric Morse-Bott function as deﬁned in Deﬁnition 2.
Examples of parametric Morse-Bott function.
A notable class of parametric Morse-Bott functions is the one containing all twice-continuously differentiable functions that are strongly convexor, more generally, possess only non-degenerate critical points in the second variable as shown in
Proposition 8 of Appendix A.2.
Note that parametric Morse-Bott functions need not be convex andcan have multiple (possibly degenerate) local minima, saddle-points, and local maxima.
Another class of functions, this time with possibly degenerate critical points, are those that can be expressed as a composition of some Morse-Bott function h and a family ( ⌧ x ) x 2 X of diffeomorphismson Y parameterized by x , i.e. g ( x, y )
=
h ( ⌧ x ( y )) .
This particular form is relevant in generativemodeling where the diffeomorphisms are deﬁned using normalizing ﬂows of parameter x [44].
The condition in Deﬁnition 2 ensures that the degree of freedom of the augmented critical set Mis exactly determined by the degree of freedom of the parameter x and the degree of degeneracy ofthe Hessian at a critical point y .
This condition is precisely what guarantees the stability of the localshape of critical points when the parameter x varies as we formalize through the next theorem.
Theorem 1 ( Morse-Bott lemma with parameters).
Let g be a function satisfying Assumption 1.
Let ( x 0 , y 0 ) in M be an augmented critical point of g .
Denote by K the null space of the Hessian A 0 :
= @ 2 yy g ( x 0 , y 0 ) and by K ?
its orthogonal complement in Y .
Let J 0 be a diagonal matrix with diagonal element given by the sign of the non-zero eigenvalues of A 0 .
Then, there existsopen neighborhoods U and V of ( x 0 , 0 , 0) and ( x 0 , y 0 ) in X ⇥ K ⇥ K ?
and X ⇥ Y , and a diffeomor- 219 phism : U !
V preserving the ﬁrst variable, i.e.
( x, r, w )=( x, y ) for any ( x, r, w ) 2 U , with( x 0 , 0 , 0)=( x 0 , y 0 ) such that g admits the representation:
[Equation]
Theorem 1, which is proven in Appendix A.3, shows that, near an augmented critical point ( x 0 , y 0 ) ,g looks like a quadratic function up to an additive term that depends only on the parameter x .
Moreover, slightly varying the parameter x , does not change the quadratic function and thus preserves thelocal shape near critical points.
Theorem 1 is an extension of the Morse-Bott lemma [16, Theorem2.10] to the case when there is a dependence on a parameter x .
It can also be seen as an extensionof the Morse lemma with parameters [16, Theorem 4] which allows dependence to a parameter xbut requires the critical points to be non-degenerate (invertible matrix A 0 ).
To our knowledge, Theorem 1 is the ﬁrst result in the literature providing a decomposition of parametric functions withdegenerate critical points into the sum of a quadratic non-degenerate term and a singular term depending only on the parameter x .
We present now a corollary of Theorem 1 which is a strengthenedversion of the standard Ł ojasiewicz inequality [36] that will be essential for our subsequent analysis.
Proposition 1 ( Locally Uniform Ł ojasiewicz gradient inequality).
Let g be a function satisfying
Assumption 1 and let ( x 0 , y 0 ) be in M the augmented critical set deﬁned in Deﬁnition 2.
Then,there exists an open neighborhood U of ( x 0 , y 0 ) and a positive number µ > 0 such that y 7!
g ( x, y )is constant on the set M \ U with some common value G ( x ) :
= g ( x, y ) and the following holds:
[Equation]
Proposition 1, which is proven in Appendix A.3, ensures that the Ł ojasiewicz gradient inequalityholds uniformly on ( x, y ) near any augmented critical point ( x 0 , y 0 ) .
This result will be essentialin Section 4.2 for deﬁning a selection  obtained as limits of gradient ﬂows and to obtain a locally uniform control of these ﬂows in the parameter x .
This in turn will allow us to obtain thedifferentiability of the selection in the parameter x whenever  ( x, y ) is a local minimum.
4.2 Smoothness of Selections Based on Gradient Flows of a Parametric Morse-Bott Function
We consider a construction for the selection  in Deﬁnition 1 as a limit of a continuous-time gradientﬂow of g .
More precisely, we deﬁne a continuous-time trajectory (  t ( x, y ))
t  0 in Y initialized at 0 ( x, y ) =
y and driven by the differential equation:
[Equation]
Provided  t ( x, y ) converges towards some element  ( x, y ) as t !
+ 1 , we can expect such a limit tosatisfy both conditions of Deﬁnition 1, therefore constituting a valid selection.
However, for generalnon-convex functions,  t ( x, y ) might not always converge [36].
To guarantee the existence andconvergence of the ﬂow, we make the following assumptions on the function g .
Assumption 2 ( Smoothness).
There exists L> 0 such that y 7! @
y g ( x, y ) is L -Lipschitz for any x 2 X .
Assumption 3 ( Coercivity).
For any x 2 X , if holds that g ( x, y ) !
+ 1 as k y k !
+ 1 .
The smoothness assumption in Assumption 2 is standard and guarantees the existence of the ﬂow bythe Cauchy-Lipschitz theorem.
The coercivity condition in Assumption 3 guarantees that  t ( x, y )cannot escape to inﬁnity.
It can be easily enforced by adding a small ` 2 -penalty to a non-negativeloss (such as cross-entropy or mean-squared loss) which is already a common practice in machinelearning.
These assumptions, along with Assumption 1 ensure that the limit  ( x, y ) always exists aswe summarize in the following proposition, which is proven in Appendix B.
Proposition 2.
Under Assumptions 1 to 3, and for any ( x, y ) 2 X ⇥
Y , the gradient ﬂow (GF)always converges towards a critical point  ( x, y ) of y 7!
g ( x, y ) and the map ( x, y ) 7!
( x, y ) is aselection map as deﬁned in Deﬁnition 1.
We call  the ﬂow selection relatively to g .
Proposition 2 is a consequence of a general result that holds for functions satisfying a Ł ojasiewiczgradient inequality [3, 40] which is the case here by Proposition 1.
From now on, we restrict ourattention to the selection  deﬁned in Proposition 2.
Even though  satisﬁes the implicit equation@ y g ( x,  ( x, y ))
= 0 , we cannot rely anymore on the implicit function theorem for studying thedifferentiability of  ( x, y ) in x since g can have degenerate critical points.
Instead, we propose tocharacterize the differentiability of  by studying the limit of U t ( x, y ) :
= @ x
t ( x, y ) which is drivenby a linear differential equation of the form:
[Equation]
Had we known in advance that  ( x, y ) is differentiable in x , the limit U 1 ( x, y ) of U t ( x, y ) ast !
+ 1 , whenever deﬁned, would be a promising candidate for the differential of  ( x, y ) in x .
Such a limit is indeed expected to satisfy the following linear equation:
[Equation]
A ﬁrst challenge is to ensure that U t does not diverge.
For critical points  ( x, y ) that are not localminima, it is easy to see that the Hessian @ 2 yy g ( x,  t ( x, y )) must have a negative eigenvalue fort large enough, therefore causing the system (2) to diverge.
Intuitively, unless  ( x, y ) is a localminimum, there is no reason to expect  ( x, y ) to be differentiable or even continuous in x , simply because  ( x, y ) would be an unstable ﬁxed-point of the ﬂow  t ( x, y ) , so that any change in x mightcause a large variation in  ( x, y ) .
The possible non-differentiability of  ( x, y ) for critical pointsthat are not local minima is not problematic in practice, since for almost all initial conditions
y ofthe ﬂow  t ( x, y ) , the limit  ( x, y ) is guaranteed to be a local minimizer [41].
In addition, we showin Proposition 13 of Appendix B.3 that if  ( x 0 , y ) is a local minimum, then  ( x, y ) must also be alocal minimum in a neighborhood of x 0 .
Nevertheless, even for local minima, if the Hessian @ 2 yy g ( x,  ( x, y )) is non-invertible, (3) mightnever hold if @ 2 xy g ( x,  ( x, y )) does not belong to the image of the Hessian.
However, we show in
Proposition 6 of Appendix A.2 that, for any pair ( x, y ) of critical points, @ 2 xy g ( x, y ) must alwaysbelong to the span of the Hessian @
yy g ( x, y ) as soon as g satisﬁes Assumption 1, therefore ensuringthat (3) admits a solution.
The following theorem, which is proven in Appendix C, establishes thedifferentiability of  at local minima and shows that @
x
is exactly given by the limit U 1 .
Theorem 2 ( Differentiability of the ﬂow selection).
Let g be a function satisfying Assumptions 1to 3 so that the ﬂow selection  is well-deﬁned.
Let ( x 0 , y 0 ) be in X ⇥ Y .
If  ( x 0 , y 0 ) is a localminimizer of y 7!
g ( x 0 , y ) , then there exists a neighborhood U of x 0 on which x 7!
( x, y 0 ) 289 is differentiable with differential @
x  ( x, y 0 )
=
U 1 ( x, y 0 ) .
Moreover, if y 0 is a local minimizer ofy 7!
g ( x 0 , y ) , then @ x  ( x 0 , y 0 ) is exactly given by:
[Equation]
where † is the pseudo inverse operator.
The expression in (4) is very similar to the one that wouldarise by application of the implicit function theorem to a strongly convex function g .
However,the proof technique does not rely on such a theorem which would not be applicable here.
The keytechnical challenges in proving the above result are: (i) showing that  ( x, y ) must be continuousat x 0 and (ii) controlling the error k
U t ( x, y )  U 1 ( x, y ) k locally uniformly in x .
The result followsby the application of classical uniform convergence results [46, Theorem 7.17].
The continuityof  is established in Proposition 12 of Appendix B.3 and relies on a stability analysis of the ﬂow  tperformed in Appendix B.2.
The uniform convergence of U t towards U 1 is shown in Proposition 17of Appendix C and relies on a local uniform convergence of the ﬂow  t towards  which is provenin Proposition 14 of Appendix B.4.
It is worth noting that, even though we identiﬁed @
x  to be U 1 ,the latter is not fully characterized by (3) as it might contain a non-zero component in the nullspace of the Hessian.
However, when ( x 0 , y 0 ) is an augmented critical pair of g , such a componentvanishes, and @ x  ( x 0 , y 0 ) is exactly determined by the minimal norm solution in (4).
The latter facthas practical implications when designing algorithms for solving (BGS) as we discuss next.
Algorithms
5.1 Unrolled Optimization for BGS
Unrolled optimization constructs a map ' T ( x, y ) approximating a critical point of the functiony 7!
g ( x, y ) for any ﬁxed x by applying a ﬁnite number T > 0 of gradient updates starting fromsome initial condition y .
By convention, we set '
( x, y )=
y .
Hence, ' T can be understood as anapproximation to the selection map deﬁned in Section 4.2.
We emphasize that ' T is not a selection (Deﬁnition 1) since ' T ( x, y ) is not a critical point of g in general.
Nevertheless, it providesa tractable approximation to critical points which is key for constructing practical algorithms forbilevel optimization.
The gradient of ' T ( x, y ) w.r.t.
x is then obtained by differentiating throughthe optimization steps and used to optimize the approximate upper-level objective:
[Equation]
Given the k -th upper-level iterate x k and an initial condition ˜
y k for the unrolled optimization, theseapproaches compute an approximation y k
= ' T ( x k – 1 , ˜ y k ) and ﬁnd an update direction d k for theupper-level variable x by differentiating L T ( x, ˜ y k ) in x at the current iterate x k – 1 .
The followingiterate x k is obtained by applying an update procedure, such as x k = x k – 1   d k for positive smallenough step-size  .
In Algorithm 1, we present several variants of these schemes, including a simplecorrection allowing them to solve (BGS) instead of an approximation.
The initial condition ˜
y k is often computed using a warm-start procedure ˜ y
k
= I M ( x k – 1 , y k – 1 ) .
The simplest procedure is to set ˜ y
k
= y k – 1 in which case I 0 ( x, y )=
y .
However, it is not uncommon to perform M> 0 optimization steps to minimize the objective y 7!
g ( x k – 1 , y ) starting from y k – 1 .
By doing so, gradient unrolling stops at ˜ y k and ignores the dependence of ˜ y k on y k – 1 , resulting in Truncated unrolled optimization [47].
Algorithmsummarizes these approaches when the binary variable AddCorrection is set to False .
To characterize the limit points of Algorithm 1, we make the following assumptions on I M , ' T .
1: Inputs: x 0 , y 0 , 2: Parameters: K , T , M ,  AddCorrection 3: for k 2 { 1 , ..., K + 1 } do 4: ˜
y k
I M ( x k – 1 , y k – 1 ) .
# Warm-start.
5: y k ' T ( x k – 1 , ˜ y k )
# Unrolled optimization.
6: d k @ x
L T ( x k – 1 , ˜ y k ) 7: if AddCorrection =
True then 8: v k @
y L T ( x k – 1 , ˜ y k ) 9: ⇠ k ⇡  ( @ yy g ( x k – 1 , y k )) † v k # Approx.
solver 10: d k d k + @
xy g ( x k – 1 , y k )
⇠ k # Grad.
correction 11: end if 12: x k x k – 1   d k # Updating x 13: end for 14: Return ( x K , y K ) .
Assumption 4.
For any non-negative integers M, T  0 , the maps I M and ' T are continuouson X ⇥ Y and take values in Y , with ' T being continuously differentiable.
Moreover, for any( x, y ) 2 X ⇥ Y s.t. @
y g ( x, y )
=
and M, T  0 , there exists a matrix D such that:
[Equation]
Finally, for any ( x, y ) 2 X ⇥ Y , and M, T  0 s.t. T + M > 0 , the equality y = ' T ( x, I M ( x, y ))implies that y is a critical point of g , i.e. @
y g ( x, y )
= 0 .
Assumption 5. '
T converges to a selection  and @ x ' T converges uniformly near local minima.
Assumption 4 is satisﬁed by many mappings used in practice such as T -steps of gradient descents orproximal-point algorithm, whenever g is twice-continuousely differentiable and L -smooth as shownin Proposition 19 of Appendix D. Assumption 5 is a discrete-time version of the uniform convergence result in Proposition 17 of Appendix C but that we directly assume here for simplicity.
Underthese assumptions we show that Algorithm 1 can ﬁnd equilibria of (BGS) up to an approximationerror resulting from the fact that ' T is not an exact selection.
Proposition 3.
Let M, T be non-negative numbers s.t.
M + T >
and let ( x k , y k ) be the iteratesof Algorithm 1 using the maps
I M and ' T and without any correction, i.e. AddCorrection = False .
If ( x k , y k ) converges to a limit point ( x ? T , y ? T ) then, under Assumption 4:
[Equation]
Let E be the set of limit points ( x ? T , y ? T ) be the set of all limit points for T  0 .
If E is bounded andy ?
T is a local minimum of g ( x ? T , . ) for any T  0 , then, under Assumptions 4 and 5, the elements of E are approximate equilibria for (BGS):
[Equation]
Proposition 3 shows that unrolled optimization algorithms approximately solve (BGS) in the limitwhere the number of unrolling steps T of the ' T goes to inﬁnity.
This result is consistent with theones obtained in [25] for the case where g is strongly convex and illustrates the high computationalcost for solving (BGS) without correcting for the bias introduced by unrolling.
Next, we show howto get rid of such a bias in light of Theorem 2.
5.2 Implicit Gradient Correction
We propose to correct the bias of unrolling by exploiting the expression of the gradient @
x  providedin Theorem 2.
The key idea is to obtain an expression for @
x
L  ( x, y ) in terms of L T and thesecond-order derivatives of g which holds for any local minimizer y of y 7!
g ( x, y ) as shown by theproposition below.
Proposition 4.
Let  be the selection deﬁned in Section 4.2 and ( x, y ) 2 X ⇥
Y be s.t.
y is a localminimum of y 7!
g ( x, y ) .
Then, under Assumptions 1 to 4, @ x L  ( x, y ) is given by the equation:
[Equation]
Proposition 4, which is proven in Appendix D, suggests a simple correction for the gradient estimate d k in Algorithm 1.
By doing so, the corrected algorithm would be performing an approximategradient descent on each of the upper-level and lower-level objectives, suggesting that the algorithmmay recover equilibrium points of (BGS) without having to increase the computation budget for theunrolling as we show later in Proposition 5.
A simple way to proceed would to compute c k satisfying the approximate equation c k ⇡  B k
( A k ) † v k , where A k = @
yy g ( x k – 1 , y k ) , B k := @
xy g ( x k – 1 , y k )and v k = @
y L T ( x k – 1 , ˜ y k ) .
More concretely, c k can be computed by setting c k
= B k ⇠ k where ⇠ k approximates the minimum norm solution to the least squares problem:
[Equation]
Approximate solution to (5).
It is possible to solve (5) approximately using an iterative procedureby constructing N iterates ⇠ t starting from ⇠ 0 = 0 and performing (conjugate) gradient descent onthe quadratic objective.
This can be implemented efﬁciently using only Hessian vector productswith the Hessian A k [37].
The constrained problem (5) can also be expressed as an unconstrainedone by re-parametrizing ⇠ =
A k z :
[Equation]
(6) has the advantage that z ?
k solves an unconstrained problem.
As such, it is more amenable toapplying a warm-start strategy, which can yield efﬁcient approximation z k to z ?
k by exploiting previously computed approximation z k  1 to z ?
k  1 [2].
This strategy can be achieved using a standarditerative algorithm P for approximately solving the least-squares problems, such as a ﬁxed number of conjugate gradient iterations, that takes as input the matrix
A k , vector v k and initializationz k  1 ⇡ z ?
k  1 and returns the next iterate z k ⇡ z ?
k .
More formally we view P as a continuous map of( A, v, z ) 7!
P ( A, v, z ) returning a vector z 0 and such that the only ﬁxed points are exact solutions tothe least square problem min z
[Equation]
2 .
We refer to Appendix D.1 for examples of such maps.
We can then deﬁne the iterates z k and ⇠ k as follows:
[Equation]
The corrected algorithm is obtained by setting the variable AddCorrection =
True in Algorithm 1and computing the ⇠ k using any approximate solver including, in particular, the ones based on awarm-start strategy as in (7).
The following proposition, with proof in Appendix D, shows that theproposed correction indeed yields equilibrium points of (BGS).
Proposition 5.
Let ( x k , y k ) be the iterates obtained using Algorithm 1 with AddCorrection =
Trueand T + M > 0 and assume that ⇠ k are computed using (7).
If ( x k , y k ) remain bounded, then,under Assumptions 1 to 4, any limit point ( x ? , y ? ) , is a critical point of y 7!
g ( x ? , y ) and if, inaddition, y ? is a local minimizer, and z k remains bounded, then ( x ⇤ , y ⇤ ) must be an equilibrium of(BGS) satisfying (SC):
[Equation]
Proposition 5 shows that the proposed correction allows to recover equilibria of (BGS) without having to increase the number of iterations T of the unrolled algorithm.
This is by contrast with Proposition 3 where T must increase to inﬁnity, which would be impractical.
We discuss in Appendix D.2how different choices for the parameters T and M recover known algorithms.
In particular, that Algorithm 1 with correction allows interpolating between two families of algorithms: (ITD) and(AID) while still recovering the correct equilibria.
Numerical results illustrating the beneﬁts of thecorrection are presented in Appendix E.
Discussion
We have introduced a bilevel game that resolves the ambiguity in bilevel optimization with nonconvex objectives using the notion of selection maps.
We have showed that many algorithms forbilevel optimization approximately solve these games up to a bias due to ﬁnite computational power.
Our study of the differentiability properties of the selection maps has resulted in practical proceduresfor correcting such a bias and required the development of new analytical tools.
This study opensthe way for several avenues of research to understand the tradeoff between unrolling and implicitgradient correction for designing efﬁcient algorithms.
In future work, studying these algorithms ina non-smooth and stochastic setting would also be of great theoretical and practical interest.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for informationon how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Please do not modifythe questions and only use the provided macros for your answers.
Note that the Checklist sectiondoes not count towards the page limit.
In your paper, please delete this instructions block and onlykeep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data
you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁ-
able information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]