Non-Convex Bilevel Games with Critical Point Selection Maps
Abstract
Bilevel optimization problems involve two nested objectives, where an upper-level objective depends on a solution to a lower-level problem.
When the latter is nonconvex, multiple critical points may be present, leading to an ambiguous deﬁnition of the problem.
In this paper, we introduce a key ingredient for resolving this ambiguity through the concept of a selection map which allows one to choose a particular solution to the lower-level problem.
Using such maps, we deﬁne a class of hierarchical games between two agents that resolve the ambiguity in bilevel problems.
This new class of games requires introducing new analytical tools in
Morse theory to extend implicit differentiation , a technique used in bilevel optimization resulting from the implicit function theorem.
In particular, we establish the validity of such a method even when the latter theorem is inapplicable due to degenerate critical points.
Finally, we show that algorithms for solving bilevel problems based on unrolled optimization solve these games up to approximation errors due to ﬁnite computational power.
A simple correction to these algorithms is then proposed for removing these errors.
Introduction
Bilevel optimization has proven to be a major tool for solving machine learning problems that possess a nested structure such as hyper-parameter optimization [17], meta-learning [6], reinforcement learning [23, 33] or dictionary learning [38].
Introduced in the ﬁeld of economic game theory in [49], a bilevel optimization problem can be understood as a game between a leader and a follower each of which optimizes their own objective function but where the leader can anticipate follower’s actions.
In the context of machine learning, the leader typically optimizes a hyper-parameter over a validation loss while the follower optimizes the model parameter on a training loss [37].
Bilevel optimization introduces many challenges.
In particular, when multiple optimal solutions are available to the follower, the leader would need to optimize a different objective depending on the follower’s strategy to select an optimal solution.
As a result, the bilevel problem becomes ambiguously deﬁned without knowing the follower’s strategy [35].
A large body of work on bilevel programs for machine learning gets around these considerations by assuming the follower to have a unique optimal choice, a situation that typically occurs when the follower’s objective is strongly convex, leading to efﬁcient and scalable algorithms [1, 2, 7, 14, 20, 32, 33, 47].
However, in many machine learning applications, the strong convexity of the follower’s objective is an unrealistic assumption.
This is particularly the case in the context of deep learning, where the follower’s objective, the training loss, can be highly non-convex in the parameters of the model and can have regions of ﬂat optima due to symmetries and other degeneracies [15, 30].
In the literature on mathematical optimization, the ambiguity in bilevel problems is often resolved by making an additional assumption on the follower’s strategy for choosing their optimal solution.
In particular, two problems are often considered: optimistic and pessimistic bilevel programs , see [13].
[Table]
Both problems rely on two assumptions: (i) the follower is using a strategy for selecting a solution to their problem that is either improving or degrading the leader’s objective and (ii) the leader knows exactly what strategy the follower is using.
These assumptions are strong from a game-theoretical perspective and often unrealistic for machine learning problems such as hyper-parameter optimization.
Still, optimistic/pessimistic bilevel games are well deﬁned and early works have proposed several algorithms to solve them with strong convergence guarantees [55, 56, 57].
Yet, these algorithms are often ill-suited to large-scale and high-dimensional problems arising in machine learning applications as they rely on second-order optimization methods such as Newton’s method [21].
For this reason, scalable ﬁrst-order algorithms for such games have been proposed recently [34, 35].
However, many of the best-performing approaches for hyper-parameter optimization rely neither on an optimistic nor a pessimistic formulation of the bilevel problem [50].
Instead, they often rely on algorithms initially designed for bilevel problems with strongly convex lower objectives even though the convexity assumption does not hold [37].
Consequently, these algorithms are solving a seemingly ill-deﬁned bilevel program due to the ambiguity in the way the follower selects their solution.
However, their ability to provide models with good empirical performance raises the question of whether these algorithms are solving another class of well-deﬁned hierarchical problems beyond optimistic and pessimistic bilevel programs that are still relevant for machine learning.
In this work, we answer the above question by introducing Bilevel Games with Selection (BGS), a class of games between two agents: a leader and a follower, where the leader uses a mechanism for anticipating the solution of the follower without knowing the exact follower’s strategy.
We deﬁne such a mechanism using the notion of a selection , which is simply a map for selecting a particular solution to the follower’s objective given the current state of the game.
In particular, BGS recovers a usual bilevel program when the follower’s objective admits a unique solution.
By playing a BGS, the agents seek an equilibrium point for which each of their objectives ceases to vary.
The equilibria are completely determined by the selection thus resulting in a well-deﬁned problem.
When the selection is differentiable, the equilibrium point can be characterized by a ﬁrst-order optimality condition which enables gradient-based approximations.
More precisely, we show that implicit differentiation [42], which, a priori, is only valid when the critical points of the follower’s objective are non-degenerate, remains applicable for solving BGS even when these critical points are degenerate.
To this end, we consider a general construction of the selection as the limit of a gradient ﬂow of the follower’s objective and prove the differentiability of such a selection near local minimizers, provided the follower’s objective satisﬁes a generalization of the Morse-Bott property [4, 16].
We then characterize the differential of the selection as a solution to a linear system thus extending implicit differentiation to degenerate critical points.
Finally, we leverage this characterization to show that popular algorithms based on iterative differentiation (ITD)
[5] ﬁnd ﬁxed points approximating the BGS’s equilibria up to approximation errors.
We then introduce a simple corrective term to these algorithms based on implicit differentiation to remove these errors.
Related Work
Iterative/Unrolled optimization (ITD) is a class of methods approximating the lower-level solution map by a differentiable function obtained through successive gradient updates [5].
When the lower-level objective is strongly convex, these algorithms solve a well-deﬁned bilevel problem up to an error that is controlled by increasing the computational budget for the approximate solution [25].
Our analysis suggests a simple algorithmic correction to these approaches which can result in solutions to a bilevel game with a constant budget for the approximate solution.
Approximate Implicit Differentiation (AID) is a class of methods approximating the variations of the lower-level solution map using the Implicit Function theorem [18, 19, 42, 43].
The nondegeneracy requirement under which the latter theorem holds restricts the applicability of AID to, essentially, strongly convex lower-level objectives.
These algorithms admit ﬁxed points that match the solutions to the bilevel problem [19, 23, 24, 25].
As such, they typically require a smaller computational budget than ITD [2, 25].
Recently, [8, 10, 9] extended AID to non-smooth objectives while still requiring non-degenerate critical points.
The present work is complementary to these works as it extends AID to smooth objectives that have possibly degenerate critical points.
Optimistic and pessimistic bilevel optimization.
When the lower-level objective is non-convex, the ambiguity of the problem arising from the multiplicity of the lower-level solutions can be re- solved by optimizing the upper-level objective over all such possible solutions [53, 58].
The optimistic and pessimistic problems arise when either minimizing or maximizing the upper-level over all such lower-level solutions.
Early works proposed to solve these problems using exact penalization [57], second-order optimization [55, 56] or smoothing method [54].
However, these approaches are hard to scale to the high dimensional problems arising in machine learning.
More recently, [35, 34] considered ﬁrst-order methods based on unrolled optimization or interior-point methods for solving optimistic bilevel problems and provided approximation guarantees.
However, as shown in [50], most practical applications to bilevel optimization rely on a formulation that goes beyond optimistic or pessimistic formulations.
The present work departs from these approaches and instead introduces a bilevel game that is more tractable to solve.
We show that popular bilevel algorithms, such as unrolled optimization, yield approximations of these games.
Non-Convex Bilevel Optimization with Selection
Notations.
Deﬁne X
= R p and Y =
R d for some positive integers p and d .
We consider two real valued functions f
and g deﬁned on X ⇥ Y and assume g to be twice-continuously differentiable.
3.1 Background on Bilevel Optimization
A bilevel program is an optimization problem where an upper-level objective f deﬁned over a set
X ⇥ Y of variables ( x, y ) is optimized in the ﬁrst variable x under the constraint that the second variable y is optimal for a lower-level objective y 7!
g ( x, y ) depending on the upper-variable x .
When g ( x, . ) admits a unique minimizer denoted by y ?
( x ) , which is the case if y 7!
g ( x, y ) is strongly convex, the bilevel problem is well-deﬁned and can be expressed as:
[Equation]
When g is non-convex, the set of minimizers T ( x ):= arg min y g ( x, y ) may contain more than one element making (BP) ambiguous.
A possible approach for resolving the ambiguity is to adopt a game-theoretical point of view, where a lower-level agent uses a particular strategy for selecting a solution in T ( x ) .
For instance, in pessimistic bilevel games, the lower agent chooses a minimizer of g ( x, . )
that maximizes f ( x, . ) while the upper agent minimizes the resulting worst-case loss F in x : ( UL ): min x 2 X
F ( x ) , and ( LL ):
[Equation]
Similarly, an optimistic bilevel game can be obtained by replacing maximization with minimization so that both agents cooperate.
While these approaches are highly relevant from a game-theoretical point of view, many machine learning applications do not rely on a pessimistic/optimistic bilevel formulation.
For instance, for hyper-parameter optimization, the lower agent may have access to training data, but it should not have access to the validation data processed (used in f ) by the upper agent.
Instead, a popular approach consists of applying algorithms designed for bilevel programs that admit unique solutions for the lower problems, even though this assumption may not hold in practice [37].
In the next section, we introduce a class of games that allow characterizing the equilibrium points obtained by these popular algorithms while resolving the ambiguity of non-convex bilevel problems and bypassing the limitations of pessimistic/optimistic bilevel formulations.
3.2 Bilevel Games with Selection (BGS)
We introduce a new class of nested games for bilevel optimization with two agents, a leader and a follower .
The follower minimizes the lower-level objective g w.r.t.
a variable y in Y .
Similarly, the leader minimizes the upper-level objective f w.r.t.
a variable x 2 X while anticipating the follower ’s solution.
More precisely, the leader has access to a selection map :  : X ⇥ Y !
Y to choose a unique critical point  ( x, y ) of y 7!
g ( x, y ) given the current state of the game ( x, y ) 2 X ⇥ Y thus allowing the leader to anticipate the follower’s solution.
Typically, the selection  ( x, y ) represents the critical point that is selected by an optimization process of g ( x, . )
starting from an initial condition y
( e.g. , the limit of a gradient ﬂow for a gradient descent algorithm).
The Bilevel Game with Selection (BGS) is therefore deﬁned as the following interdependent optimization problems:
[Equation]
Given a selection map  , the game (BGS) is well-deﬁned and does not suffer from the ambiguity problem in (BP).
The explicit dependence of  ( x, y ) on the initialization y might seem unnecessary at ﬁrst, as one could simply ﬁx y to some value y 0 and consider only the dependence on the variable x .
However, such a dependence on the variable y allows performing warm-start [50], where the lower-level problem is optimized starting from a previous state of the game, thus resulting in computational savings Figure 1.
We provide below a formal deﬁnition for the selection map.
Deﬁnition 1 ( Selection map).
Given a continuously differentiable function g : X ⇥ Y !
R , the map  : X ⇥ Y !
Y is a selection if it satisﬁes the following properties for any pair ( x, y ) 2 X ⇥ Y :
Criticality: The element y
=  ( x, y ) is a critical point of g ( x, . ) , i.e. @
y g ( x, y 0 )
= 0 .
Self-consistency: If y is a critical point of g ( x, . )
i.e. @
y g ( x, y )
= 0 , then  ( x, y ) =
y .
Criticality ensures the leader possesses a hierarchical advantage
in
that they know what are the optimal choices accessible to the follower.
Self-consistency implies that the leader makes a guess that is not contradicting the current choice y of the follower.
Both properties ensure the leader can rationally anticipate the follower’s actions from the current state of the game ( x, y ) .
We will see in
Section 4, under mild assumptions on g , that it is always possible to deﬁne a selection  as the limit of a continuous-time gradient ﬂow of y 7!
g ( x, y ) initialized at y .
Moreover, as we discuss later in
Section 5, the selection does not need to be explicitly constructed for solving (BGS) in practice.
It can be simply related to the implicit bias of the algorithm used for solving the follower’s problem.
Connection to (BP).
When the lower-level objective y 7!
g ( x, y ) admits a unique minimizer y ?
( x ) , it is easy to check that there exists a unique selection map  satisﬁes  ( x, y )=
y ?
( x ) .
Hence, (BGS) recovers the bilevel problem in (BP) as a particular case.
Connection to (pessimistic-BG) or the optimistic variant.
Key differences between (BGS) and pessimistic or optimistic games is that (i) the follower has never access to the upper function f with (BGS), which matches practical hyper-parameter optimization applications where f relies on a validation dataset, whereas g relies on a distinct training set; (ii) the leader in (pessimistic-BG) does not take into account the strategy used by the follower, whereas the leader in (BGS) makes more rational choices by guessing the strategy of the follower through the selection map  .
First-order equilibrium conditions.
The agents can play the game (BGS) by successively taking actions ( x k , y k ) to improve their own objectives x 7!
L  ( x, y k  1 ) and y 7!
g ( x k , y ) , by hoping the strategy will reach an equilibrium pair ( x ⇤ , y ⇤ )
Figure 1(Right).
In the case where f , g and  are differentiable at ( x ⇤ , y ⇤ ) , the equilibrium pair is characterized by a ﬁrst-order stationary condition:
[Equation]
When g is smooth and strongly convex in y , the implicit function theorem [28, Theorem 5.9] ensures that  is differentiable and provides an expression of @ x  ( x ? , y ? )
as a solution to a linear system which key for implicit differentiation.
This allows to devise efﬁcient algorithms using estimates of the gradient @
x L  , see, e.g. , [2].
However, extensions of the implicit function theorem, such as the constant rank theorem [29, Theorem 4.12], for cases where g has possibly degenerate critical points require strong assumptions on g which are unrealistic in machine learning.
In the next section, we provide new analytical tools for extending implicit differentiation by studying the differentiability of a family of selection maps corresponding to a large class of functions g .
The resulting expression will be key for devising ﬁrst-order methods to solve (BGS), as discussed in Section 5.
Selection Based on Gradient Flows for Parameteric Morse-Bott Functions
In this section, we extend implicit differentiation to a class of functions with possibly degenerate critical points.
To this end, we consider a particular selection  ( x, y ) obtained as the limit of a gradient ﬂow (  t ( x, y ))
t  0 of g ( x, . ) initialized at y .
We then study the differentiability w.r.t.
x of the selection by analyzing the dynamics of such a gradient ﬂow.
For general non-convex functions, the selection might be non-differentiable since a small perturbation to the parameter x can change the geometry of the critical points of g , causing the perturbed ﬂow to move away from the non-perturbed one (see Figure 2).
We are therefore interested in functions g preserving the local geometry near critical points as x varies.
In Section 4.1, we introduce such a class of functions called parametric
Morse-Bott functions, which covers many practical machine learning models.
We then show, in
Section 4.2, that the selection resulting from such a function is differentiable near local minima.
4.1 Parameteric Morse-Bott Functions
We introduce parametric Morse-Bott functions, a class of parametric functions g : X ⇥ Y !
R with parameter x in X extending the more familiar notion of Morse-Bott functions (Appendix A.1, [16]) to account for the effect of the parameter x on the geometry of critical points.
Deﬁnition 2 ( Parametric Morse-Bott function.).
Let g : X ⇥
Y be a real-valued twice continuousely differentiable function and deﬁne the set of augmented critical points M as follows:
[Equation]
Let ( x 0 , y 0 ) 2 M .
We say that g is Morse-Bott at y 0 w.r.t.
x 0 , if there exists an open neighbordhood
V of ( x 0 , y 0 )
s.t.
the intersection M \ V is a C 2 -connected sub-manifold of X ⇥ Y of dimension:
[Equation]
[Equation] g is a parametric Morse-Bott function if for any ( x 0 , y 0 ) 2 M , g is Morse-Bott at y 0 w.r.t.
x 0 .
The functions in Deﬁnition 2 satisfy a condition that is stronger than simply satisfying the MorseBott property at any parameter value x
(Deﬁnition 3 of Appendix A.1).
Indeed, we show in Proposition 7 of Appendix A.2 that, for any x 0 2 X , the function y 7!
g ( x 0 , y ) is a Morse-Bott function, meaning that the critical set C ( x 0 ) of y 7!
g ( x 0 , . )
near a critical point y 0 is locally a C 2 connected sub-manifold of Y of dimension equal to the dimension of the null-space of the Hessian @
yy g ( x 0 , y 0 ) .
For conciseness, we introduce the following assumption which ensures g satisﬁes the condition of Deﬁnition 2 as well as possesses continuous third-order derivatives.
Assumption 1 ( Parameteric Morse-Bott property).
The function g is at least three-times continuously differentiable and is a parameteric Morse-Bott function as deﬁned in Deﬁnition 2.
Examples of parametric Morse-Bott function.
A notable class of parametric Morse-Bott functions is the one containing all twice-continuously differentiable functions that are strongly convex or, more generally, possess only non-degenerate critical points in the second variable as shown in
Proposition 8 of Appendix A.2.
Note that parametric Morse-Bott functions need not be convex and can have multiple (possibly degenerate) local minima, saddle-points, and local maxima.
Another class of functions, this time with possibly degenerate critical points, are those that can be expressed as a composition of some Morse-Bott function h and a family ( ⌧ x ) x 2 X of diffeomorphisms on Y parameterized by x , i.e. g ( x, y )
=
h ( ⌧ x ( y )) .
This particular form is relevant in generative modeling where the diffeomorphisms are deﬁned using normalizing ﬂows of parameter x [44].
The condition in Deﬁnition 2 ensures that the degree of freedom of the augmented critical set M is exactly determined by the degree of freedom of the parameter x and the degree of degeneracy of the Hessian at a critical point y .
This condition is precisely what guarantees the stability of the local shape of critical points when the parameter x varies as we formalize through the next theorem.
Theorem 1 ( Morse-Bott lemma with parameters).
Let g be a function satisfying Assumption 1.
Let ( x 0 , y 0 ) in M be an augmented critical point of g .
Denote by K the null space of the Hessian A 0 :
= @ 2 yy g ( x 0 , y 0 ) and by K ?
its orthogonal complement in Y .
Let J 0 be a diagonal matrix with diagonal element given by the sign of the non-zero eigenvalues of A 0 .
Then, there exists open neighborhoods U and V of ( x 0 , 0 K , 0 K ? ) and ( x 0 , y 0 ) in X ⇥ K ⇥ K ?
and X ⇥ Y , and a diffeomor- phism : U !
V preserving the ﬁrst variable, i.e.
( x, r, w )=( x, y ) for any ( x, r, w ) 2 U , with ( x 0 , 0 K , 0 K ? )
=( x 0 , y 0 ) such that g admits the representation:
[Equation]
Theorem 1, which is proven in Appendix A.3, shows that, near an augmented critical point ( x 0 , y 0 ) , g looks like a quadratic function up to an additive term that depends only on the parameter x .
Moreover, slightly varying the parameter x does not change the quadratic function and thus preserves the local shape near critical points.
Theorem 1 is an extension of the Morse-Bott lemma [16, Theorem 2.10] to the case when there is a dependence on a parameter x .
It can also be seen as an extension of the Morse lemma with parameters [16, Theorem 4] which allows dependence to a parameter x but requires the critical points to be non-degenerate (invertible matrix A 0 ).
To our knowledge, Theorem 1 is the ﬁrst result in the literature providing a decomposition of parametric functions with degenerate critical points into the sum of a quadratic non-degenerate term and a singular term depending only on the parameter x .
We present now a corollary of Theorem 1 which is a strengthened version of the standard Ł ojasiewicz inequality [36] that will be essential for our subsequent analysis.
Proposition 1 ( Locally Uniform Ł ojasiewicz gradient inequality).
Let g be a function satisfying
Assumption 1 and let ( x 0 , y 0 ) be in M the augmented critical set deﬁned in Deﬁnition 2.
Then, there exists an open neighborhood U of ( x 0 , y 0 ) and a positive number µ > 0 such that y 7!
g ( x, y ) is constant on the set M \ U with some common value G ( x ) :
= g ( x, y ) and the following holds:
[Equation]
Proposition 1, which is proven in Appendix A.3, ensures that the Ł ojasiewicz gradient inequality holds uniformly on ( x, y ) near any augmented critical point ( x 0 , y 0 ) .
This result will be essential in Section 4.2 for deﬁning a selection  obtained as limits of gradient ﬂows and to obtain a locally uniform control of these ﬂows in the parameter x .
This in turn will allow us to obtain the differentiability of the selection in the parameter x whenever  ( x, y ) is a local minimum.
4.2 Smoothness of Selections Based on Gradient Flows of a Parametric Morse-Bott Function
We consider a construction for the selection  in Deﬁnition 1 as a limit of a continuous-time gradient ﬂow of g .
More precisely, we deﬁne a continuous-time trajectory (  t ( x, y ))
t  0 in Y initialized at
0 ( x, y )
=
y and driven by the differential equation:
[Equation]
Provided  t ( x, y ) converges towards some element  ( x, y ) as t !
+ 1 , we can expect such a limit to satisfy both conditions of Deﬁnition 1, therefore constituting a valid selection.
However, for general non-convex functions,  t ( x, y ) might not always converge [36].
To guarantee the existence and convergence of the ﬂow, we make the following assumptions on the function g .
Assumption 2 ( Smoothness).
There exists L> 0 such that y 7! @
y g ( x, y ) is L -Lipschitz for any x 2 X .
Assumption 3 ( Coercivity).
For any x 2 X , it holds that g ( x, y ) !
+ 1 as k y k !
+ 1 .
The smoothness assumption in Assumption 2 is standard and guarantees the existence of the ﬂow by the Cauchy-Lipschitz theorem.
The coercivity condition in Assumption 3 guarantees that  t ( x, y ) cannot escape to inﬁnity.
It can be easily enforced by adding a small ` 2 -penalty to a non-negative loss (such as cross-entropy or mean-squared loss) which is already a common practice in machine learning.
These assumptions, along with Assumption 1 ensure that the limit  ( x, y ) always exists as we summarize in the following proposition, which is proven in Appendix B.
Proposition 2.
Under Assumptions 1 to 3, and for any ( x, y ) 2 X ⇥
Y , the gradient ﬂow (GF) always converges towards a critical point  ( x, y ) of y 7!
g ( x, y ) and the map ( x, y ) 7!
( x, y ) is a selection map as deﬁned in Deﬁnition 1.
We call  the ﬂow selection relatively to g .
Proposition 2 is a consequence of a general result that holds for functions satisfying a Ł ojasiewicz gradient inequality [3, 40] which is the case here by Proposition 1.
From now on, we restrict our attention to the selection  deﬁned in Proposition 2.
Even though  satisﬁes the implicit equation @
y g ( x,  ( x, y ))
= 0 , we cannot rely anymore on the implicit function theorem for studying the differentiability of  ( x, y ) in x since g can have degenerate critical points.
Instead, we propose to characterize the differentiability of  by studying the limit of U t ( x, y ) :
= @ x
t ( x, y ) which is formally driven by a linear differential equation of the form:
[Equation]
Had we known in advance that  ( x, y ) is differentiable in x , the limit U 1 ( x, y ) of U t ( x, y ) as t !
+ 1 , whenever deﬁned, would be a promising candidate for the differential of  ( x, y ) in x .
Such a limit is indeed expected to satisfy the following linear equation:
[Equation]
A ﬁrst challenge is to ensure that U t does not diverge.
For critical points  ( x, y ) that are not local minima, it is easy to see that the Hessian @ 2 yy g ( x,  t ( x, y )) must have a negative eigenvalue for t large enough, therefore causing the system (2) to diverge.
Intuitively, unless  ( x, y ) is a local minimum, there is no reason to expect  ( x, y ) to be differentiable or even continuous in x , simply because  ( x, y ) would be an unstable ﬁxed-point of the ﬂow  t ( x, y ) , so that any change in x might cause a large variation in  ( x, y ) .
The possible non-differentiability of  ( x, y ) for critical points that are not local minima is not problematic in practice, since for almost all initial conditions y of the ﬂow  t ( x, y ) , the limit  ( x, y ) is guaranteed to be a local minimizer [41].
In addition, we show in Proposition 13 of Appendix B.3 that if  ( x 0 , y ) is a local minimum, then  ( x, y ) must also be a local minimum in a neighborhood of x 0 .
Nevertheless, even for local minima, if the Hessian @ 2 yy g ( x,  ( x, y )) is non-invertible, (3) might never hold if @ 2 xy g ( x,  ( x, y )) does not belong to the image of the Hessian.
However, we show in
Proposition 6 of Appendix A.2 that, for any pair ( x, y ) of critical points, @ 2 xy g ( x, y ) must always belong to the span of the Hessian @
yy g ( x, y ) as soon as g satisﬁes Assumption 1, therefore ensuring that (3) admits a solution.
The following theorem, which is proven in Appendix C, establishes the differentiability of  at local minima and shows that @
x
is exactly given by the limit U 1 .
Theorem 2 ( Degenerate implicit differentiation.).
Let g be a function satisfying Assumptionsto 3 so that the ﬂow selection  is well-deﬁned.
Let ( x 0 , y 0 ) be in X ⇥ Y .
If  ( x 0 , y 0 ) is a local minimizer of y 7!
g ( x 0 , y ) , then there exists a neighborhood U of x 0 on which x 7!
( x, y 0 ) is differentiable with differential @
x  ( x, y 0 )
=
U 1 ( x, y 0 ) .
Moreover, if y 0 is a local minimizer of y 7!
g ( x 0 , y ) , then, denoting by † the pseudo inverse operator, @ x
( x 0 , y 0 ) is exactly given by:
[Equation]
The expression in (4) is very similar to the one that would arise by application of the implicit function theorem to a strongly convex function g .
However, the proof technique does not rely on such a theorem which would not be applicable here.
The key technical challenges in proving the above result are: (i) showing that  ( x, y ) must be continuous at x 0 and (ii) controlling the error k
U t ( x, y )  U 1 ( x, y ) k locally uniformly in x .
The result follows by the application of classical uniform convergence results [46, Theorem 7.17].
The continuity of  is established in Proposition 12 of
Appendix B.3 and relies on a stability analysis of the ﬂow  t performed in Appendix B.2.
The uniform convergence of U t towards U 1 is shown in Proposition 17 of Appendix C and relies on a local uniform convergence of the ﬂow  t towards  which is proven in Proposition 14 of Appendix B.4.
It is worth noting that, even though we identiﬁed @
x  to be U 1 , the latter is not fully characterized by (3) as it might contain a non-zero component in the null-space of the Hessian.
However, when ( x 0 , y 0 ) is an augmented critical pair of g , such a component vanishes, and @ x  ( x 0 , y 0 ) is exactly determined by the minimal norm solution in (4).
The latter fact has practical implications when designing algorithms for solving (BGS) as we discuss next.
Algorithms 5.1 Unrolled Optimization for BGS
Unrolled optimization constructs a map ' T ( x, y ) approximating a critical point of the function y 7!
g ( x, y ) for any ﬁxed x by applying a ﬁnite number T > 0 of gradient updates starting from some initial condition y .
By convention, we set '
( x, y )=
y .
Hence, ' T can be understood as an approximation to the selection map deﬁned in Section 4.2.
We emphasize that ' T is not a selection (Deﬁnition 1) since ' T ( x, y ) is not a critical point of g in general.
Nevertheless, it provides a tractable approximation to critical points which is key for constructing practical algorithms for bilevel optimization.
The gradient of ' T ( x, y ) w.r.t.
x is then obtained by differentiating through the optimization steps and used to optimize the approximate upper-level objective:
[Equation]
Given the k -th upper-level iterate x k and an initial condition ˜
y k for the unrolled optimization, these approaches compute an approximation y k
= ' T ( x k – 1 , ˜ y k ) and ﬁnd an update direction d k for the upper-level variable x by differentiating L T ( x, ˜ y k ) in x at the current iterate x k – 1 .
The following iterate x k is obtained by applying an update procedure, such as x k = x k – 1   d k for positive small enough step-size  .
In Algorithm 1, we present several variants of these schemes, including a simple correction allowing them to solve (BGS) instead of an approximation.
The initial condition ˜
y k is often computed using a warm-start procedure ˜ y
k
= I M ( x k – 1 , y k – 1 ) .
The simplest procedure is to set ˜ y
k
= y k – 1 in which case I 0 ( x, y )=
y .
However, it is not uncommon to perform M> 0 optimization steps to minimize the objective y 7!
g ( x k – 1 , y ) starting from y k – 1 .
By doing so, gradient unrolling stops at ˜ y k and ignores the dependence of ˜ y k on y k – 1 , resulting in Truncated unrolled optimization [47].
Algorithmsummarizes these approaches when the binary variable AddCorrection is set to False .
To characterize the limit points of Algorithm 1, we make the following assumptions on I M , ' T .
Algorithm 1: Inputs: x 0 , y 0 , 2: Parameters: K , T , M ,  AddCorrection 3: for k 2 { 1 , ..., K + 1 } do 4: ˜
y k
I M ( x k – 1 , y k – 1 ) .
# Warm-start.
5: y k ' T ( x k – 1 , ˜ y k )
# Unrolled optimization.
6: d k @ x
L T ( x k – 1 , ˜ y k ) 7: if AddCorrection =
True then 8: v k @
y L T ( x k – 1 , ˜ y k ) 9: ⇠ k ⇡  ( @ yy g ( x k – 1 , y k )) † v k # Approx.
solver 10: d k d k + @
xy g ( x k – 1 , y k )
⇠ k # Grad.
correction 11: end if 12: x k x [Equation] – 1   d k # Updating x 13: end for 14: Return ( x K , y K ) .
Assumption 4.
For any non-negative integers M, T  0 , the maps I M and ' T are continuous on X ⇥ Y and take values in Y , with ' T being continuously differentiable.
Moreover, for any ( x, y ) 2 X ⇥ Y s.t. @
y g ( x, y )
=
and M, T  0 , there exists a matrix D such that:
[Equation]
Finally, for any ( x, y ) 2 X ⇥ Y , and M, T  0 s.t. T + M > 0 , the equality y = ' T ( x, I M ( x, y )) implies that y is a critical point of g , i.e. @
y g ( x, y )
= 0 .
Assumption 5. '
T converges to a selection  and @ x ' T converges uniformly near local minima.
Assumption 4 is satisﬁed by many mappings used in practice such as T -steps of the gradient descent or proximal point algorithms, whenever g is twice-continuousely differentiable and L -smooth as shown in Proposition 19 of Appendix D. Assumption 5 is a discrete-time version of the uniform convergence result in Proposition 17 of Appendix C but that we directly assume here for simplicity.
Under these assumptions we show that Algorithm 1 can ﬁnd equilibria of (BGS) up to an approximation error resulting from the fact that ' T is not an exact selection.
Proposition 3.
Let M, T be non-negative numbers s.t.
M + T >
and let ( x k , y k ) be the iterates of Algorithm 1 using the maps
I M and ' T and without any correction, i.e. AddCorrection = False .
If ( x k , y k ) converges to a limit point ( x ? T , y ?
T ) then, under Assumption 4:
[Equation]
Let E be the set of limit points ( x ? T , y ?
T ) for T  0 .
If E is bounded and y ?
T is a local minimum of g ( x ?
T , . )
for any T  0 , then, under Assumptions 4 and 5, the elements of E are approximate equilibria for (BGS):
[Equation]
Proposition 3 shows that unrolled optimization algorithms approximately solve (BGS) in the limit where the number of unrolling steps T of the ' T goes to inﬁnity.
This result is consistent with the ones obtained in [25] for the case where g is strongly convex and illustrates the high computational cost for solving (BGS) without correcting for the bias introduced by unrolling.
Next, we show how to get rid of such a bias in light of Theorem 2.
5.2 Implicit Gradient Correction
We propose to correct the bias of unrolling by exploiting the expression of the gradient @
x  provided in Theorem 2.
The key idea is to obtain an expression for @
x
L  ( x, y ) in terms of L T and the second-order derivatives of g which holds for any local minimizer y of y 7!
g ( x, y ) as shown by the proposition below.
Proposition 4.
Let  be the selection deﬁned in Section 4.2 and ( x, y ) 2 X ⇥
Y be s.t.
y is a local minimum of y 7!
g ( x, y ) .
Then, under Assumptions 1 to 4, @ x L  ( x, y ) is given by the equation:
[Equation]
Proposition 4, which is proven in Appendix D, suggests a simple correction for the gradient estimate d k in Algorithm 1.
By doing so, the corrected algorithm would be performing an approximate gradient descent on each of the upper-level and lower-level objectives, suggesting that the algorithm may recover equilibrium points of (BGS) without having to increase the computation budget for the unrolling as we show later in Proposition 5.
A simple way to proceed would to compute c k satisfying the approximate equation c k ⇡  B k
( A k ) † v k , where A k = @
yy g ( x k – 1 , y k ) , B k := @
xy g ( x k – 1 , y k ) and v k = @
y L T ( x k – 1 , ˜ y k ) .
More concretely, c k can be computed by setting c k
= B k ⇠ k where ⇠ k approximates the minimum norm solution to the least squares problem:
[Equation]
Approximate solution to (5).
It is possible to solve (5) approximately using an iterative procedure by constructing N iterates ⇠ t starting from ⇠ 0 = 0 and performing (conjugate) gradient descent on the quadratic objective.
This can be implemented efﬁciently using only Hessian vector products with the Hessian A k [37].
The constrained problem (5) can also be expressed as an unconstrained one by re-parametrizing ⇠ =
A k z :
[Equation]
Eq.
(6) has the advantage that z ?
k solves an unconstrained problem.
As such, it is more amenable to applying a warm-start strategy, which can yield efﬁcient approximation z k to z ?
k by exploiting previously computed approximation z k  1 to z ?
k  1 [2].
This strategy can be achieved using a standard iterative algorithm P for approximately solving the least-squares problems, such as a ﬁxed number of conjugate gradient iterations, that takes as input the matrix
A k , vector v k and initialization z k  1 ⇡ z ?
k  1 and returns the next iterate z k ⇡ z ?
k .
More formally we view P as a continuous map of ( A, v, z ) 7!
P ( A, v, z ) returning a vector z 0 and such that the only ﬁxed points are exact solutions to the least square problem min z
A 2 z + [Equation]  2 .
We refer to Appendix D.1 for examples of such maps.
We can then deﬁne the iterates z k and ⇠ k as follows:
[Equation]
The corrected algorithm is obtained by setting the variable AddCorrection =
True in Algorithmand computing the ⇠ k using any approximate solver including, in particular, the ones based on a warm-start strategy as in (7).
The following proposition, with proof in Appendix D, shows that the proposed correction indeed yields equilibrium points of (BGS).
Proposition 5.
Let ( x k , y k ) be the iterates obtained using Algorithm 1 with AddCorrection =
True and T + M > 0 and assume that ⇠ k are computed using (7).
If ( x k , y k , z k ) k
0 converges to a limit point ( x ? , y ? , z ? )
, then y ? is a critical point of y 7!
g ( x ? , y ) and if, in addition, y ? is a local minimizer, then ( x ⇤ , y ⇤ ) must be an equilibrium of (BGS) satisfying (SC):
[Equation]
Proposition 5 shows that the proposed correction allows to recover equilibria of (BGS) without having to increase the number of iterations T of the unrolled algorithm.
This is by contrast with Proposition 3 where T must increase to inﬁnity, which would be impractical.
We discuss in Appendix D. how different choices for the parameters T and M recover known algorithms.
In particular, that
Algorithm 1 with correction allows interpolating between two families of algorithms: (ITD) and (AID) while still recovering the correct equilibria.
Numerical results illustrating the beneﬁts of the correction are presented in Appendix E.
Discussion
We have introduced a bilevel game that resolves the ambiguity in bilevel optimization with nonconvex objectives using the notion of selection maps.
We have shown that many algorithms for bilevel optimization approximately solve these games up to a bias due to ﬁnite computational power.
Our study of the differentiability properties of the selection maps has resulted in practical procedures for correcting such a bias and required the development of new analytical tools.
This study opens the way for several avenues of research to understand the tradeoff between unrolling and implicit gradient correction for designing efﬁcient algorithms.
In future work, studying these algorithms in a non-smooth and stochastic setting would also be of great theoretical and practical interest.
Funding This project was supported by ANR 3IA MIAI@Grenoble Alpes (ANR-19-P3IA-0003).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]