BiGrad: Differentiating through Bilevel Optimization Programming
Abstract
Integrating mathematical programming, and in particular Bilevel Optimization Programming, within deep learning architectures has vast applications in various domains from ma- chine learning to engineering.
Bilevel programming is able to capture complex interactions when two actors have con- flicting objectives.
Previous approaches only consider singlelevel programming.
In this paper, we thus propose Differenti- ating through Bilevel Optimization Programming (BiGrad) as approach for end-to-end learning of models that use Bilevel Programming as a layer.
BiGrad has wide applicability and it can be used in modern machine learning frameworks.
We focus on two classes of Bilevel Programming: continuous and combinatorial optimization problems.
The framework ex- tends existing approaches of single level optimization programming.
We describe a class of gradient estimators for the combinatorial case which reduces the requirements in term of computation complexity; for the continuous vari- ables case the gradient computation takes advantage of pushback approach (i.e. vector-jacobian product) for an efficient implementation.
Experiments suggest that the proposed approach successfully extends existing single level approaches to Bilevel Programming.
Introduction
Neural networks provide unprecedented improvements in perception tasks, however, they struggle to learn basic logic operations (Garcez et al. 2015) or relationships.
When modelling complex systems, for example decision systems, it is not only beneficial to integrate optimization components into larger differentiable system, but also to use general purpose solvers (e g. for Integer Linear Programming or Nonlinear Programming (Bertsekas 1997; Boyd and Vandenberghe 2004)) and problem specific implementation, to discover the governing discrete or continuous relationships.
Recent approaches propose thus differentiable layers that incorporate either quadratic (Amos and Kolter 2017), convex (Agrawal et al. 2019a), cone (Agrawal et al. 2019b), equilibrium (Bai, Kolter, and Koltun 2019), SAT (Wang et al.
or combinatorial (Poganˇci´c et al.
Mandi and
Guns 2020; Berthet et al. 2020) programs.
Use of optimization programming as layer of differentiable systems, requires to compute the gradients through these layers, which is either specific to the optimization problem or zero almost everywhere, when dealing with discrete variables.
Proposed gradient estimates either relax the combinatorial problem (Mandi and Guns 2020), or perturb the input variables (Berthet et al.
2020; Domke 2010) or linearly approximate the loss function (Pogan ˇ ci ´ c et al. 2019).
These approaches though, do now allow to directly express models with conflicting objectives, for example in structural learning (Elsken, Metzen, and Hutter 2019) or adversarial system (Goodfellow et al. 2014).
We thus consider the use of bilevel optimization programming as a layer.
Bilevel Optimization Program (Kleinert et al. 2021;
Colson,
Marcotte, and Savard 2007; Dempe 2018; Stackelberg et al.
1952), also known as generalization of Stackelberg Games, is the extension of single-level optimization program, where the solution of one optimization problem (i.e. the outer problem) depends on the solution of another optimization problem (i.e. the inner problem).
This class of problems can model interactions between two actors 1 , where the action of the first depends on the knowledge of the counter-action of the second.
Bilevel Programming finds application in various domains, as in Electricity networks, Economics, Environmental policy, Chemical plant, defence and planning (Dempe 2018; Sinha, Malo, and Deb 2017).
In general,
Bilevel programs are NP-hard (Sinha, Malo, and Deb 2017), they require specialized solvers and it is not clear how to extend previous approaches, since standard chain rule is not directly applicable.
By modelling the bilevel optimization problem as an implicit layer (Bai, Kolter, and Koltun 2019), we consider the more general case where 1) the solution of the bilevel problem is computed separately by a bilevel solver; thus leveraging on powerfully solver developed over various decades (Kleinert et al. 2021); and 2) the computation of the gradient is more efficient, since we do not have to propagate gradient through the solver.
We thus propose Differentiating through Bilevel Optimization Programming (BiGrad): • BiGrad comprises of forward pass, where existing solvers can be used, and backward pass, where BiGrad estimates gradient for both continuous and combinatorial problems based on sensitivity analysis; • we show how the proposed gradient estimators relate
to the single-level analogous and that the proposed approach is beneficial in both continuous and discrete cases.
Examples of Bilevel Optimization Problems Physical System with control sub-system example
Bilevel Programming is to model the interaction of a dynamical system ( x ) and its control sub-system ( y ), as for example an industrial plant or a physical process.
The control sub-system changes based on the state of the underlying dynamical system, which itself solves a physics constraint optimization problem (Raissi, Perdikaris, and Karniadakis 2019; de Avila Belbute-Peres et al. 2018).
Interdiction problem example Two actors discrete Interdiction problems (Fischetti et al. 2019), where one actor ( x ) tries to interdict the actions of another actors ( y ) under budget constraints, arise in various areas, from marketing, protecting critical infrastructure, preventing drug smuggling to hinder nuclear weapon proliferation.
Min-max problem example Min-max problems are used to model robust optimization problems (Ben-Tal, El Ghaoui, and Nemirovski 2009), where a second variable represents the environment and is constrained to an uncertain set that captures the unknown variability of the environment.
Adversarial attack in Machine Learning Bilevel probram is used the represents the interaction between a machine learning model ( y ) and a potential attacker ( x ) (Goldblum, Fowl, and Goldstein 2019) and is used to increase the resilience to intentional or unintended adversarial attacks.
Differentiable Bilevel Optimization Layer
We model the Bilevel Optimization Program as an Implicit
Layer (Bai, Kolter, and Koltun 2019), i.e. as the solution of an implicit equation H ( x, y, z ) = 0 , in order to derive the gradient using the implicit function theorem, where z is given and represents the parameters of our system we want to estimate, and x, y are output variables (Fig.1).
We also assume we have access 2 to a solver ( x, y ) =
Solve H ( z ) .
The bilevel Optimization Program is then used a layer of a differentiable system, whose input is d and output is given by u = h ψ ◦ Solve H ◦ h θ ( d ) =
h ψ,θ ( d ) , where ◦ is the function composition operator.
We want to learn the parameters ψ, θ of the function h ψ,θ ( d ) that minimize the loss function
L ( h ψ,θ ( d ) , u ) , using the training data D tr = { ( d, u ) N tr i =1 } .
In order to be able to perform the end-to-end training, we need to back-propagate the gradient of the Bilevel Optimization Program Layer, which can not be accomplish only using chain rule.
2. Continuous Bilevel Programming
We now present the definition of the continous Bilevel Optimization problem, which comprises of two non-linear function f, g , as
[Equation]
where the left part problem is called outer optimization problem and resolves for the variable x ∈ X , with X
= R n .
The right problem is called the inner optimization problem and solves for the variable y ∈ Y , with Y = R m .
The variable z ∈ R p is the input variable and is a parameter for the bilevel problem.
Min-max is special case of Bilevel optimization problem
min y ∈ Y max x ∈ X g ( x, y, z ) , where the minimization functions are equal and opposite in sign.
2. Combinatorial Bilevel Programming
When the variables are discrete, we restrict the objective functions to be multi-linear (Greub 1967).
Various important combinatorial problems are linear in discrete variables (e.g. VRP, TSP, SAT 3 ), one example form is the following
[Equation]
The variables x, y have domains in x ∈ X, y ∈ Y , where
X, Y are convex polytopes that are constructed from a set of distinct points X ⊂ R n , Y ⊂ R m , as their convex hull.
The outer and inner problems are Integer Linear Programs (ILPs).
The multi-linear operator is represented by the inner product ⟨ x, y ⟩ A = x T Ay .
We only consider the case where we have separate parameters for the outer and inner problems, z ∈ R p and w ∈ R q .
BiGrad: Gradient estimation
Even if the discrete and continuous variable cases share a similar structure, the approach is different when evaluating the gradients.
We can identify the following common basic steps (Alg.1):
In the forward pass, solve the combinatorial or continuous Bilevel Optimisation problem as defined in Eq.1(or Eq.2) using existing solver; 2.
During the backward pass, compute the gradient d z L (and d w L ) using the suggested gradients (Sec.3.1 and Sec.3.2) starting from the gradients on the output variables ∇ x L and ∇ y L .
3 Vehicle Routing Problem, Boolean satisfiability problem.
3. Continuous Optimization
To evaluate the gradient of the variables z versus the loss function L , we need to propagate the gradients of the two output variables x, y through the two optimization problems.
We can use the implicit function theorem to approximate locally the function z → ( x, y ) .
We have thus the following main results 4 .
Theorem 1.
Consider the bilevel problem of Eq.1, we can build the following set of equations that represent the equivalent problem around a given solution x ∗ , y ∗ , z ∗ :
[Equation]
where
[Equation]
where we used the short notation f
= f ( x, y, z ) , g = g ( x, y, z ) , F = F ( x, y, z ) , G = G ( x, y, z )
Theorem 2.
Consider the problem defined in Eq.1, then the total gradient of the parameter z w.r.t.
the loss function L ( x, y, z ) is computed from the partial gradients ∇ x L, ∇ y L, ∇ z L as
[Equation]
The implicit layer is thus defined by the two conditions
F ( x, y, z ) =
and G ( x, y, z ) = 0 .
We notice that Eq.
can be solved without explicitly computing the Jacobian matrices and inverting the system, but adopting the VectorJacobian product approach we can proceed from left to right to evaluate d z L .
In the following section we describe how affine equality constraints and nonlinear inequality can be used when modelling f, g .
We also notice that the solution of Eq.5 does not require to solve the original problem, but only to apply matrix-vector products, i.e. linear algebra, and the evaluation of the gradient that can be computed using automatic differentiation.
Linear Equality constraints To extend the model of Eq.
to include linear equality constraints of the form Ax = b and
By = c on the outer and inner problem variables, we use the following change of variables
[Equation]
where A ⊥ , B ⊥ are the orthogonal space of A and B , i.e.
AA ⊥ = 0 , BB ⊥ = 0 , and x 0 , y 0 are one solution of the equations, i.e. Ax
=
b, By 0 = c .
Non-linear Inequality constraints Similarly, to extend the model of Eq.1 when we have non-linear inequality constraints, we use the barrier method approach (Boyd and Vandenberghe 2004), where the variable is penalized with a logarithmic function to violate the constraints.
Specifically, let us consider the case where f i , g i are inequality constraint functions, i.e. f i < 0 , g i < 0 , for the outer and inner problems.
We then define new functions
[Equation]
where t is a variable parameter, which depends on the violation of the constraints.
The closer the solution is to violate the constraints, the larger the value of t is.
Bilevel
Cone programming We show here how
Theorem.2 can be applied to bi-level cone programming extending single-level cone programming results (Agrawal et al. 2019b), where we can use efficient solvers for cone programs to compute a solution of the bilevel problem (Ouattara and Aswani 2018)
[Equation]
In this bilevel cone programming, the inner and outer problem are both cone programs, where R ( y ) , P ( x ) represents a linear transformation, while C, r, D, p are new parameters of the problem, while K is the conic domain of the variables.
In the hypothesis that a local minima of Eq.8 exists, we can use an interior point method to find such point.
To compute the bilevel gradient, we then use the residual maps (Busseti,
Moursi, and Boyd 2019) of the outer and inner problems.
Indeed, we can then apply Theorem 2, where F = N 1 ( x, Q, y ) and G = N 2 ( y, Q, x ) are the normalized residual maps defined in (Busseti, Moursi, and Boyd 2019; Agrawal et al.
2019a) of the outer and inner problems.
When we consider discrete variables, the gradient is zero almost everywhere.
We thus need to resort to estimate gradients.
For the bilevel problem with discrete variables of Eq.2, when the solution of the bilevel problem exists and its solution is given (Kleinert et al. 2021), Thm.3 gives the gradients of the loss function with respect to the input parameters.
Theorem 3.
Given the Eq.2 problem, the partial variation of a cost function L ( x, y, z, w ) on the input parameters has the following form:
[Equation]
The ∇ x y, ∇ y x terms capture the interaction between outer and inner problems.
We could estimate the gradients in
Thm.3 using the perturbation approach suggested in (Berthet et al. 2020), which estimate the gradient as the expected value of the gradient of the problem after perturbing the input variable, but, similar to REINFORCE (Williams 1992), this introduces large variance.
While it is possible to reduce variance in some cases (Grathwohl et al. 2017) with the use of additional trainable functions, we consider alternative approaches as described in the following.
Differentiation
of
blackbox
combinatorial solvers (Poganˇci´c et al. 2019) propose a way to propagate the gradient through a single level combinatorial solver, where ∇ z L ≈ 1 τ [ x ( z + τ ∇ x L ) − x ( z )] when x ( z ) =
arg max x ∈ X ⟨ x, z ⟩ .
We thus propose to compute the variation on the input variables from the two separate problems of the Bilevel Problem:
[Equation]
or alternatively, if we have only access to the Bilevel solver and not to the separate ILP solvers, we can express ∇
[Equation]
where x ( z, y ) and y ( w, x ) represent the solutions of the two problems separately, s ( v )
=
( z, w ) → ( x, y ) the complete solution to the Bilevel Problem, τ → 0 is a hyper-parameter and E = (cid:20)
A
C (cid:21) .
This form is more convenient than Eq.9, since it does not require to compute the cross terms, ignoring thus the interaction of the two levels.
Straight-Through gradient
In estimating the input variables z, w of our model, we may not be interested in the interaction between the two variable x, y .
Let us consider, for example, the squared ℓ 2 loss function defined over the output variables
[Equation]
where L 2 ( x ) = 12 ∥ x − x ∗ ∥ 22 and x ∗ is the true value.
The loss is non zero only when the two vectors disagree, and with integer variables, it counts the difference squared or, in case of the binary variables, it counts the number of differences.
If we compute ∇ x L 2 ( x ) =
( x − x ∗ ) in the binary case, we have that ∇ x i L 2 ( x ) =
+1 if x ∗ i =
∧ x i = 1 , ∇ x i L 2 ( x ) = − 1 if x ∗ i =
∧ x i = 0 , and 0 otherwise.
This information can be directly used to update the z i variable in the linear term ⟨ z, x ⟩ , thus we can estimate the gradients of the input variables as ∇ z i L 2 = − λ ∇ x i L 2 and ∇ w i L 2 = − λ ∇ y i L 2 , with some weight λ > 0 .
The intuition is that, the weight z i associated with the variable x
i is increased, when the value of the variable x i reduces.
In the general multilinear case we have additional multiplicative terms.
Following this intuition (see Sec.A.3), we thus use as an estimate of the gradient of the variables
[Equation]
This is equivalent in Eq.2 where ∇
z x = ∇ w y = − I and ∇ y x = 0 , thus ∇ x y = 0 .
This update is also equivalent to
Eq.10, without the soluton computation.
The advantage of this form is that it does not requires to solve for an additional solution in the backward pass.
For the single level problem, gradient has the same form of the Straight-Through gradient proposed by (Bengio, L´eonard, and Courville 2013), with surrogate gradient ∇ z x = − I .
Related Work
Bilevel Programming in machine learning Various papers model machine learning problem as Bilevel problems, for example in Hyper-parameter Optimization (MacKay et al. 2019;
Franceschi et al. 2018), Meta-Feature Learning (Li and Malik 2016), Meta-Initialization Learning (Rajeswaran et al. 2019), Neural Architecture Search (Liu, Simonyan, and Yang 2018), Adversarial Learning (Li et al.
, Deep Reinforcement Learning (Vahdat et al. 2020) and Multi-Task Learning (Alesiani et al. 2020).
In these works the main focus is to compute the solution of the bilevel optimization problems.
In (MacKay et al. 2019; Lorraine and Duvenaud 2018), the best response function is modeled as a neural network and the solution is found using iterative minimization, without attempting to estimate the complete gradient.
Many bilevel approaches rely on the use of the implicit function to compute the hyper-gradient (Sec. 3.5 of (Colson, Marcotte, and Savard 2007)), but do not use bilevel as layer.
Quadratic, Cone and Convex single-level Programming
Various works have addressed the problem of differentiate through quadratic, convex or cone programming (Amos 2019; Amos and Kolter 2017; Agrawal et al.
2019b,a).
In these approaches the optimization layer is modelled as an implicit layer and for the cone/convex case the normalized residual map is used to propagate the gradients.
Contrary to our approach, these work only address single level problems.
These approaches do not consider combinatorial optimization.
Implicit layer Networks While classical deep neural neural networks perform a single pass through the network at inference time, a new class of systems performs inference by solving an optimization problem.
Example of this are Deep
Equilibrium Network (DEQ) (Bai, Kolter, and Koltun 2019) and NeurolODE (NODE) (Chen et al. 2018).
Similar to our
[Figure]
approach, the gradient is computed based on sensitivity analysis of the current solution.
These methods only consider continuous optimization.
Combinatorial optimization Various papers estimate gradients of single-level combinatorial problems using relaxation.
(Wilder, Dilkina, and Tambe 2019; Elmachtoub and Grigas 2017; Ferber et al. 2020; Mandi and Guns 2020) for example use ℓ 1 , ℓ 2 or log barrier to relax the Integer Linear Programming (ILP) problem.
Once relaxed the problem is solved using standard methods for continuous variable optimization.
An alternative approach is suggested in other papers.
For example in (Pogan ˇ ci ´ c et al. 2019)
the loss function is approximated with a linear function and this leads to an estimate of the gradient of the input variable similar to the implicit differentiation by perturbation form (Domke 2010).
(Berthet et al. 2020) is another approach that uses also perturbation and change of variables to estimate the gradient in a ILP problem.
SatNet (Wang et al. 2019) solves
MAXSAT problems by solving a continuous semidefinite program (SDP) relaxation of the original problem.
These works only consider single-level problems.
Discrete latent variables Discrete random variables provide an effective way to model multi-modal distributions over discrete values, which can be used in various machine learning problems, e.g. in language models (Yang et al.
2017) or for conditional computation (Bengio, L´eonard, and
Courville 2013).
Gradients of discrete distribution are not mathematical defined, thus, in order to use gradient based method, gradient estimations have been proposed.
A class of methods is based on Gumbel-Softmax estimator (Jang, Gu, and Poole 2016; Maddison, Mnih, and Teh 2016;
Paulus,
Maddison, and Krause 2021).
Experiments
We evaluate BiGrad with continuous and combinatorial problems to shows that improves over single-level approaches.
In the first experiment we compare the use of BiGrad versus the use of the implicit layer proposed in (Amos
[Table]
and Kolter 2017) for the design of Optimal Control with adversarial noise.
In the second part, after experimenting with adversarial attack, we explore the performance of BiGrad with two combinatorial problems with Interdiction, where we adapted the experimental setup proposed in (Poganˇci´c et al. 2019).
In these latter experiments, we compare the formulation in Eq.11 (denoted by Bigrad(BB)) and the formulation of Eq.12 (denoted by Bigrad(PT)).
In addition we compare with the single level BB-1 from (Poganˇci´c et al.
and single level straight-through (Bengio, L´eonard, and Courville 2013; Paulus, Maddison, and Krause 2021), with the surrogate gradient ∇ z x = − I , (PT-1) gradient estimations.
We compare against Supervised learning (SL), which ignores the underlay structure of the problem and directly predicts the solution of the bilevel problem.
5. Optimal Control with adversarial disturbance
We consider the design of a robust stochastic control for a
Dynamical System (Agrawal et al. 2019b).
The problem is to find a feedback function u
= ϕ ( x ) that minimizes min
[Equation]
13b
[Equation]
where x t ∈ R n is the state of the system, while w t is a i.i.d.
random disturbance and x 0 is given initial state.
To solve this problem we use Approximate Dynamic Programming (ADP) (Wang and Boyd 2010) that solves a proxy quadratic problem
[Equation]
We can use the optimization layer as shown in Fig.2(a) and update the problem variables (e.g. P, Q, q ) using gradient descent.
We use the linear quadratic regulator (LQR) solution as initial solution (Kalman 1964).
The optimization module is replicated for each time step t , similarly to Recursive Neural Network (RNN).
We can build a resilient version of the controller in the hypothesis that an adversarial is able to inject a noise of limited energy, but arbitrary dependent on the control u , by solving the following bilevel optimization problem
[Equation]
where Q ( u, x ) =
u T Pu + x t Qu + q t u
and we want to learn the parameters
z =
( P, Q, q ) , where y
= u t , x = ϵ of Eq.1.
[Table] [Table]
We evaluate the performance to verify the viability of the proposed approach and compare with LQR and OptNet (Amos and Kolter 2017), where the outer problem is substituted with a best response function that computes the adversarial noise based on the computed output; in this case the adversarial noise is a scaled version of Qu of Eq.14.
Tab. and Fig.2(b) present the performance using BiGrad, LQR and the adversarial version of OptNet.
BiGrad improves over two-step OptNet (Tab.1), because is able to better model the interaction between noise and control dynamic.
5. Adversarial ML with discrete latent variables
Machine learning models are heavily affected by the injection of intentional noise (Madry et al. 2017; Goodfellow,
Shlens, and Szegedy 2014).
Adversarial attack typically requires the access to the machine learning model, in this way the attack model can be used during training to include its effect.
Instead of training an end-to-end system as in (Goldblum, Fowl, and Goldstein 2019), where the attacker is aware of the model, we consider the case where the attacker can inject a noise at feature level, as opposed at input level (as in (Goldblum, Fowl, and Goldstein 2019)), this allows us to model the interaction as a bilevel problem.
Thus, to demonstrate the use of a bilevel layer, we design a system that is composed of a feature extraction layer, followed by a discretization layer that operates on the space of { 0 , 1 } m , where m is the hidden feature size, followed by a classification layer.
The network used in the experiments is composed of two convolutional layers with max-pooling and two linear layers, all with relu activation functions, while the classification is a linear layer.
We consider an more limited attacker that is not aware of the loss function of the model and does not have access to the full model, but rather only to the input of the discrete layer and is able two switch Q discrete variables, The interaction of the discrete layer with the attacker is described by the following bilevel problem:
[Equation]
where Q represents the sets of all possible attack, B the budget of the discritization layer and y is the output of the layer.
For the simulation, we compute the solution by sorting the features by values and considering only the first B values, while the attacker will obscure (i.e. set to zero) the first Q positions.
The output y thus will have ones on the
Q to B non-zero positions, and zero elsewhere.
We train three models, on CIFAR-10 dataset for 50 epochs.
For comparison we consider:1) the vanilla CNN network (i.e. without the discrete features); 2) the network with the single level problem (i.e. the single-level problem without attacker) and; 3) the network with the bilevel problem (i.e. the minmax discretization problem defined in Eq.16).
We then test the networks to adversarial attack using the PGD (Madry et al. 2017) attack similar to (Goldblum, Fowl, and Goldstein 2019).
Similar results apply for FGSM attack (Fast
Gradient Sign Attack) (Goodfellow, Shlens, and Szegedy 2014).
We also tested the network trained as vanilla network, where we added the min-max layer after training.
From the results (Tab.3), we notice: 1) The min-max network shows improved resilience to adversarial attack wrt to the vanilla network, but also with respect to the max (single-level) network; 2)
The min-max layer applied to the vanilla trained network is beneficial to adversarial attack; 3)
The min-max network does not significantly change performance in presence of adversarial attack at the discrete layer (i.e. between
Q=0 and Q=10).
This example shows how bilevel-layers can be successfully integrated into Machine Learning system as differentiable layers.
5. Dynamic Programming: Shortest path with Interdiction
We consider the problem of Shortest Path with Interdiction, where the set of possible valid paths (see Fig.3(a)) is Y and the set of all possible interdiction is X .
The mathematical problem can be written as
[Equation]
where ⊙ is the element wise product.
This problem is multilinear in the discrete variables x, y, z .
The z, w variables are output of neural network whose input are the Warcraft II tile images.
The aim is to train the parameters of weight
[Table]
[Table] [Table]
[Figure]
network, such that we can solve the shortest path problem only based on the input image.
For the experiments, we followed and adapted the scenario of (Pogan ˇ ci ´
c et al. 2019)
and used the Warcraft II tile maps of (Guyomarch 2017).
We implemented the interdiction Game using a two stage min-max-min algorithm
(K ¨ ammerling and Kurtz 2020).
In
Fig.3(b) it is possible to see the effect of interdiction on the final solution.
Tab.2 shows the performances of the proposed approaches, where we allow for B = 3 interdictions and we used tile size of 12 × 12 , 18 × 18 , 24 × 24 .
The loss function is the Hamming and ℓ 1 loss evaluated on both the shortest path y and the intervention x .
The gradient estimated using Eq.
(BB) provides more accurate results, at double of computation cost of PT.
Single level BB-1 approach outperforms PT, but shares similar computational complexity, while single level PT-1 is inferior to PT.
As expected, SL outperforms other methods during training, but completely fails during validation.
Bigrad improves over single-level approaches, because includes the interaction of the two problems.
5. Combinatorial Optimization: Travel Salesman Problem (TSP) with Interdiction
Travel Salesman Problem (TSP) with interdiction consists of finding shortest route y ∈ Y that touches all cities, where some connections x ∈ X can be removed.
The mathematical
[Figure]
problem to solve is given by
[Equation]
(18) where z, w are cost matrices for salesman and interceptor.
Similar to the dynamic programming experiment, we implemented the interdiction Game using a two stage min-maxmin algorithm (K ¨ ammerling and Kurtz 2020).
Fig.4 shows the effect of a single interdiction.
The aim is to learn the weight matrices, trained with interdicted solution on subset of the cities.
Tab.4 describes the performance in term of accuracy on both shortest tour and intervention.
We use Hamming and ℓ 1 loss function.
We only allow for B = 1 intervention, but considered k = 8 , 10 and 12 cities from a total of 100 cities.
Single and two level approaches perform similarly in the train and validation.
Since the number of interdiction is limited to one, the performance of the single level approach is not catastrophic, while the supervised learning approach completely fails in the validation set.
Bigrad thus improves over single-level and SL approaches.
Since Bigrad(PT) has similar performance of BiGrad(BB), thus PT is preferable in this scenario, since it requires less computation resources.
Conclusions
BiGrad generalizes existing single level gradient estimation approaches and is able to incorporate Bilevel Programming as learnable layer in modern machine learning frameworks, which allows to model conflicting objectives as in adversarial attack.
The proposed novel gradient estimators are also efficient and the proposed framework is widely applicable to both continuous and discrete problems.
The impact of BiGrad has a marginal or similar cost with respect to the complexity of computing the solution of the Bilevel Programming problems.
We show how BiGrad is able to learn complex logic, when the cost functions are multi-linear.
Optimization Programming A. Relationship with other related work
Predict then optimize Predict then Optimize (two-stage)
(Elmachtoub and Grigas 2017; Ferber et al. 2020) or solving linear programs and submodular maximization from (Wilder, Dilkina, and Tambe 2019) solve optimization problems when the cost variable or the minimization function is directly observable.
In contrary, in our approach we only have access to a loss function on the output of the bilevel problem, thus allowing to use as a layer.
A. Proofs
Proof of Linear Equality constraints.
Here we show that
[Equation]
includes all solution of Ax = b .
First we have that AA ⊥ =
and Ax 0 = b by definition.
This implies that Ax ( u ) =
A ( x 0 + A ⊥ u ) =
Ax 0 = b .
Thus ∀ u
→ Ax ( u ) = b .
The difference x ′ − x 0 belongs to the null space of A , indeed
A ( x ′ − x 0 ) =
Ax ′ − Ax 0 =
b − b = 0 .
The null space of A has size n − ρ ( A ) .
If ρ ( A ) = n , where A ∈ R m × n , m ≥ n , then there is only one solution x = x
= A † b , A † the pseudo inverse of A .
If ρ ( A ) < n , then ρ ( A ⊥ )) = n − ρ ( A ) is a based of all vectors s.t.
Ax ( u ) =
b , since ρ ( A ⊥ ))
= n − ρ ( A ) is the size of the null space of A .
In fact A ⊥ is the base for the null space of A .
The same applies for y ( v ) =
y
+ B ⊥
v
and By ( v ) =
c .
Proof of Theorem 1.
The second equation is derived by imposing the optimally condition on the inner problem.
Since we do not have inequality and equality constraints we optimal solution shall equate the gradient w.r.t.
y to zero, thus
G = ∇ y g = 0 .
The first equation is also related to the optimality of the x variable w.r.t.
to the total derivative or hypergradient, thus we have that
=
d x f = ∇ x f + ∇ y f ∇ x y .
In order to compute the variation of y , i.e. ∇ x y we apply the implicit theorem to the inner problem, i.e. ∇ x G + ∇ y G ∇ x y = 0 , thus obtaining ∇ x y = −∇ − 1 y G ∇ x G .
Proof of Theorem 2.
In order to prove the theorem, we use the Discrete Adjoin Method (DAM).
Let consider a cost function or functional L ( x, y, z ) evaluated at the output of our system.
Our system is defined by the two equations
F = 0 , G = 0 from Theorem 1.
Let us first consider the total variations: d L, d F = 0 , d G = 0 , where the last conditions are true by definition of the bilevel problem.
When we expand the total variations, we obtain
[Equation]
We now consider d L + d Fλ + d Gγ =
[ ∇ x L + ∇ x Fλ + ∇ x Gγ ]d x +[ ∇ y L + ∇ y Fλ + ∇ y Gγ ]
d y +[ ∇ z L + ∇ z Fλ + ∇ z Gγ ]d z .
We ask the first two terms to be zero to find the two free variables λ, γ :
[Equation]
or in matrix form
[Equation]
We can now compute the d z L = ∇ z L + ∇ z Fλ + ∇ z Gγ with λ, γ from the previous equation.
[Figure]
Proof of Theorem 3.
The partial derivatives are obtained by using the perturbed discrete minimization problems defined by Eqs.24.
We first notice that ∇ x min y ∈ Y ⟨ x, y ⟩ = arg min y ∈ Y ⟨ x, y ⟩ .
This result is obtained by the fact that min y ∈ Y ⟨ x, y ⟩ = ⟨ x, y ∗ ⟩ , where y ∗ = arg min y ∈ Y ⟨ x, y ⟩ and applying the gradient w.r.t.
the continuous variable x ; while Eqs.
23 are the expected functions of the perturbed minimization problems.
Thus, if we compute the gradient of the perturbed minimizer, we obtain the optimal solution, proper scaled by the inner product matrix.
For example ∇ x ˜Φ η = Ax ∗ ( z, y ) , with A the inner product matrix.
To compute the variation on the two parameter variables, we have that d L = ∇ x L d x + ∇ y L d y + ∇ z L d z + ∇ w L d w and that d w/ d z = 0 , d z/ d w = 0 from the dependence diagram of Fig.
A. Gradient Estimation based on perturbation
We can use the gradient estimator using the perturbation approach proposed in (Berthet et al. 2020).
We thus have
[Equation]
η →(22b) ∇
[Equation]
(22c) ∇
[Equation] [Equation]
z, y )
[Equation]
(22d) ∇
[Equation]
y
[Equation]
∇
[Equation]
∇
[Equation]
x (22e) and ˜Φ η ( z, y ) =
E u ∼
U
[Equation]
( z
[Equation]
ηu, y ) (23a) ˜Ψ
η ( w, x )
[Equation]
E
[Equation]
( w
[Equation]
ηu, x ) (23b) ˜Θ
η
[Equation] [Equation]
E
[Equation]
ηu ) (23c) ˜ W
η ( y, z )
[Equation]
( z,
[Equation]
+ ηu ) (23d)
, while
[Equation]
which are valid under the conditions of (Berthet et al. 2020), while τ and µ are hyper-parameters.
A. Alternative derivation
Let consider the problem min x ∈ K ⟨ z, x ⟩ A and let us define Ω x a penalty term that ensures x ∈ K .
We can define the generalized lagragian L ( z, x, Ω) =
⟨ z, x ⟩ A +Ω x .
One example of Ω x = λ T | x − K
( x ) | or Ω x = − ln | x − K ( x ) | where
K ( x ) is the projection into K .
To solve the Lagragian, we solve the unconstrained problem min x max Ω x L ( z, x, Ω x ) .
At the optimal point ∇ x L = 0 .
Let us define F = ∇ x L =
A T z + Ω ′ x , then ∇ x F = Ω ′′ x and ∇ z F = A T .
If we have
F ( x, z ) = 0 and a cost function
L ( x, z ) , we can compute d z L = ∇ z L − ∇ x L ∇ − 1 x F ∇ z F .
Now F ( x, z, Ω x ) = 0 , we can apply the previous result and d z L = ∇ z L − ∇ x L Ω ′′− 1 x
A T .
If we assume Ω ′′ x = I and ∇ z L = 0 , then d z L = − A ∇ x L .
A. Memory Efficiency
For continuous optimization programming, by separating the computation of the solution and the computation of the gradient around the current solution we 1) compute the gradient more efficiently, in particular we compute second order gradient taking advantage of the vector-jacobian product (push-back operator) formulation without explicitly inverting and thus building the jacobian or hessian matrices; 2) use more advanced and not differentialble solution techniques to solve the bilevel optimization problem that would be difficult to integrate using automatic differentiable operations.
Using VJP we reduce memory use from O ( n 2 ) to
O ( n ) .
Indeed using an iterative solver, like generalized minimal residual method (GMRES) (Saad and Schultz 1986), we only need to evaluate the gradients of Eq.5 and not invert the matrix neither materialize the large matrix and computing matrix-vector products.
Similarly, we use Conjugate
Gradient (CG) method to compute Eq.4, which requires to only evaluate the gradient at the current solution and nor inverting neither materializing the Jacobian matrix.
An implementation of a bilevel solver would have a memory complexity of O ( Tn ) , where T are the number of iterations of the bilevel algorithm.
A. Experimental Setup and Computational Resources
For the Optimal Control with adversarial disturbance we follow a similar setup of (Agrawal et al. 2019a), where we added the adversarial noise as described in the experiments.
For the Combinatorial Optimization, we follow the setup of (Poganˇci´c et al. 2019).
The dataset is generated by solving the bilevel problem on the same data of (Poganˇci´c et al.
For section 5.3, we use the warcraft terrain tiles and generate optimal bilevel solution with the correct parameters ( z, w ) , where z is the terrain transit cost and w is the interdiction cost, considered constant to 1 in our experiment.
X is the set of all feasible interdictions, in our experiment we allow the maximum number of interdictions to be B .
For section 5.4, on the other hand the z represents the true distances among cities and w a matrix of the interdiction cost, both unknown to the model.
X is the set of all possible interdictions.
In these experiments, we solved the bilevel problem using the min-max-min algorithm (K ¨ ammerling and Kurtz 2020).
For the Adversarial Attack, we used two convolutional layers with max-pooling, relu activation layer, followed by the discrete layer of size m = 2024 , B = 100 , Q = 0 , 10 .
A final linear classification layer is used to classify CIFAR10.
We run over 3 runs, 50 epochs, learning rate lr = 3 e − 4 and
Adam optimizer.
Experiments were conducted using a standard server with 8 CPU, 64Gb of RAM and GeForce RTX 2080 GPU with 6Gb of RAM.
A. Jacobian-Vector and Vector-Jacobian Products
The Jacobian-Vector Product (JVP) is the operation that computes the directional derivative J f ( x ) u , with direction u ∈ R m , of the multi-dimensional operator f : R m → R n , with respect to x ∈ R m , where J f ( x ) is the Jacobian of f evaluated at x .
On the other hand, the Vector-Jacobian product (VJP) operation, with direction v ∈ R n , computes the adjoint directional derivative v T J f ( x ) .
JVP and VJP are the essential ingredient for automatic differentiation (Elliott 2018; Baydin et al. 2018).