NCVX : A General-Purpose Optimization Solver for Constrained Machine and Deep Learning
Abstract
Imposing explicit constraints is relatively new but increasingly pressing in deep learning, stimulated by, e.g., trustworthy AI that performs robust optimization over complicated perturbation sets and scientific applications that need to respect physical laws and constraints.
However, it can be hard to reliably solve constrained deep learning problems without optimization expertise.
The existing deep learning frameworks do not admit constraints.
General-purpose optimization packages can handle constraints but do not perform auto-differentiation and have trouble dealing with nonsmoothness.
In this paper, we introduce a new software package called NCVX , whose initial release contains the solver PyGRANSO , a PyTorch-enabled general-purpose optimization package for constrained machine/deep learning problems, the first of its kind .
NCVX inherits auto-differentiation,
GPU acceleration, and tensor variables from PyTorch, and is built on freely available and widely used open-source frameworks.
NCVX is available at https://ncvx.org , with detailed documentation and numerous examples from machine/deep learning and other fields.
1. Introduction
Mathematical optimization is an indispensable modeling and computational tool for all science and engineering fields, especially for machine/deep learning.
To date, researchers have developed numerous foolproof techniques, user-friendly solvers, and modeling languages for convex (CVX) problems, such as SDPT3 [65], Gurobi [28], Cplex [14], TFOCS [5], CVX(PY) [22, 27],
AMPL [24], YALMIP [45].
These developments have substantially lowered the barrier of CVX optimization for non-experts.
However, practical problems, especially from machine/deep learning, are often nonconvex (NCVX), and possibly also constrained (CSTR) and nonsmooth (NSMT).
There are methods and packages handling NCVX problems in restricted settings: PyTorch [55] and TensorFlow [1] can solve large-scale NCVX, NSMT problems without constraints.
CSTR problems can be heuristically turned into penalty forms and solved as unconstrained, but this may not produce feasible solutions for the original problems.
When the constraints are simple, structured methods such as projected (sub)gradient and Frank-Wolfe [60] can be used.
When the constraints are differentiable manifolds, one can consider manifold optimization methods and packages, e.g., (Py)manopt [6, 66], Geomstats [52], McTorch [50], and Geoopt [39].
For general CSTR problems, KNITRO [57] and IPOPT [73] implement interior-point methods, while ensmallen [18] and
GENO [42] rely on augmented Lagrangian methods.
However, moving beyond smooth (SMT) constraints, both of these families of methods, at best, handle only special types of NSMT constraints.
Finally, packages specialized for machine learning, e.g., scikit-learn [56], MLib [51] and Weka [71], often use problem-specific solvers that cannot be easily extended to new formulations.
2. The GRANSO and NCVX packages
GRANSO 1 is among the first optimization packages that can handle general NCVX, NSMT, CSTR problems [19]:
[Equation]
Here, the objective f and constraint functions c i ’s are only required to be almost everywhere continuously differentiable.
GRANSO is based on quasi-Newton updating with sequential quadratic programming (BFGS-SQP) and has the following advantages: (1) unified treatment of NCVX problems : no need to distinguish CVX vs NCVX and SMT vs NSMT problems, similar to typical nonlinear programming packages; (2) reliable step-size rule : specialized methods for NSMT problems, such as subgradient and proximal methods, often entail tricky step-size tuning and require the expertise to recognize the structures [60], while GRANSO chooses step sizes adaptively via a gold-standard line search; (3) principled stopping criterion : GRANSO stops its iteration by checking a theory-grounded stationarity condition for NMST problems, whereas specialized methods are usually stopped when reaching ad-hoc iteration caps.
However, GRANSO users must derive gradients analytically 2 and then provide code for these computations, a process which is often error-prone in machine learning and impractical for deep learning.
Furthermore, as part of the MATLAB software ecosystem, GRANSO is generally not compatible with popular machine/deep learning frameworks—mostly in Python and R—and users’ own existing toolchains.
To overcome these issues and facilitate both high performance and ease of use in machine/deep learning, we introduce a new software package called NCVX , whose initial release contains the solver PyGRANSO , a PyTorch-port of GRANSO with several new and key features: (1) auto-differentiation of all gradients, a critical feature to make PyGRANSO user-friendly; (2) support for both CPU and GPU computations for improved hardware acceleration and massive parallelism; (3) support for general tensor variables including vectors and matrices, as opposed to the single vector of concatenated optimization variables that GRANSO uses; (4) integrated support for OSQP [61] and other QP solvers for respectively computing search directions and the stationarity measure on each iteration.
OSQP generally outperforms commercial QP solvers in terms of scalability and speed.
All of these enhancements are crucial for solving large-scale machine/deep learning problems.
NCVX , licensed under the AGPL V3, is built entirely on freely available and widely used open-source frameworks; see https://ncvx.org for documentation and examples.
3. Usage examples: dictionary learning and neural perceptual attack
In order to make NCVX friendly to non-experts, we strive to keep the user input minimal.
The user is only required to specify the optimization variables (names and dimensions of variables) and define the objective and constraint functions.
Here, we briefly demonstrate the usage of PyGRANSO solver on a couple of machine/deep learning problems.
Orthogonal dictionary learning (ODL, [3])
One hopes to find a “transformation” q ∈ R n to sparsify a data matrix Y ∈ R n × m :
[Equation]
where the constraint ∥ q ∥ 2 = 1 is to avoid the trivial solution q = 0 .
Eq.
(2) is NCVX, NSMT, and CSTR: nonsmoothness comes from the objective, and nonconvexity comes from the constraint.
Demos 1 & 2 show the implementations of ODL in GRANSO and PyGRANSO , respectively.
Note that the analytical gradients of the objective and constraint functions are not required in PyGRANSO .
function[f,fg,ci,cig,ce,ceg]=fn(q) f=1/m*norm(q’*Y,1);%obj fg=1/m*Y*sign(Y’*q);%objgrad ci =
[];cig =
[];%no ineq constr ce
=
q’*q - 1; % eq constr ceg =
2*q; % eq constr grad endsoln =
granso(n,fn);
Demo 1: GRANSO for ODL def fn(X_struct): q=X_struct.q f=1/m*norm(q.
T@Y, p=1) # obj ce
= pygransoStruct()
ce.c1 = q.T@q - 1 # eq constr return [f,None,ce] var_in
= {"q": [n,1]}# def variable soln =
pygranso(var_in, fn)
Demo 2: PyGRANSO for ODL
Neural perceptual attack (NPA, [41])
The CSTR deep learning problem, NPA, is shown below:
[Equation]
Here, x is an input image, and the goal is to find its perturbed version (cid:101) x that is perceptually similar to x (encoded by the constraint) but can fool the classifier f (encoded by the objective).
The loss
L ( · , · ) is the margin loss used in Laidlaw et al.
Both f in the objective and ϕ in the constraint are deep neural networks with ReLU activations, making both the objective and constraint functions
NSMT and NCVX.
The d ( x , (cid:101) x ) distance is called the Learned Perceptual Image Patch Similarity (LPIPS)
Demo 3 is the PyGRANSO example for solving Eq.
Note that the codes for data loading, model specification, loss function, and LPIPS distance are not included here.
It is almost impossible to derive analytical subgradients for Eq.
(3), and thus the auto-differentiation feature in PyGRANSO is necessary for solving it.
def comb_fn(X_struct): adv_inputs=
X_struct.x_tilde f=MarginLoss(model(adv_inputs),labels)
# obj ci =
pygransoStruct() ci.c1 = lpips_dists(adv_inputs) - 0.5 # ineq constr.
bound eps=0.
return [f,ci,None] # No eq constr var_in
= {"x_tilde": list(inputs.shape)}
# define variable soln = pygranso(var_in,comb_fn)
Demo 3: PyGRANSO for NPA
4. Constrained deep learning applications
In this section, we highlight 4 families of constrained deep learning problems with highly nontrivial, often nonsmooth, constraints.
These constraints cannot be easily built into the underlying neural networks.
PyGRANSO can directly solve these problems, and detailed codes and tutorials are available on https://ncvx.org/examples .
4.1. Robustness of deep learning models
In visual recognition, deep neural networks (DNNs) are not robust against perturbations—either adversarially constructed or naturally occurring—that are easily discounted by human perception [26, 33].
To formalize robustness, one popular way is the adversarial loss defined as [48]
[Equation]
where f θ is the DNN model, and ∆( x ) is the set of allowable perturbations with a radius ϵ measured with respect to the metric d .
Early works assume that ∆( x ) is the ℓ p norm ball intersected with the natural image box, i.e., { x ′ ∈ [0 , 1] n : ∥ x − x ′ ∥
p ≤ ϵ } , where p = 1 , 2 , ∞ are popular choices [26, 48].
To capture visually realistic perturbations, recent work has also modeled nontrivial transformations [33] that use non- ℓ p metrics.
For empirical robustness evaluation (RE), solving Eq.
(4) leads to the worst perturbations that could fool f θ .
An alternative formalism of robustness is the robustness radius (or minimum distortion radius), defined as the minimal level of perturbation that can cause f θ to change its predicted class:
[Equation]
where the superscript for f θ indexes its elements, i.e., the output logits.
Solving Eq.
(5) produces not only a minimally distorted perturbation x ′ but also a robustness radius, making it another popular choice for RE [15, 16].
For small and restricted f θ and selected d , Eq.
(5) can be solved exactly by mixed integer programming [7, 38, 64].
For general f θ and selected d , lower bounds of the robustness radius can be computed [47, 69, 70, 74].
But in general, Eq.
(5) is heuristically solved via gradient-based methods or iterative linearization [8, 15, 30, 53, 58, 59, 63].
Eqs.
(4) and (5) are difficult constrained problems, especially when d is sophisticated—necessary for modeling realistic perturbations [23, 33, 34, 40, 72].
Popular exisiting solvers for them rely on explicit projections onto simple sets, and they will not work when d is a non- ℓ p metric.
Also, previous work has shown that the quality of the solution using these handcrafted methods is sensitive to key hyperparameters: e.g., step-size schedule and iteration budget [9, 16].
With PyGRANSO , we can reliably solve Eqs.
(4) and (5) with general d ’s with minimal hyperparameter tuning.
4.2. Neural structural optimization
Designing physical structures such as bridges, optical devices and airplanes often boils down to structural optimization, a fundamental family of problems in physical science and engineering [10, 11, 13, 35].
A primitive version of structural optimization takes the form 3 :
[Equation]
[Equation] where x ∈ R d is the vectorized binary design variable that indicates where the material should be put to form the structure, u ∈ R n is the displacement vector, K is the global stiffness matrix, and f is the vector of external force.
For the constraints, K ( x ) u = f encodes the physical laws (e.g.,
Hooke’s law) and is often called the equilibrium constraint , and V ( x ) = v 0 limits the amount of material to be used.
Recent work [35] has introduced the deep image prior (DIP) [67] idea into Eq.
(6): x is reparametrized as x = g θ ( β ) , where g θ is a deep neural network parametrized by θ , and β is a fixed random input.
DIP has shown great promise for solving difficult visual inverse problems (see, e.g., discussions in [44, 68, 77]), and can implicitly promote spatial continuity of the design—which is not explicitly modeled in Eq.
Also, the overparametrization in DIP tends to ease the global optimization.
Relaxing the integer constraint x ∈ { 0 , 1 } d into a box constraint x ∈ [0 , 1] d as is typically done in structural optimization, we arrive at
[Equation]
In order to solve Eq.
(7), recent work [35] transforms Eq.
(7) into an unconstrained problem, which includes 1) eliminating u from the physical constraint by solving the linear system K ( g θ ( β ))
u
= f ; 2) enforcing the couple of constraints V ( g θ ( β )) ≤ v 0 and g θ ( β ) ∈ [0 , 1] d via reparametrization, binary search, and implicit differentiation.
By contrast, using PyGRANSO we can directly solve
Eq.
(7) without any of the problem-specific tricks.
For realistic design problems, x needs to be discrete-valued (it can have more than 2 discrete values in multi-material design), and the constraint
K ( g θ ( β )) u
= f becomes nonlinear— K becomes a nonlinear operator acting on u due to the governing PDEs [10, 11].
PyGRANSO can easily handle these general cases also, whereas the tricks used in [35] cannot be generalized.
4.3. Orthogonal recurrent neural networks (RNNs)
The exploding and vanishing gradient issues are common in RNNs, and they could occur whenever the recurrent kernel does not have unit eigenvalues [2, 4, 29, 32, 43], i.e., the associated weight matrix is not orthogonal.
Recent work proposes imposing orthogonality constraint on the weight matrix directly [31, 43]:
[Equation]
where f θ is the RNN parameterized by θ , and W hh is the recurrent kernel of the RNN (i.e., a subvector of θ ).
Since the constraints define the famous special orthogonal group which is a smooth manifold, manifold optimization methods can be developed to solve Eq.
But these methods entail heavy mathematics from differential geometry, and hence is impractical for nonexperts to implement and build on (there could be additional constraints in real applications).
Again,
PyGRANSO is able to directly handle the nonlinear constraints, whether the user recognizes or not that the constraint forms the special orthogonal group.
4.4. Knowledge-aware machine learning (KAML)
KAML concerns learning that incorporates prior knowledge, e.g., physical laws as constraints.
As an example, it can take the general form of minimizing a loss subject to PDE constraints [21, 49]:
[Equation]
Here, Ω and ∂ Ω denote the domain and its boundary respectively, and the loss L depends on the functional variable u ( x ) over the domain Ω .
The constraints are PDEs, with boundary and/or initial conditions B ( · ) = 0 .
If the loss is a constant, this reduces to solving the PDE problem about u ( x ) .
In supervised learning scenarios, the loss could take the form of 1 /N · (cid:80) Ni =1 ℓ ( y i , u ( x i )) , where { ( x i , y i ) } is the training set and u is the predictor to be learned from the training set.
To solve Eq.
(9), one can parametrize u ( x ) as a neural network, i.e., u ( x ; θ ) .
This is natural in modern supervised learning, and is called physics-informed neural networks (PINNs) in the numerical PDE community [12, 17, 21, 36, 46, 54].
In numerical PDEs, classical methods use finite-difference approximations for all the partial derivatives, whereas the PINN idea directly works with continuous functions that allow partial derivatives to be computed via auto-differentiation.
This “mesh-free” nature of PINNs holds the promise for high-precision solutions even for highdimensional PDEs.
The state-of-the-art methods for solving the DNN-parametrized version of Eq.
(9) use penalty methods, Lagrangian methods, and augmented Lagrangian methods [17, 21, 46, 49], which often involve delicate tuning of multiple hyperparameters and could lead to infeasible solutions.
By contrast, PyGRANSO stops iterations by rigorous check of constraint violation and stationarity.
5. Roadmap
Although NCVX , with the PyGRANSO solver, already has many powerful features, we plan to further improve it by adding several major components: (1) symmetric rank one (SR1) :
SR1, another major type of quasi-Newton methods, allows less stringent step-size search and tends to help escape from saddle points faster by taking advantage of negative curvature directions [20]; (2) stochastic algorithms : in machine learning, computing with large-scale datasets often involves finite sums with huge number of terms, calling for stochastic algorithms for reduced per-iteration cost and better scalability [62]; (3) conic programming (CP) : semidefinite programming and second-order cone programming, special cases of CP, are abundant in machine learning, e.g., kernel machines [76]; (4) minimax optimization (MMO) : MMO is an emerging modeling technique in machine learning, e.g., generative adversarial networks (GANs) [25] and multi-agent reinforcement learning [37].