Convexity Certiﬁcates from Hessians
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The Hessian of a differentiable convex function is positive semideﬁnite.
Therefore,checking the Hessian of a given function is a natural approach to certify convexity.
However, implementing this approach is not straightforward, since it requiresa representation of the Hessian that allows its analysis.
Here, we implementthis approach for a class of functions that is rich enough to support classicalmachine learning.
For this class of functions, it was recently shown how tocompute computational graphs of their Hessians.
We show how to check thesegraphs for positive-semideﬁniteness.
We compare our implementation of the
Hessian approach with the well-established disciplined convex programming (DCP)approach and prove that the Hessian approach is at least as powerful as the DCPapproach for differentiable functions.
Furthermore, we show for a state-of-the-artimplementation of the DCP approach that, for differentiable functions, the Hessianapproach is actually more powerful, that is, it can certify the convexity of a largerclass of differentiable functions.
Introduction
Convex optimization forms the backbone of classical machine learning.
Formulating machinelearning problems as convex optimization problems is attractive, because these problems can besolved globally and efﬁciently.
Several optimization frameworks consisting of a modeling languageand solver exist and are used in machine learning.
Two examples of such frameworks are CVX [Grantand Boyd, 2014] and GENO
[Laue et al. , 2019], which take different approaches.
CVX takes a formal speciﬁcation of an optimization problem and transforms it into a normal formsuch as a convex quadratic program (QP) or semideﬁnite program (SDP).
The transformation is onlypossible if the speciﬁed problem is convex.
Therefore, a convexity test is performed ﬁrst in CVX.
The test is based on a calculus for convex functions called disciplined convex programming (DCP) by
Grant et al.
[2006a], which takes advantage, for example, of the fact that the sum of convex functionsand the positive scaling of a convex function are convex again.
The convexity calculus requires a setof functions, called atoms, whose convexity has to be certiﬁed by other means.
GENO also takes the formal speciﬁcation of an optimization problem, but does not transform it intostandard form.
Instead it uses the speciﬁcation to generate a solver for the speciﬁed problem orproblem class by means of automatic (symbolic) differentiation.
Thus, solvers can also be generatedfor non-convex problems.
However, such solvers usually do not converge to a global optimum, butonly provide a local optimum.
Therefore, in order to assess the quality of a solution, a convexitycertiﬁcate calculated from the speciﬁcation is also of interest to GENO.
For compiling a speciﬁcation into a solver, GENO employs a matrix calculus
[Laue et al. , 2018] forcomputing derivatives of matrix and tensor expressions.
The matrix calculus can also be used tocompute second order derivatives, that is, Hessians.
A test for convexity can thus be reduced to a testof positive semideﬁniteness of the Hessian, which certiﬁes the convexity of a function.
This is the approach that we take here.
Therefore, we design an algorithm for certifying positive semideﬁnitenessof matrix expressions.
For certifying convexity, the algorithm can then be applied to Hessians thatare computed by the matrix calculus in normalized vectorized form.
We start with a formally deﬁned language of mathematical functions (see the supplementary materialfor its grammar).
For expressions from this language, we generate computational graphs of their
Hessians using a matrix calculus and check these graphs for positive semideﬁnitness.
Our algorithmfor certifying positive semideﬁniteness propagates positivity information of subexpressions to the rootof the computational graph by using simple rules for positive semideﬁnite matrices.
Non-trivial subexpressions can be certiﬁed as convex by matching them to known expression templates.
Surprisingly,there is, as we show, a single template that is powerful enough to certify the convexity of large classesof matrix expressions.
Using this template, we show that our Hessian approach covers all twicedifferentiable functions with vector input that can be certiﬁed as convex by the DCP implementationwithin CVX.
Furthermore, we provide classes of differentiable convex functions that can be certiﬁedby our approach, but cannot be certiﬁed as convex by CVX.
An anonymized implementation of ourconvexity certiﬁcation algorithm can be found at https://convcert.pythonanywhere.com .
Related work
In general, certifying that a function is convex is known to be strongly NP-hard [Ahmadi et al. ,2013], even when the function is a multivariate polynomial of degree three and the domain is abounded box
[Ahmadi and Hall, 2020].
However, since certifying convexity is such an importantissue in optimization and machine learning there are many convexity proofs for speciﬁc problems, see
Klibanov [1997] for an example.
Unfortunately, these proofs are problem- and often even instancespeciﬁc and cannot be generalized.
But, there are also generic approaches that we discuss in thefollowing.
Most generic approaches either use a rule-based approach or analyze the Hessian.
Rule-based approaches We have already mentioned the disciplined convex programming (DCP)approach taken by CVX [Grant and Boyd, 2008, 2014;
Agrawal et al. , 2017], which has been portedfrom Matlab to other programming languages, like CVXPY [Diamond and Boyd, 2016] for Python,
CVX for Julia [Udell et al. , 2014], or CVX for R [Fu et al. , 2020].
Any convex function that cannotbe derived from the atomic convex functions by the DCP rule set cannot be certiﬁed as convex by the
DCP approach.
However, functions that have been certiﬁed as convex by other means can be addedmanually as atomic functions.
Hence, over the course of the last few years, many convex functionshave been added as atoms, like for instance quadratic functions x (cid:62) Ax for positive semideﬁnitematrices A , or the negative entropy function x (cid:62) log( x ) .
In our approach, we certify the convexity ofthese functions automatically.
Tawarmalani and Sahinidis [2005] describe a polyhedral branch-and-cut approach for ﬁnding a globaloptimum for non-convex optimization problems.
They use rule-based convexity tests for decomposinga non-convex problem into a set of convex problems.
Posypkin and Khamisov [2021] use intervalarithmetic and a set of rules for determining the convexity of univariate functions, similar to the CVXapproach.
However, unlike CVX their approach can only be applied to simple univariate functions.
Similarly, Singh and Lucet [2021] analyze univariate piecewise polynomial functions.
Fourer etal.
[2010] summarize and generalize the convexity detection methods described by Schichl and
Neumaier [2005] and Fourer and Orban [2010].
Their approach, traversing a function’s computationalgraph and applying composition rules when convexity holds, is again similar to the DCP approach.
Hessian based approaches Camino et al.
[2003] use the software package NCAlgebra by Heltonand de Oliveira [2000] for computing the Hessian of a function that is deﬁned over matrices.
Based ona symbolic Cholesky decomposition, they check the non-negativity of the eigenvalues of the Hessian.
If all eigenvalues are non-negative, the symbolic Hessian is positive semideﬁnite and thus can certifyconvexity.
However, this approach only works for very simple functions, i.e., only polynomials ofmatrices and their inverses.
In these cases, the Hessian is always a quadratic function for which thesymbolic Cholesky decomposition can be computed.
This approach works for the function x (cid:62) Ax ,but not for the negative entropy function x (cid:62) log( x ) .
Using automatic differentiation for computing Hessians and then certifying convexity was proposedby Nenov et al.
However, the proposal does not include speciﬁc algorithms and has not beenimplemented yet.
Other approaches An approach that is neither rule-based nor based on analyzing the Hessian wasproposed by Carmon et al.
[2017] who establish a relationship between the number of iterationsneeded for minimizing a function and its convexity.
In general, if a function is convex, convergencerates can be estimated.
Hence, if during the minimization process these convergence rates areviolated, then one has found a certiﬁcate that this function is not convex.
However, if convergencerates are satisﬁed during the optimization process, no statement about convexity can be made.
Ingeneral, if one starts close to a local minimum, then the function looks locally like a convex functionto the minimization process while globally it does not need to be convex.
Instead of looking atthe convergence rates, one could modify this approach by computing the Hessian in each iterationand checking its smallest eigenvalue.
If it is negative, then non-convexity can be certiﬁed.
Again,convexity cannot be asserted by such an approach.
The DCP and the Hessian approach
[Figure]
This section gives a brief high-level overview of the DCP approach and our implementation ofthe Hessian approach.
Details are provided in the following sections.
The two approaches arealgorithmically similar.
Both use some a priori information that is propagated through expressiontrees for the function or its Hessian, respectively.
They differ in the form of the a priori informationand in the rules that are used for propagating the information.
Noteworthy, in contrast to the Hessianapproach, the DCP approach also works for non-differentiable functions.
DCP approach.
The DCP approach comprises a rule set and a set of atomic expressions, shortatoms, that are already known to be convex.
It applies to expressions that are recursively buildfrom functions, constants, and variables.
The expressions can be organized in an expression treewhose inner nodes are function symbols and whose leaves are constants and variables as shown in
Figure 1.
The set of function symbols includes the atoms.
Function symbols can be labeled as convex ,
[Equation]
for the function f .
The leaves x i are either variables or constants, all other nodes g i and f are abstract function symbols.
The root of this sub-tree is labeled by the function symbol f .
113 concave , afﬁne , and monotonously increasing (decreasing) .
Constants and variables can be labeled asnon-negative or non-positive .
The DCP rules are used to propagate label information from the leavesto the root of the expression tree, which represents the whole expression.
Therefore, if it is possibleto propagate the label convex to the root, then the expression is proven to be convex.
Hessian approach.
The Hessian of a twice differentiable function f : R n → R is a quadratic form
H ( f ) , that is, for v ∈ R n the Hessian evaluates as v (cid:62) H ( f ) v .
The function f is convex if H ( f ) ispositive semideﬁnite (psd).
Here, we assume that we can compute H ( f ) in vectorized form, that is,in standard matrix language that does not make use of explicit indices.
Our implementation works fora formally deﬁned language for representing multivariate functions (see the supplemental material)that allows to compute their Hessians in normalized vectorized form.
By representing the Hessian byan expression tree, the Hessian approach for computing convexity certiﬁcates propagates positivityand type information from the leaves of the tree to the root by using a rule set that we introduce in
Sections 4 and 5.
Rule sets
In this section we show that for a twice differentiable function the DCP rule set is implied by a set ofpositivity rules for the Hessian of the function.
However, this is not enough to show that the DCPapproach is implied by a positivity calculus, since this also requires that the atoms used in the DCP approach can be certiﬁed as convex by the positivity calculus.
We discuss CVX’s DCP atoms in
Section 6.
The DCP rule set by Grant et al.
[2006b] comprises the following rules for functions on R n :
(cid:80) mi
=
α
i f i is convex if either α i ≥ 0 and f i is convex, or α i ≤ 0 and f i is concave, for all
i ∈ [ m ] .
f ( g 1 , g 2 , . . .
, g m ) is convex if f is convex and for each g
i one of the following conditions
holds:
(a) f is increasing in argument i and g i is convex.
(b) f is decreasing in argument i and g i is concave.
(c) g i is afﬁne or constant.
f ( g 1 , g 2 , . . .
, g m ) is concave if f is concave and for each g
i one of the following conditions
holds
(a) f is increasing in argument i and g i is concave.(b)
f is decreasing in argument i and g i is convex.
(c) g i is afﬁne or constant.4.
f ( g 1 , g 2 , . . .
, g m ) is afﬁne if f is afﬁne and each function
g
i is afﬁne.
Note that products of functions cannot be treated within the DCP framework, that is, expressions ofthe form f 1 · f 2 .
Even when f 1 and f 2 are known to be afﬁne, nothing can be said about the product.
Here, we only discuss certifying convexity, since concavity of a function f can be certiﬁed by theconvexity of − f .
Hence, in the following, we do not consider the DCP Rule 3.
We show that the
DCP Rules 1 and 2 for twice differentiable functions are implied by positivity rules for their Hessians.
DCP Rule 4 that asserts that h is afﬁne, which is a stronger property, can be addressed by addingthe rules 0 · M = M · 0 = 0 and M +
=
M to the positivity rules.
More speciﬁcally, we have thefollowing proposition.
Proposition 1.
For twice differentiable functions, the DCP rule set is implied by the followingpositivity rules for ( n × n ) -matrices:
If c ≥ 0 and M (cid:23) 0 , then c · M (cid:23) 0 .2.
If c ≤ 0 and M (cid:22) 0 , then c · M (cid:23) 0 .3.
If M 1 (cid:23) 0 and M 2 (cid:23) 0 , then M 1 + M 2 (cid:23) 0 .
If M (cid:23) 0 and A is an arbitrary ( m × n ) matrix, then AMA (cid:62) (cid:23) 0 .
Proof.
We exploit that a twice differentiable function h is convex if its Hessian H ( h ) is positivesemideﬁnite (psd) and that it is concave if its Hessian H ( h ) is negative semideﬁnite (nsd).
For DCP
Rule 1, we observe that H ( f i ) (cid:23) 0 if f i is convex, and H ( f i ) (cid:22)
if f i is concave.
Hence, by
Positivity Rules 1 and 2,
[Equation]
for all i ∈ [ m ] , and then by Positivity Rule 3,
[Equation]
which certiﬁes the convexity of (cid:80) mi
=
α
i f i .
For the remaining DCP Rules 2 and 4, let h be recursively deﬁned as h
= f ( g 1 , g 2 , . . .
, g m ) .
Thegradient of h at x can be written as
[Equation]
g m ) · ∇ ( g i ) and its Hessian is given as
H
[Equation]
i,j
[Equation]
H
[Equation]
f
[Equation]
g
[Equation]
, g
[Equation]
g m ) · ∇
[Equation]
g
[Equation]
∇
[Equation]
g
[Equation]
i
[Equation]
∇ ( f ) i ( g 1 , g 2 , . . . , g m ) · H ( g i ) .
For DCP Rule 2a, we ﬁrst argue separately that both sums for H ( h ) are psd.
The ﬁrst sum can bewritten as GH ( f ) G (cid:62) , where the columns of G are the gradients ∇ ( g i ) .
Since the convexity of fimplies that H ( f ) (cid:23) 0 it follows from Positivity Rule 4 that the ﬁrst sum for H ( h ) is psd.
For thesecond sum, we use that f is increasing in its i -th argument, which implies that ∇ ( f )
i ≥ 0 , and thatthe g i are convex, which implies that H ( g i ) (cid:23) 0 .
Hence, by Positivity Rule 1 all terms in the secondsum are psd, which together with Positivity Rule 3 implies that also the second sum for H ( h ) is psd.
Thus, we can conclude from Positivity Rule 3 that H ( h ) (cid:23) 0 , which certiﬁes the convexity of h .
Theclaims for DCP Rules 2b, 2c and 4 follow similarly.
Implementing the Hessian approach
For implementing the Hessian approach we need to compute a representation of the Hessian that isamenable to analysis.
The matrix calculus by Laue et al.
[2018] computes derivatives of vectorizedexpressions again in vectorized form, that is, in a form without explicit indices.
Here, we employcomputational graphs for the expressions of second derivatives, that is, Hessians, computed by thematrix calculus.
We normalize these computational graphs into expression DAGs (directed acyclicgraphs) that contain each subexpression exactly once, see Figures 2 and 3 for examples of normalizedexpression DAGs.
In the following, we demonstrate our implementation of the Hessian approach using illustrativeexamples before we summarize the algorithm that underlies our implementation of the Hessianapproach.
5.1 A generic example
As a ﬁrst generic example we discuss the ordinary least squares regression problem
[Equation]
where X ∈ R m × n is a data matrix, y ∈ R m is a label vector, and w ∈ R n is the parameter vector thatneeds to be optimized.
Using matrix calculus, the Hessian for the objective function of this problemcan be computed in vectorized form as 2 · X (cid:62) X .
An expression DAG for the Hessian is shown in
Figure 2.
Our implementation of the Hessian approach computes positivity information for each node ∗ ∗ (cid:62)
X
Rule Rule
[Figure]
E v a l u a ti on
194 of the DAG in a bottom-up strategy from the leaves to the root.
There are positivity rules that applyto the two multiplication nodes of the DAG, namely ﬁrst Rule 4 for the expression X (cid:62) X
and then
Rule 1 for the expression 2 · X (cid:62) X , where it is already known that X (cid:62) X is psd.
Let us compare the Hessian approach to the DCP approach for certifying the convexity of the ordinaryleast squares problem.
The DAG for the ordinary least squares objective function is shown in Figure 3.
The DCP approach traverses this DAG in a bottom-up fashion.
Starting from the leaves, the matrixvector product Xw is afﬁne by deﬁnition and thus the corresponding multiplication node is labeledafﬁne, similarly the subtraction node for Xw − y is labeled afﬁne, since the addition/subtraction ofafﬁne functions is afﬁne again, and the transposition node in ( Xw − y )
(cid:62) is labeled afﬁne, since thetransposition preserves this property.
However, in general nothing can be said about the product ofafﬁne functions, which means that the standard DCP rules are not enough to label the root of the
DAG.
Typically, the problem is dealt with by adding a new atomic function to the DCP atoms thatsquares its input.
∗ (cid:62) − ∗
X w y label
[Figure]
afﬁne label : afﬁne label : ? label
[Figure]
afﬁne
Variable
E v a l u a ti on
5.2 Propagating information about subexpressions
A second instructive example is the univariate function f ( x ) = x log( x ) that neither the Hessiannor the DCP approach can certify as convex without information beyond the rules.
However, theinformation required by the Hessian approach is easier to provide on the language level.
The Hessianof f ( x ) is the expression 1 /x , which, in general, can be positive, negative or even undeﬁned at 0 .
However, we already know that x ∈ (0 , ∞ ) , because the domain of the logarithm are the positivereals.
This information is enough to decide the positivity of the Hessian.
Hence, we can exploitpositivity information about the elementary functions that are supported by our formal language.
Here are some additional rules
[Equation]
and exp( x ) , arccos( x ) , log(1 + x ) , (cid:107) x (cid:107) ≥ 0 .
Let us compare this again to the DCP approach that directly works on the expression x log( x ) .
Here,the problem is at the root node that multiplies the afﬁne function x with the concave function log( x ) .
Since, in general, nothing can be said about the product of an afﬁne function and a concave function,it is not possible to certify the convexity of x log( x ) from the DCP rules.
Indeed, CVX adds theentropy function − x (cid:62) log( x ) as an atom that has been certiﬁed as concave by other means likeanalyzing its second order derivative.
Another instructive example is the univariate function log(1 + exp( x )) whose Hessian is
[Equation]
The Hessian can be certiﬁed positive by propagating the known positivity information for theexponential function through the normalized expression DAG for the Hessian, as can be seen in
Figure 4.
CVX adds this function as an atom named logistic , probably because the derivative ofthis function is the logistic function 1 / (1 + exp( − x )) .
Note that the naming problem does not existin the Hessian approach.
A fourth example is the power function x p .
The Hessian of the power function is p ( p − 1) x
p − 2 .
Sincep ( p − 1) ≥ 0 for p ≥ 1 and p ≤ 0 we can certify the convexity of the power function from its Hessianif p = 2 n, n ∈ N , or (cid:0) ( p ≥ 1) ∨ ( p < 0) (cid:1) ∧ ( x > 0) , or (cid:0) ( p = 1) ∨ ( p = 2 k, k ∈ Z ) (cid:1) ∧ ( x < 0) .
Furthermore, we can certify the convexity of − x p from its Hessian if (0 < p < 1) ∧ ( x ≥ 0) .
Thatis, the Hessian approach can certify the convexity of power functions from constraints on x andinformation about the power parameter p , whereas the DCP approach needs atoms for the differentcases.
In general, the Hessian approach can exploit derived and user provided information about variables,parameters, and, more generally, subexpressions.
However, the next example shows that this is notenough.
∗ − / + exp x ∈ (0 , ∞ ) ∈ [max { 1 , exp( x ) } , ∞ ) ∈ [0 , 1] ∈ [0 , 1] ∈ [0 , 1] ∈ [0 , 1] ∈ [0 , 1] ∈ [0 , 1]
5.3 A psd expression template
Our ﬁfth example is the CVX atom log_sum_exp that computes log (cid:0) sum ( exp ( x )) (cid:1) .
Its Hessian isgiven, again in vectorized form, as
[Equation]
Both the ﬁrst and the second term are readily certiﬁed as positive, but their difference is not, because in general, nothing can be said about the difference of two psd matrices.
However, we will show that the Hessian matches the psd expression template from Proposition 2, see also the Figure to the right.
The template can be used to certify many expressions as convex.
Whenever a subexpression of a larger expression DAG matches the template, we can label the subexpression DAG as psd and propagate this information within the larger DAG.
Indeed, adding this template to our rule set is powerful enough to cover all differentiable atoms with vector input, and thus by Proposition 1 all differentiable functions with vector input, that can be certiﬁed as convex by the DCP implementation within CVX.
− diag / (cid:62) ∗ (cid:12) (cid:12) z
y sum label : psd
Proposition 2.
For y ∈ R n and z ∈ R n ≥ 0 , all matrices of the following form are psd:
[Equation]
Proof.
We have to show that v (cid:62) Av ≥ 0 for all v ∈ R n for the template matrix A .
It turns out thatv (cid:62)
Av is the variance of some random variable.
We compute that
[Equation]
where var ( V y ) is the variance of the random variable V y that takes the value
y i v i with probabilityz
i / sum ( z ) for i ∈ [ n ] .
Note that the Hessian of the CVX atom log_sum_exp matches the template if z in the template isinstantiated by exp ( x ) / sum ( exp(x) ) and y by vector (1) , which is illustrated in Figure 5.
5.4 Summary of the Hessian approach
Our algorithm that implements the Hessian approach combines the individual steps mentioned above.
First, the Hessian of a given expression is computed as a normalized expression DAG.
Then, the algorithm processes the normalized expression DAG from the leaves to root and annotates the nodeswith positivity and type information, that is, scalar , vector , or matrix .
The type of inner nodes, that is,operator nodes, can always be inferred from the types of their children.
All leaves either represent variables or parameters of known type.
They can be annotated with userprovided information like psd or nsd for matrices and range of allowed values for scalars and vectors,like for instance x ≥ 0 , z ∈ R n ≥ 0 or p ≥ 1 as in the examples before.
At inner nodes that representscalar or vector subexpressions allowed ranges are computed from the corresponding informationat their children.
For inner nodes that represent matrices the rules from Proposition 1 are checkedand the nodes labeled accordingly.
While it is enough for Rules 1 to 3 to check only the childrenof the nodes that either multiply a scalar and a matrix (Rules 1 and 2) or add to matrices (Rule 3),checking Rule 4 at a node that multiplies two matrices requires looking at descendants up to depththree.
Similarly, checking the template from Proposition 2 at a node that subtracts one matrix fromanother one, descendants up to depth ﬁve need to be checked.
If the template is matched, then thenode is labeled psd.
Finally, if the root node has been labeled psd, then the whole expression hasbeen certiﬁed positive and the labeling of the normalized expression DAG provides a certiﬁcate ofpositivity.
The algorithm is illustrated in Figure 5.
Since the psd expression template and the rules from Proposition 1 only require to check substructuresof constant depth (up to depth ﬁve for the template) the running time of the algorithm is linear in thesize of the expression DAG of the Hessian.
[Equation]
Rule Rule
Rule
TemplateMatching
Rule
[Figure]
on the Hessian approach on the example of the Hessian of the log_sum_exp function.
Atoms of CVX’s implementation of the DCP approach
In addition to the rule set, the DCP approach requires functions (atoms) that are already known to beconvex.
Here, we show that all twice differentiable atoms with vector input that can be certiﬁed asconvex by the DCP implementation within CVX can also be certiﬁed by our implementation of the
Hessian approach.
In the next section we provide examples of convex function that are not certiﬁedas convex by CVX, but can be certiﬁed by our Hessian approach.
Among CVX’s DCP atoms are standard convex univariate functions like exp , neg_log , neg_sqrt ,and square that we discuss in the supplemental material.
Multivariate DCP atoms in CVX aresum ( x ) , that is, the function (cid:80) ni
=1 x i , and quadratic forms
quad_form ( x, A ) , that is, x (cid:62) Ax for apsd matrix A .
Both functions are readily certiﬁed as convex by their Hessians.
A more interestingatom is inv_prod ( x ) that computes (cid:0) (cid:81) ni
=1 x i (cid:1) − 1 .
In vectorized notation this function can bewritten as f ( x ) = 1 / exp (cid:0) sum ( log ( x )) (cid:1) and its vectorized Hessian is given asf ( x ) · ( vector (1) (cid:11) x )( vector (1) (cid:11) x ) (cid:62) + diag ( vector ( f ( x )) (cid:11) ( x (cid:12) x )) , where vector (1) is the all ones vector, (cid:12) and (cid:11) denote elementwise multiplication and divison,respectively.
It follows from the positivity of f ( x ) and Positivity Rules 1 and 4 that the ﬁrst termin the sum for the Hessian is psd, and from the positivity of f ( x ) and the positivity of the entriesof x (cid:12) x it follows that also the second term is psd.
Hence, the Hessian is psd by Positivity Rule 3,which certiﬁes the convexity of f ( x ) .
CVX also contains four atoms (combinations of the operators sum, log, and exp) that superﬁciallylook similar to inv_prod , but cannot be certiﬁed directly from the positivity calculus.
However, forthese problems, the Hessian is the difference of two psd matrices that can be matched to the templateexpression from Proposition 2.
We have already discussed the CVX atom log_sum_exp .
The threeremaining atoms are the harmonic mean, the geometric mean, and p -norms.
Here, we only discussthe harmonic mean.
See the supplemental material for p -norms and the geometric mean.
The negative harmonic mean neg_harmonic_mean ( x ) for x ∈ R n + is deﬁned as − n/ (cid:80) ni
=1 x − 1 i .
Itcan be written in vectorized notation as
[Equation]
Its Hessian, which computes to
[Equation]
is matched by the psd expression template if both y and z are instantiated by vector (1) (cid:11) x .
Notethat 1 / sum
( z ) in the template becomes − f ( x ) = 1 / sum (cid:0) vector (1) (cid:11) x ) (cid:1) in the instantiation.
Beyond CVX’s atoms
There are two main classes of functions that cannot be treated by DCP.
The ﬁrst are products oftwo non-constant expressions such as x · exp( x ) for x ≥ 0 .
The second are compositions that arenot following DCP’s Rules 2, 3 or 4 such as neg_entr (cosh( x )) =
cosh( x ) log(cosh
( x )) , whereneg_entr ( x ) =
x log( x ) is convex but not increasing, and thus composing it with the convexfunction cosh( x ) , DCP Rule 2a does not apply here.
Both examples so far do not need the templateexpression from Proposition 2 to be certiﬁed convex by the Hessian approach.
This is different forthe multivariate function
[Equation]
which can be expressed in vectorized form as sum (cid:0) exp ( x ) (cid:1) log (cid:0) 1 + sum (cid:0) exp ( x ) (cid:1)(cid:1) .
Its Hessian isthe sum of three matriceslog
[Equation]
[Table] where the last matrix matches the template expression if y is instantiated by vector (1) and z byexp ( x ) , while the ﬁrst two matrices can be certiﬁed as psd directly from the positivity rules.
In the supplemental material, we provide more examples of differentiable functions with vector inputthat cannot be certiﬁed as convex by the DCP implementation within CVX, but can be certiﬁed byour implementation of the Hessian approach.
Conclusions
We have presented the ﬁrst generic implementation of the Hessian approach for certifying convexityand shown that it complements the well-established disciplined convex programming approach, whichalso works for non-differentiable functions.
Our implementation works on a formal language that isclose to natural problem formulations in textbooks and rich enough to express many classical machinelearning problems, including problems not found in standard libraries like scikitlearn.
An anynomizedversion of our implementation can be found at https://convcert.pythonanywhere.com .
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] These impacts are probably only very indirect.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A](b)
Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[N/A]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]