Sound and Complete Verification of Polynomial Networks
s
Abstract
Polynomial Networks (PNs) have demonstrated promising performance on faceand image recognition recently.
However, robustness of PNs is unclear and thusobtaining certificates becomes imperative for enabling their adoption in real-worldapplications.
Existing verification algorithms on ReLU neural networks (NNs)based on branch and bound (BaB) techniques cannot be trivially applied to PNverification.
In this work, we devise a new bounding method, equipped with BaBfor global convergence guarantees, called VPN.
One key insight is that we obtainmuch tighter bounds than the interval bound propagation baseline.
This enablessound and complete PN verification with empirical validation on MNIST, CIFAR10and STL10 datasets.
We believe our method has its own interest to NN verification.
Introduction
Polynomial Networks (PNs) have demonstrated promising performance across image recognitionand generation [Chrysos et al., 2021b, Chrysos and Panagakis, 2020] being state-of-the-art on largescale face recognition 1 .Unlike
the conventional Neural Networks (NNs), where non-linearitiy isintroduced with the use of activation functions [LeCun et al., 2015], PNs are able to learn nonlinear mappings without the need of activation functions by exploiting multiplicative interactions(Hadamard products).
Recent works have uncovered interesting properties of PNs, like their largermodel expressivity [Fan et al., 2021] or their spectral bias [Choraria et al., 2022].
However, onecritical issue before considering PNs for real-world applications is their robustness.
Neural networks are prone to small (often imperceptible to the human eye), but malicious perturbationsin the input data points [Szegedy et al., 2014, Goodfellow et al., 2015].
Those perturbations can havea detrimental effect on image recognition systems, e.g., as illustrated in face recognition [Goswamiet al., 2019, Zhong and Deng, 2019, Dong et al., 2019, Li et al., 2020].
Guarding against suchattacks has so far proven futile [Shafahi et al., 2019, Dou et al., 2018].
Instead, a flurry of researchhas been published on certifying robustness of NNs against this performance degradation [Katzet al., 2017, Ehlers, 2017, Tjeng et al., 2019, Bunel et al., 2020a, Wang et al., 2021, Ferrari et al.
However, most of the verification algorithms for NNs are developed for the ReLU activationfunction by exploiting its piecewise linearity property and might not trivially extend to other nonlinearactivation functions [Wang et al., 2021].
Indeed, Zhu et al.
[2022] illustrate that guarding PNs againstadversarial attacks is challenging.
Therefore, we pose the following question:
Can we obtain certifiable performance for PNs against adversarial attacks?
In this work, we answer affirmatively and provide a method for the verification of PNs.
Concretely,we take advantage of the twice-differentiable nature of PNs to build a lower bounding method based on α -convexification
[Adjiman and Floudas, 1996], which is integrated into a Branch and Boundalgorithm [Land and Doig, 1960] to guarantee completeness of our verification method.
In order touse α -convexification a lower bound α of the minimum eigenvalue of the Hessian matrix over thepossible perturbation set is needed.
We use interval bound propagation together with the theoreticalproperties of the lower bounding Hessian matrix [Adjiman et al., 1998], in order to develop analgorithm to efficiently compute α .
Our contributions can be summarized as follows: ( i )
We propose the first algorithm for the verificationof PNs.
( ii ) We thoroughly analyze the performance of our method by comparing it with a black-boxsolver and an interval bound propagation (IBP) BaB algorithm.
( iii ) We empirically show thatusing α -convexitication for lower bounding provides tighter bounds than IBP for PN verification.
Toencourage the community to improve the verification of PNs, we intend to make our source codepublic upon the acceptance of the paper.
The proposed approach can practically verify PNs and that could theoretically be applied forsound and complete verification of any twice-differentiable network.
Recent works showing twicedifferentiability of ReLU NNs almost everywhere [Yao et al., 2018] also suggest that our approachcould be used in the case of ReLU NNs, which we believe is an interesting avenue for future work.
Notation: We use the shorthand [ n ] := { 1 , 2 , . . . , n } for a positive integer n .
We use bold capital(lowercase) letters, e.g., X ( x ) for representing matrices (vectors).
The j th column of a matrix X isgiven by x : j .
The element in the i th row and j th column is given by x ij , similarly, the i th element of avector x is given by x i .
The element-wise (Hadamard) product, symbolized with ∗ , of two matrices(or vectors) in R d 1 × d 2 (or R d ) gives another matrix (or vector) in R d 1 × d 2 (or R d ).
The ℓ ∞ norm of avector x ∈ R d is given by: || x || ∞ =
max i ∈ [ d ] | x i | .
Lastly, the operators L and U give the lower andupper bounds of a scalar, vector or matrix function by IBP, see Section 3.1.
Roadmap: We provide the necessary background by introducing the PN architecture and formalizingthe Robustness Verification problem in Section 2.
Section 3 provides a sound and complete methodcalled VPN to tackle PN verification problem.
In Section 4, we discuss the related works in thecontext of our contributions.
Section 5 is devoted to experimental validation.
Additional experiments,details and proofs are deferred to the appendix.
Background
To make the paper self-contained, we introduce the PN architecture in Section 2.1 and the Robustness
Verification problem in Section 2.2.
2.1 Polynomial Networks (PNs)
Polynomial Networks (PNs) are inspired by the fact that any smooth function can be approximated viaa polynomial expansion
[Stone, 1948].
However, the number of parameters increases exponentiallywith the polynomial degree, which makes it intractable to use high degree polynomials for highdimensional data problems such as image classification where the input can be in the order of 10 5[Deng et al., 2009].
Chrysos et al.
[2021b] introduce a joint factorization of polynomial coefficientsin a low-rank manner, reducing the number of parameters to linear with the polynomial degree andallowing the expression as a Neural Network.
We briefly recap one fundamental factorization below.
Let N be the polynomial degree,
z ∈ R
d be the input vector, d , k and o be the input, hidden andoutput sizes, respectively.
The recursive equation of PNs can be expressed as:
[Equation]
where x (1) =
W ⊤ [1] z , f ( z ) =
Cx ( N ) + β and ∗ denotes the Hadamard product.
W [ n ] ∈ R d × k and
C ∈ R o × k are weight matrices, β ∈ R o is a bias vector.
A graphical representation of a third degree
PN architecture corresponding to Eq.
(1) can be found in Fig.
Further details on the factorization(as well as other factorizations) are deferred to the Appendix B.1 (Appendix B.2).
2.2 Robustness Verification
Robustness Verification [Bastani et al., 2016, Liu et al., 2021] consists of verifying that a propertyregarding the input and output of a NN is satisfied, e.g. checking whether or not a small perturbation z
W [1] ∗ +
W [2] ∗ +
W [3]
C + β f ( z ) x (1) x (2) x (3) in the input will produce a change in the network output that makes it classify the input into anotherclass.
Let f : [0 , 1] d
→ R o be a function, e.g., a NN or a PN, that classifies inputs z into a class c ,such that c = arg max f ( z ) .
We want to verify that for any input satisfying a set of constraints C in ,the output of the network will satisfy a set of output constraints C out .
That is, we want to check thefollowing logical formula is satisfied:
[Equation]
In this work we focus on adversarial robustness [Szegedy et al., 2014, Carlini and Wagner, 2017]in classification.
Assume an observation z 0 is given and let t = arg max f ( z 0 ) be the correct class,we want to check whether every input in a neighbourhood of z 0 , is classified as t .
We focus onadversarial attacks restricted to neighbourhoods defined in terms of ℓ ∞ norm, which is a popularnorm-bounded attack in the verification community.
Then, the constraint sets become:
[Equation]
In other words, we need an algorithm that given a function f , an input z 0 and an adversarial budgetϵ , checks whether Eq.
(2) is satisfied.
In the case of ReLU NNs, this has been proven to be an
NP-complete problem [Katz et al., 2017].
This can be reformulated as a mathematical optimizationproblem.
For every adversarial class γ ̸ = t = arg max f ( z 0 ) , we can solve:
[Equation]
If the solution z ∗ with v ∗ = f ( z ∗ ) t − f ( z ∗ ) γ ≤ f ( z ) t − f
( z ) γ , ∀ z ∈ C in satisfies v ∗ > 0 thenrobustness is verified for the adversarial class γ .
There are two main properties that we would like to have in a verification algorithm: soundness andcompleteness .
An algorithm is sound ( complete ) if every time it verifies (falsifies) a property, it isguaranteed to be the correct answer.
In practice, when an algorithm is guaranteed to provide the exactglobal minima of Eq.
(4), i.e., v ∗ , it is said to be sound and complete (usually referred in the literatureas simply complete [Ferrari et al., 2022]), whereas if a lower bound of it is provided ˆ v ∗ ≤ v ∗ , thealgorithm is sound but not complete .
We will not consider just complete verification, which simplyaims at looking for adversarial examples, e.g., Madry et al.
For a deeper discussion onsoundness and completeness , we refer to Liu et al.
Method
Our method, called VPN, can be categorized in the the Branch and Bound (BaB) framework [Landand Doig, 1960], a well known approach to global optimization [Horst and Tuy, 1996] and NNverification [Bunel et al., 2020a].
This kind of algorithms guarantee finding a global minima of theproblem in Eq.
(4) by recursively splitting the original feasible set into disjoint sets (branching) whereupper and lower bounds of the global minima are computed (bounding).
This mechanism can be usedto discard subsets where the global minima cannot be achieved (its lower bound is greater than theupper bound of another subset).
Our method is based on an α -BaB algorithm
[Adjiman et al., 1998], which is characterized for usingα -convexification [Adjiman and Floudas, 1996] for computing a lower bound of the global minimaof each subset.
To be specific, α -convexification aims to obtain a convex lower bounding function ofany twice-differentiable function f : R d → R such that
[Equation]
as its α -convexified version.
Let H f ( z )
= ∇ 2 zz f
( z ) be the Hessian matrix of f , f α is convexin z ∈ [ l , u ] for α ≥ max { 0 , − 12 min { λ min ( H f ( z )) :
z ∈ [ l , u ] }} .
Moreover, it holds thatf α
( z ; α, l , u ) ≤ f ( z ) , ∀ z ∈ [ l , u ] .
To make PN verification feasible, we need to study IBP for PNs and design an efficient estimate onthe α parameter, which are our main technical contributions in the algorithmic aspect.
In our case,every feasible set, starting with the input set C in (Eq.
(3)), is split by taking the widest variable intervaland dividing it in two by the middle point.
Then, the upper bound of each subproblem is givenby applying standard Projected Gradient Descent (PGD)
[Kelley, 1999] over the original objectivefunction.
This is a common approach to find adversarial examples [Madry et al., 2018], but as theobjective is non-convex, it is not sufficient for sound and complete verification.
The lower bound isgiven by applying PGD over the α -convexified objective g α , as it is convex, PGD converges to theglobal minima and a lower bound of the original objective.
The α parameter is computed only onceper verification problem.
Further details on the algorithm and the proof of convergence of Eq.
(4)exist in Appendix C.
In the sequel, we detail our method to compute a lower bound on the minimum eigenvalue of the Hessian matrix into three main components: interval propagation, lower bounding Hessians, and fastestimation on such lower bounding via power method.
3.1 Interval Bound Propagation through a PN
Interval bound propagation (IBP) is a key ingredient of our verification algorithm.
Suppose we havean input set defined by an ℓ ∞ -norm ball like in Eq.
This set can be represented as a vector ofintervals ˜
z
=
([ l 1 , u 1 ] ⊤ , [ l 2 , u 2 ] ⊤ , · · · , [ l d , u d ] ⊤ ) =
[ l , u ] ∈ R d × 2 , where [ l i , u i ] are the lower andupper bound for the i th coordinate.
Let L and U be the lower and upper bound IBP operators.
Giventhis input set, we would like to obtain bounds on the output of the network ( f ( z ) i ) , the gradient( ∇ z f ( z ) i ) , and the Hessian ( ∇ 2 zz f
( z ) i ) for any z ∈ ˜ z .
The operators L ( g ( z )) and U ( g ( z )) of anyfunction g : R d → R satisfy:
[Equation]
We will define these upper and lower bound operators in terms of the operations present in a PN.
Using interval propagation [Moore et al., 2009], we can define:
Identity
[Equation]
linear mapping
[Equation]
multiplication
[Equation]
where w + i = max { 0 , w i } and w − i = min { 0 , w i } , h i ( z ) is any real-valued function of z and | · | isthe set cardinality.
Note that the set S is equivalent to:
[Equation]
With these basic operations one can define bounds on any intermediate output, gradient or Hessian ofa PN, e.g., the lower bound on the recursive formula from Eq.
(1) can be expressed as:
[Equation]
which only consists on a linear mapping and a multiplication of intervals.
We extend the upper andlower bound ( L ( · ) and U ( · ) ) operators to also work on vectors and matrices by applying them atevery position of the vector or matrix:
[Equation]
) for g ( z ) ∈ R m and G ( z ) ∈ R m 1 × m 2 .
One can directly use IBP to obtain bounds on the verificationobjective from Eq.
(4) with a single forward pass of the bounds through the network and obtaining
L ( g ( z ))
=
L ( f ( z ) t ) −U ( f ( z ) γ ) .
IBP is a common practice in NN verification to obtain fast bounds[Wang et al., 2018a].
3.2 Lower bound of the minimum eigenvalue of the Hessian
Here we describe our method to compute a lower bound on the minimum eigenvalue of the Hessianmatrix in the feasible set.
Before deriving the lower bound, we need the first and second order partialderivatives of PNs.
Let g ( z ) =
f ( z ) t − f ( z ) a be the objective function for t = arg max f ( z 0 ) and any a ̸ = t .
In orderto compute the parameter α for performing α -convexification, we need to know the structure of ourobjective function.
In this section we compute the first and second order partial derivatives of the PN.
The gradient and Hessian matrices of the objective function (see Eq.
(4)) are easily found to be:
[Equation]
we now define the gradients ∇ z x ( n ) i and Hessians ∇ 2 zz x ( n ) i of Eq.
(1) in a recursive way:
[Equation]
with ∇ z x (1) i =
w [1]: i and ∇ 2 zz x
(1) i =
d × d , being
d × d a d × d matrix with 0 in every position.
In the next, we are ready to compute a lower bound on the minimum eigenvalue of the Hessian matrixin the feasible set.
Firstly, for any z ∈ [ l , u ] and any polynomial degree N , we can express the set of possible Hessians
H =
{ H g ( z ) :
z ∈ [ l , u ] } as an interval matrix.
An interval matrix is a matrix [ M ] ∈ R d × d × 2where every position [ m ] ij =
[ L ( m ij ) , U ( m ij )] is an interval.
Therefore, if H g ( z ) is boundedfor z ∈ [ l , u ] , then we can represent H
= { H g ( z ) :
H g ( z ) ∈
[ M ] } =
{ H g ( z ) : L ( m ij ) ≤
H g ( z ) ij ≤ U ( m ij ) , ∀ i, j ∈ [ d ] } .
For every set of Hessians, we can define the lower bounding Hessian L H .
Described in Adjiman et al.[1998], This matrix satisfies that λ min ( L H ) ≤ λ min ( H g ( z )) ,
∀ H g ( z ) ∈ H , z ∈ [ l , u ] .
Let L ( M )and U ( M ) , the lower bounding Hessian is defined as follows:
[Equation]
where 1 is a vector of ones and diag ( v ) is a diagonal matrix with the vector v in the diagonal.
Then, we can obtain the spectral radius ρ ( L H ) with a power method.
As the spectral radius satisfiesρ ( L H ) ≥ | λ i ( L H ) | , ∀ i ∈ [ d ] , the following inequality holds: [Equation]
[Equation]
allowing us to use α = ρ ( L H )
[Equation]
u ] [Equation]
3.3 Efficient power method for spectral radius computation of the lower bounding Hessian
By using interval propagation, one can easily compute sound lower and upper bounds on each positionof the Hessian matrix, compute the lower bounding Hessian and perform a power method with it toobtain the spectral radius ρ .
However, this method would not scale well to high dimensional scenarios.
For instance, in the STL10 case, with 96 × 96 RGB images ( d = 96 · 96 · 3 = 27 , 648 ) our Hessianmatrix would require in the order of O ( d 2 )
=
O (10 9 ) real numbers to be stored.
This makes itintractable to perform a power method over such an humongous matrix, or even to compute the lowerbounding Hessian.
Alternatively, we take advantage of the low rank decomposition characterizing
PNs to efficiently perform a power method over the lower bounding Hessian.
Standard power method for spectral radius computation Given any squared and real valuedmatrix M ∈ R d × d and an initial vector v 0 ∈ R d that is not an eigenvector of M , the sequence:
[Equation]
converges to the eigenvector with the largest eigenvalue in absolute value, i.e. the eigenvectorwhere the spectral norm is attained, being the spectral norm ρ ( M ) =
|| M ( Mv n − 1 ) ||
[Mises and
Pollaczek-Geiringer, 1929].
Power method over lower bounding Hessian of PNs
We can employ IBP (Section 3.1) in order to obtain an expression of the lower bounding Hessian( L H ) and evaluate Eq.
(15) as:
[Equation]
Applying IBP on Eq.
(10) we obtain:
[Equation]
We can recursively evaluate L ( ∇ 2 zz x ( n ) i ) v
and U ( ∇ 2 zz x ( n ) i ) v efficiently as these matrices can beexpressed as a sum of rank- 1 matrices:
Proposition 1.
Let δ ∈
[ L ( δ ) , U ( δ )] be a real-valued weight, the matrix-vector products L ( δ ·∇ 2 zz x ( n ) i ) v
and U ( δ · ∇ 2 zz x ( n ) i ) v can be evaluated as:
[Equation]
[Equation]
[Equation]
where δ ′ ∈
[ L ( δ ) , U ( δ )] · [ L ( w ⊤ [ n ]: i z
+ 1) , U ( w ⊤ [ n ]: i z + 1)] and vectors L ( δ · ∇ z x ( n − 1) i ) and U ( δ ·∇ z x ( n − 1) i ) can be obtained through IBP on Eq.
Lastly, by applying recursively Proposition 1 from n = N to n = 1 , starting with δ = 1 , we cansubstitute the results on Eq.
(17) and then on Eq.
(16) to efficiently evaluate a step of the powermethod (Eq.
(15)) without needing to store the lower bounding Hessian matrix or needing to performexpensive matrix-vector products.
Overall, our lower bounding method consists in computing a valid value of α that satisfies that theα -convexified
objective
g α is convex, following Eqs.
(4) and (5).
In particular, we use α = ρ ( L H ).ρ
( L H ) is computed via a power method, where the main operation L H v is evaluated without theneed to compute or store the L H matrix.
Provided this valid α , we perform PGD over g α and thisprovides a lower bound of the global minima of Eq.  Related Work
In this section, we give an overview of neural network verification and polynomial networks, that arecentered around our target in this work.
4.1 Neural Network Verification
Early works on sound and complete NN verification were based on Mixed Integer Linear Programming(MILP) and Satisfiability Modulo Theory (SMT) solvers [Katz et al., 2017, Ehlers, 2017, Bastaniet al., 2016, Tjeng et al., 2019] and were limited to both small datasets and networks. The utilization of custom BaB algorithms enabled verification to scale to datasets and networks thatare closer to those used in practice. Bunel et al.
[2020a] review earlier methods like Katz et al.
[2017]and show they can be formulated as BaB algorithms.
BaDNB [Palma et al., 2021] proposes a novelbranching strategy called Filtered Smart Branching and uses the Lagrangian decomposition-basedbounding algorithm proposed in Bunel et al.
[2020b].
β -CROWN
[Wang et al., 2021] proposes abound propagation based algorithm.
MN-BaB [Ferrari et al., 2022] proposes a cost adjusted branchingstrategy and leverages multi-neuron relaxations and a GPU-based solver for bounds computing.
Ourwork centers on the bounding algorithm by proposing a general convex lowerbound adapted to PNs.
BaB algorithms for ReLU networks focus their branching strategies on the activity of ReLU neurons.
This has been observed to work better than input set branching for ReLU networks [Bunel et al.,2020a].
Similarly to our method, Anderson et al.
[2019], Wang et al.
[2018a], Royo et al.
[2019] useinput set branching strategies.
4.2 Polynomial Networks
First works have been focused on developing the foundations and showcasing the performance of
PNs in different tasks [Chrysos et al., 2021b, Chrysos and Panagakis, 2020].
Also, in Chrysos et al.[2021a], PN classifiers are formulated in a common framework where other previous methods like
Wang et al.
[2018b] can be framed.
Lately, more emphasis has been put onto proving theoreticalproperties of PNs [Fan et al., 2021, Choraria et al., 2022].
In Zhu et al.
[2022], they derive Lipschitzconstant and complexity bounds for two PN decompositions in terms of the l ∞ and l 2 norms.
Theyalso analyze robustness of PNs against PGD adversarial attacks by measuring percentage of imageswhere PGD fails to find an adversarial example, which is a complete but not sound verificationmethod.
Our verification method is sound and complete.
Experiments
In this section we show the efficiency of our method by comparing against a simple Black-box solver.
Tightness of bounds is also analyzed in comparison with IBP.
Finally, a study of the performance ofour method in different scenarios is performed.
Unless otherwise specified, every network is trainedfor 100 epochs with Stochastic Gradiend Descent (SGD), with a learning rate of 0 .
001 , which isdivided by 10 at epochs [40 , 60 , 80] , momentum 0 .
9 , weight decay 5 · 10 − 5 and batch size 128 .
Wethoroughly evaluate our method over the popular image classification datasets MNIST [LeCun et al.
,1998], CIFAR10 [Krizhevsky et al., 2014] and {+STL
+} [Coates et al., 2011].
Every experiment is doneover the first 1000 images of the test dataset, this is a common practice in verification [Singh et al.
For images that are correctly classified by the network, we sequentially verify robustnessagainst the remaining classes in decreasing order of network output.
Each verification problem isgiven a maximum execution time of 60 seconds, we include experiments with different time limitsin Appendix A. Note that the execution time can be longer as execution is cut in an asynchronousway, i.e., after we finish the iteration of the BaB algorithm where the time limit is reached.
All of ourexperiments were run on a single-GPU machine. [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
7 Degree ϵ = 0 .
α -convexification bounds are significantly tighter than IBP for small ϵ values and all PN degrees from 2 to 7.
5.1 Comparison with Black-box solver
In this experiment, we compare the performance of our BaB verification algorithm with the Blackbox solver Gurobi [Gurobi Optimization, LLC, 2022].
Gurobi can globally solve Quadratically
Constrained Quadratic Programs whether they are convex or not.
As this solver cannot extend tohigher degree polynomial functions, we train 2 nd degree PNs with hidden size k = 16 to compare theverification time of our method with Gurobi.
In order to do so, we express the verification objective asa quadratic form g
( z ) =
f ( z ) t − f ( z ) a = z ⊤
Qz + q ⊤ z + c this together with the input constraintsz ∈ [ l , u ] is fed to Gurobi and optimized until convergence.
The black-box solver approach neither scales to higher dimensional inputs nor to higher polynomialdegrees.
With this approach we need O ( d 2 ) memory to store the quadratic form, which makes itunfeasible for datasets with higher resolution images than CIFAR10.
On the contrary, as seen in
Table 1, our approach does not need so much memory and can scale to datasets with larger input sizeslike STL10.
5.2 Comparison with IBP
In this experiment we compare the tightness of the lower bounds provided by IBP and αconvexification and their effectiveness when employed for verification.
This is done by executing oneupper bounding step with PGD and one lower bounding step for IBP and α -convexification methodsover the initial feasible set provided by ϵ (see Eq.
We compare the average of the distance fromeach lower bound to the PGD upper bound over the first 1000 images of the MNIST dataset for PNswith hidden size k = 25 and degrees ranging from 2 to 7 .
We also evaluate verified accuracy of 2 nd(PN_Conv2) and 4 th (PN_Conv4) PNs with both bounding methods and a maximum time of 120seconds, for details on the architecture of these networks, we refer to Appendix A. When using IBP, the objective.
Acc.% is the clean accuracy of the network, Ver.% is the verified accuracy and U.B. its upper bound. When using α -convexification bounds we get verified accuracies really close to the upper bound, while when using IBP verified accuracy is 0 for every network- ϵ pair, which makes it unsuitable for PN verification. [Table]
we get a much looser lower bound than with α -convexification, see Fig.
Only for high-degree,high- ϵ combinations IBP lower bounds are closer to the PGD upper bound.
In practice, this is nota problem for verification, as for epsilons in the order of 0 .
1 , it is really easy to find adversarialexamples with PGD and there will be no accuracy left to verify.
[-7 Degree 10 −10 −
-]
[-7 Degree ϵ = 
-]
The looseness of the IBP lower bound is confirmed when comparing the verified accuracy with IBPand α -convexification, see Table 2.
With the latter, we are able to verify the accuracy of 2 nd and 4 thorder PNs almost exactly (almost no gap between the verified accuracy and its upper bound) in everystudied dataset, while with the former, we are not able to verify robustness for a single image in anynetwork- ϵ pair, confirming the fact that IBP cannot be used for PN verification.
Conclusion
We propose a method to verify polynomial networks (PNs).
Our method, which can be categorizedas a α -BaB global optimization algorithm, is a novel approach to verification of PNs.
We believe canbe extended to cover other twice-differentiable networks in the future.
We exhibit that our methodoutperforms existing methods, such as black-box solvers and IBP.
Our method enables verificationin datasets such as STL10, which includes RGB images of 96 × 96 resolution.
This is larger thanthe images typically used in previous verification methods.
Our method can further encourage thecommunity to extend verification to a broader class of functions as well as conduct experiments indatasets of higher resolution.
Limitations : As discussed in Appendix A.2, our verification method does not scale to high-degree PNs. Even though we can verify high-accuracy PNs (see Table 2), we are still far from verifyingthe top performing deep PNs studied in Chrysos et al. [2021b].
Another problem that we share with
ReLU NN verifiers is the scalability to networks with larger input size [Wang et al., 2021].
{+In thiswork we are able to verify networks trained in STL
+}
[Coates et al., 2011], but these networks areshallow, yet their verification still takes a long time, see Table 2.
Societal impact : The performance on standard image classification benchmarks has increasedsubstantially the last few years, owing to the success of neural networks.
Their success enablestheir adoption in tackling real-world problems.
However, robustness and trustworthiness of neuralnetworks is of critical importance before their adoption in real-world applications.
Our method is averifier that focuses on polynomial networks and enables the complete verification of PNs.
Therefore,we expect that by using the proposed method, certain properties of the robustness could be verified ina principled way.
We expect this to have a predominantly positive societal impact as either a tool forpre-trained models or tool for certifying models as part of their debugging.
However, it can also beused as a tool to find weaknesses of pretrained PNs by adversarial agents.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] In Section 6 we discuss thelimitations.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our
method aims at verifying PNs robustness to adversarial attacks, which has no direct
negative societal impact, see Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] The complete proofof propositions, lemmas and theorems is present in the appendix.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We use publiclyavailable datasets and our code and instructions are available in the supplementary
material. (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] We follow the standard practice in the literature forreporting the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] A single GPU was used in eachexperiment.
The single GPU is part of our internal cluster.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A] Standard image-based datasets are
used in this work.
All of them are publicly available and we cite their respecting papers,
where the licenses are mentioned.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]