Sound and Complete Verification of Polynomial Networks
s
Abstract
Polynomial Networks (PNs) have demonstrated promising performance on faceand image recognition recently.
However, robustness of PNs is unclear and thusobtaining certificates becomes imperative for enabling their adoption in real-worldapplications.
Existing verification algorithms on ReLU neural networks (NNs)based on branch and bound (BaB) techniques cannot be trivially applied to PNverification.
In this work, we devise a new bounding method, equipped with BaBfor global convergence guarantees, called VPN.
One key insight is that we obtainmuch tighter bounds than the interval bound propagation baseline.
This enablessound and complete PN verification with empirical validation on MNIST, CIFAR10and STL10 datasets.
We believe our method has its own interest to NN verification.
Introduction
Polynomial Networks (PNs) have demonstrated promising performance across image recognitionand generation [Chrysos et al., 2021b, Chrysos and Panagakis, 2020] being state-of-the-art on largescale face recognition 1 .Unlike
the conventional Neural Networks (NNs), where non-linearitiy isintroduced with the use of activation functions [LeCun et al., 2015], PNs are able to learn nonlinear mappings without the need of activation functions by exploiting multiplicative interactions(Hadamard products).
Recent works have uncovered interesting properties of PNs, like their largermodel expressivity [Fan et al., 2021] or their spectral bias [Choraria et al., 2022].
However, onecritical issue before considering PNs for real-world applications is their robustness.
Neural networks are prone to small (often imperceptible to the human eye), but malicious perturbationsin the input data points [Szegedy et al., 2014, Goodfellow et al., 2015].
Those perturbations can havea detrimental effect on image recognition systems, e.g., as illustrated in face recognition [Goswamiet al., 2019, Zhong and Deng, 2019, Dong et al., 2019, Li et al., 2020].
Guarding against suchattacks has so far proven futile [Shafahi et al., 2019, Dou et al., 2018].
Instead, a flurry of researchhas been published on certifying robustness of NNs against this performance degradation [Katzet al., 2017, Ehlers, 2017, Tjeng et al., 2019, Bunel et al., 2020a, Wang et al., 2021, Ferrari et al.
However, most of the verification algorithms for NNs are developed for the ReLU activationfunction by exploiting its piecewise linearity property and might not trivially extend to other nonlinearactivation functions [Wang et al., 2021].
Indeed, Zhu et al.
[2022] illustrate that guarding PNs againstadversarial attacks is challenging.
Therefore, we pose the following question:
Can we obtain certifiable performance for PNs against adversarial attacks?
In this work, we answer affirmatively and provide a method for the verification of PNs.
Concretely,we take advantage of the twice-differentiable nature of PNs to build a lower bounding method based on α -convexification
[Adjiman and Floudas, 1996], which is integrated into a Branch and Boundalgorithm [Land and Doig, 1960] to guarantee completeness of our verification method.
In order touse α -convexification a lower bound α of the minimum eigenvalue of the Hessian matrix over thepossible perturbation set is needed.
We use interval bound propagation together with the theoreticalproperties of the lower bounding Hessian matrix [Adjiman et al., 1998], in order to develop analgorithm to efficiently compute α .
Our contributions can be summarized as follows: ( i )
We propose the first algorithm for the verificationof PNs.
( ii ) We thoroughly analyze the performance of our method by comparing it with a black-boxsolver and an interval bound propagation (IBP) BaB algorithm.
( iii ) We empirically show thatusing α -convexitication for lower bounding provides tighter bounds than IBP for PN verification.
Toencourage the community to improve the verification of PNs, we intend to make our source codepublic upon the acceptance of the paper.
The proposed approach can practically verify PNs and that could theoretically be applied forsound and complete verification of any twice-differentiable network.
Recent works showing twicedifferentiability of ReLU NNs almost everywhere [Yao et al., 2018] also suggest that our approachcould be used in the case of ReLU NNs, which we believe is an interesting avenue for future work.
Notation: We use the shorthand [ n ] := { 1 , 2 , . . . , n } for a positive integer n .
We use bold capital(lowercase) letters, e.g., X ( x ) for representing matrices (vectors).
The j th column of a matrix X isgiven by x : j .
The element in the i th row and j th column is given by x ij , similarly, the i th element of avector x is given by x i .
The element-wise (Hadamard) product, symbolized with ∗ , of two matrices(or vectors) in R d 1 × d 2 (or R d ) gives another matrix (or vector) in R d 1 × d 2 (or R d ).
The ℓ ∞ norm of avector x ∈ R d is given by: || x || ∞ =
max i ∈ [ d ] | x i | .
Lastly, the operators L and U give the lower andupper bounds of a scalar, vector or matrix function by IBP, see Section 3.1.
Roadmap: We provide the necessary background by introducing the PN architecture and formalizingthe Robustness Verification problem in Section 2.
Section 3 provides a sound and complete methodcalled VPN to tackle PN verification problem.
In Appendix B, we discuss the related works in thecontext of our contributions.
Section 4 is devoted to experimental validation.
Additional experiments,details and proofs are deferred to the appendix.
Background
To make the paper self-contained, we introduce the PN architecture in Section 2.1 and the Robustness
Verification problem in Section 2.2.
2.1 Polynomial Networks (PNs)
Polynomial Networks (PNs) are inspired by the fact that any smooth function can be approximated viaa polynomial expansion
[Stone, 1948].
However, the number of parameters increases exponentiallywith the polynomial degree, which makes it intractable to use high degree polynomials for highdimensional data problems such as image classification where the input can be in the order of 10 5[Deng et al., 2009].
Chrysos et al.
[2021b] introduce a joint factorization of polynomial coefficientsin a low-rank manner, reducing the number of parameters to linear with the polynomial degree andallowing the expression as a Neural Network.
We briefly recap one fundamental factorization below.
Let N be the polynomial degree,
z ∈ R
d be the input vector, d , k and o be the input, hidden andoutput sizes, respectively.
The recursive equation of PNs can be expressed as:
[Equation]
where x (1) =
W ⊤ [1] z , f ( z ) =
Cx ( N ) + β and ∗ denotes the Hadamard product.
W [ n ] ∈ R d × k and
C ∈ R o × k are weight matrices, β ∈ R o is a bias vector.
A graphical representation of a third degree
PN architecture corresponding to Eq.
(1) can be found in Fig.
Further details on the factorization(as well as other factorizations) are deferred to the Appendix C.1 (Appendix C.2).
2.2 Robustness Verification
Robustness Verification [Bastani et al., 2016, Liu et al., 2021] consists of verifying that a propertyregarding the input and output of a NN is satisfied, e.g. checking whether or not a small perturbation z
W [1] ∗ +
W [2] ∗ +
W [3]
C + β f ( z ) x (1) x (2) x (3) in the input will produce a change in the network output that makes it classify the input into anotherclass.
Let f : [0 , 1] d
→ R o be a function, e.g., a NN or a PN, that classifies inputs z into a class c ,such that c = arg max f ( z ) .
We want to verify that for any input satisfying a set of constraints C in ,the output of the network will satisfy a set of output constraints C out .
That is, we want to check thefollowing logical formula is satisfied:
[Equation]
In this work we focus on adversarial robustness [Szegedy et al., 2014, Carlini and Wagner, 2017]in classification.
Assume an observation z 0 is given and let t = arg max f ( z 0 ) be the correct class,we want to check whether every input in a neighbourhood of z 0 , is classified as t .
We focus onadversarial attacks restricted to neighbourhoods defined in terms of ℓ ∞ norm, which is a popularnorm-bounded attack in the verification community.
Then, the constraint sets become:
[Equation]
In other words, we need an algorithm that given a function f , an input z 0 and an adversarial budgetϵ , checks whether Eq.
(2) is satisfied.
In the case of ReLU NNs, this has been proven to be an
NP-complete problem [Katz et al., 2017].
This can be reformulated as a mathematical optimizationproblem.
For every adversarial class γ ̸ = t = arg max f ( z 0 ) , we can solve:
[Equation]
If the solution z ∗ with v ∗ = f ( z ∗ ) t − f ( z ∗ ) γ ≤ f ( z ) t − f
( z ) γ , ∀ z ∈ C in satisfies v ∗ > 0 thenrobustness is verified for the adversarial class γ .
There are two main properties that we would like to have in a verification algorithm: soundness andcompleteness .
An algorithm is sound ( complete ) if every time it verifies (falsifies) a property, it isguaranteed to be the correct answer.
In practice, when an algorithm is guaranteed to provide the exactglobal minima of Eq.
(4), i.e., v ∗ , it is said to be sound and complete (usually referred in the literatureas simply complete [Ferrari et al., 2022]), whereas if a lower bound of it is provided ˆ v ∗ ≤ v ∗ , thealgorithm is sound but not complete .
We will not consider just complete verification, which simplyaims at looking for adversarial examples, e.g., Madry et al.
For a deeper discussion onsoundness and completeness , we refer to Liu et al.
Method
Our method, called VPN, can be categorized in the the Branch and Bound (BaB) framework [Landand Doig, 1960], a well known approach to global optimization [Horst and Tuy, 1996] and NNverification [Bunel et al., 2020a].
This kind of algorithms guarantee finding a global minima of theproblem in Eq.
(4) by recursively splitting the original feasible set into smaller sets (branching) where upper and lower bounds of the global minima are computed (bounding).
This mechanism can be usedto discard subsets where the global minima cannot be achieved (its lower bound is greater than theupper bound of another subset).
Our method is based on an α -BaB algorithm
[Adjiman et al., 1998], which is characterized for usingα -convexification [Adjiman and Floudas, 1996] for computing a lower bound of the global minimaof each subset.
To be specific, α -convexification aims to obtain a convex lower bounding function ofany twice-differentiable function f : R d → R .
In Adjiman and Floudas [1996], they propose twomethods:
• Uniform diagonal shift (single α ):
[Equation]
is its α -convexified version, note that z i is the i th element of vector
z
[Equation]
Let
H g ( z ) =
∇ 2 zz g
( z ) be the Hessian matrix of g , g α is convex in z ∈
[ l , u ] for α ≥max { 0 ,
[Equation]
12 min {
[Equation]
min ( H g ( z )) :
z ∈ [ l , u ] }} , where λ min is the minimum eigenvalue.
Moreover, it holds that g α ( z ; α, l , u ) ≤ g ( z ) , ∀ z ∈
[ l , u ] .
• Non-uniform diagonal shift (multiple α s):
[Equation]
is its α -convexified version.
In Adjiman et al.
[1998], they show that for any vector d > 0 ,setting
[Equation]
makes g α convex in z ∈ [ l , u ] .
The choice of the vector d is arbitrary, in this same paper,they propose to use d = 1 or d = u − l , we choose d = u − l as it shows better results in
Adjiman et al.
To make PN verification feasible, we need to study IBP for PNs and design an efficient estimate onthe α ( α in the case of Non-uniform diagonal shift case) parameter, which are our main technicalcontributions in the algorithmic aspect.
In our case, every feasible set, starting with the input set C in(Eq.
(3)), is split by taking the widest variable interval and dividing it in two by the middle point.
Thisis a rather simple, but theoretically powerful strategy, see Lemma 8.
Then, the upper bound of eachsubproblem is given by applying standard Projected Gradient Descent (PGD)
[Kelley, 1999] over theoriginal objective function.
This is a common approach to find adversarial examples [Madry et al.
,2018], but as the objective is non-convex, it is not sufficient for sound and complete verification.
Thelower bound is given by applying PGD over the α -convexified objective g α ( g α ), as it is convex, PGDconverges to the global minima and a lower bound of the original objective.
The α ( α ) parameteris computed only once per verification problem.
Further details on the algorithm and the proof ofconvergence of Eq.
(4) exist in Appendix D. A schematic of our method is available in Fig.
In Sections 3.1 to 3.3, we detail our method to compute a lower bound on the minimum eigenvalueof the Hessian matrix into three main components: interval propagation, lower bounding Hessians,and fast estimation on such lower bounding via power method.
To conclude the description of ourmethod, in Appendix A we describe the α estimation for the Non-uniform diagonal shift case.
3.1 Interval Bound Propagation through a PN
Interval bound propagation (IBP) is a key ingredient of our verification algorithm.
Suppose we havean input set defined by an ℓ ∞ -norm ball like in Eq.
This set can be represented as a vector ofintervals ˜
z
=
([ l 1 , u 1 ] ⊤ , [ l 2 , u 2 ] ⊤ , · · · , [ l d , u d ] ⊤ ) =
[ l , u ] ∈ R d × 2 , where [ l i , u i ] are the lower andupper bound for the i th coordinate.
Let L and U be the lower and upper bound IBP operators.
Giventhis input set, we would like to obtain bounds on the output of the network ( f ( z ) i ) , the gradient( ∇ z f ( z ) i ) , and the Hessian ( ∇ 2 zz f
( z ) i ) for any z ∈ ˜ z .
The operators L ( g ( z )) and U ( g ( z )) of anyfunction g : R d → R satisfy:
[Equation]
PN f
Input set C in
True class t
Adversarial class γ
Subsets
Branch
Compute α (Sections 3.2 and 3.3) min z ∈ C g
( z ) min z ∈ C g
α
( z )
Check bounds
Verified
Falsified
PGD PGD
C in
C
C α
U
L
L ≤branch C
C 1 ,C
L >
Subsets is empty
U ≤
We will define these upper and lower bound operators in terms of the operations present in a PN.
Using interval propagation [Moore et al., 2009], we can define:
Identity
[Equation]
linear mapping  
L ( (cid:88) i
w i h
[Equation]
( z ))
[Equation]
i w
[Equation]
h
[Equation]
w
[Equation]
U (
[Equation]
i w i h i ( z ))
[Equation]
(cid:88) i
w
[Equation]
i L ( h i ( z ))
+ w
+ i U
( h i ( z )) multiplication  
S =  
L ( h 1 ( z ))
L ( h 2 ( z )) ,
L ( h 1 ( z ))
U ( h 2 ( z )) ,
U ( h 1 ( z ))
L ( h 2 ( z )) ,
U ( h 1 ( z )) U ( h 2 ( z ))
  , | S | =
L ( h 1 ( z ) h 2 ( z ))
=
min S ,
U ( h 1 ( z ) h 2 ( z )) =
[Equation]
where w + i = max { 0 , w i } and w − i = min { 0 , w i } , h i ( z ) is any real-valued function of z and | · | isthe set cardinality.
Note that the set S is equivalent to:
[Equation]
With these basic operations one can define bounds on any intermediate output, gradient or Hessian ofa PN, e.g., the lower bound on the recursive formula from Eq.
(1) can be expressed as:
[Equation]
which only consists on a linear mapping and a multiplication of intervals.
We extend the upper andlower bound ( L ( · ) and U ( · ) ) operators to also work on vectors and matrices by applying them atevery position of the vector or matrix:
[Equation]
for g ( z ) ∈ R m and G ( z ) ∈ R
m 1 × m 2 .
One can directly use IBP to obtain bounds on the verificationobjective from Eq.
(4) with a single forward pass of the bounds through the network and obtaining
L ( g ( z ))
=
L ( f ( z ) t ) −U ( f ( z ) γ ) .
IBP is a common practice in NN verification to obtain fast bounds
3.2 Lower bound of the minimum eigenvalue of the Hessian
Here we describe our method to compute a lower bound on the minimum eigenvalue of the Hessianmatrix in the feasible set.
Before deriving the lower bound, we need the first and second order partialderivatives of PNs.
Let g ( z ) =
f ( z ) t − f ( z ) a be the objective function for t = arg max f ( z 0 ) and any a ̸ = t .
In orderto compute the parameter α for performing α -convexification, we need to know the structure of ourobjective function.
In this section we compute the first and second order partial derivatives of the PN.
The gradient and Hessian matrices of the objective function (see Eq.
(4)) are easily found to be:
[Equation]
we now define the gradients ∇ z x ( n ) i and Hessians ∇ 2 zz x ( n ) i of Eq.
(1) in a recursive way:
[Equation] [Equation]
[Equation]
with ∇ z x (1) i =
w [1]: i and ∇ 2 zz x
(1) i =
d × d , being
d × d a d × d matrix with 0 in every position.
Inthe next, we are ready to compute a lower bound on the minimum eigenvalue of the Hessian matrixin the feasible set.
Firstly, for any z ∈ [ l , u ] and any polynomial degree N , we can express the set of possible Hessians
H =
{ H g ( z ) :
z ∈ [ l , u ] } as an interval matrix.
An interval matrix is a matrix [ M ] ∈ R d × d × 2where every position [ m ] ij =
[ L ( m ij ) , U ( m ij )] is an interval.
Therefore, if H g ( z ) is boundedfor z ∈ [ l , u ] , then we can represent H
= { H g ( z ) :
H g ( z ) ∈
[ M ] } =
{ H g ( z ) : L ( m ij ) ≤
H g ( z ) ij ≤ U ( m ij ) , ∀ i, j ∈ [ d ] } .
For every set of Hessians, we can define the lower bounding Hessian L H .
Described in Adjiman et al.[1998], This matrix satisfies that λ min ( L H ) ≤ λ min ( H g ( z )) ,
∀ H g ( z ) ∈ H , z ∈ [ l , u ] .
Let L ( M )and U ( M ) , the lower bounding Hessian is defined as follows:
[Equation]
where 1 is a vector of ones and diag ( v ) is a diagonal matrix with the vector v in the diagonal.
Then, we can obtain the spectral radius ρ ( L H ) with a power method.
As the spectral radius satisfiesρ ( L H ) ≥ | λ i ( L H ) | , ∀ i ∈ [ d ] , the following inequality holds:
[Equation]
allowing us to use α = ρ ( L H )
[Equation]
[ l , u ] }} .
3.3 Efficient power method for spectral radius computation of the lower bounding Hessian
By using interval propagation, one can easily compute sound lower and upper bounds on each positionof the Hessian matrix, compute the lower bounding Hessian and perform a power method with itto obtain the spectral radius ρ .
However, this method would not scale well to high dimensionalscenarios.
For instance, in the STL10 case, with 96 × 96 RGB images ( d = 96 · 96 · 3 = 27 , 648 )our Hessian matrix would require in the order of O ( d 2 )
=
O (10 9 ) real numbers to be stored.
Thismakes it intractable to perform a power method over such an humongous matrix, or even to computethe lower bounding Hessian.
Alternatively, we take advantage of the possibility of expressing the
L H matrix as a sum of rank-1 matrices, to enable performing a power method over it.
Standard power method for spectral radius computation Given any squared and real valuedmatrix M ∈ R d × d and an initial vector v 0 ∈ R d that is not an eigenvector of M , the sequence:
[Equation]
converges to the eigenvector with the largest eigenvalue in absolute value, i.e. the eigenvector wherethe spectral norm is attained, being the spectral norm ρ ( M )
=
(cid:112) ||
M ( Mv n − 1 ) ||
[Mises and
Pollaczek-Geiringer, 1929].
Power method over lower bounding Hessian of PNs
We can employ IBP (Section 3.1) in order to obtain an expression of the lower bounding Hessian( L H ) and evaluate Eq.
(17) as:
[Equation]
Applying IBP on Eq.
(12) we obtain:
[Equation]
We can recursively evaluate L ( ∇ 2 zz x ( n ) i ) v
and U ( ∇ 2 zz x ( n ) i ) v efficiently as these matrices can beexpressed as a sum of rank- 1 matrices:
Proposition 1.
Let δ ∈
[ L ( δ ) , U ( δ )] be a real-valued weight, the matrix-vector products L ( δ ·∇ 2 zz x ( n ) i ) v
and U ( δ · ∇ 2 zz x ( n ) i ) v can be evaluated as:
[Equation] [Equation]
where δ ′ ∈
[ L ( δ ) , U ( δ )] · [ L ( w ⊤ [ n ]: i z
+ 1) , U ( w ⊤ [ n ]: i z + 1)] and vectors L ( δ · ∇ z x ( n − 1) i ) and U ( δ ·∇ z x ( n − 1) i ) can be obtained through IBP on Eq.
Lastly, by applying recursively Proposition 1 from n = N to n = 1 , starting with δ = 1 , we cansubstitute the results on Eq.
(19) and then on Eq.
(18) to efficiently evaluate a step of the powermethod (Eq.
without needing to store the lower bounding Hessian matrix or needing to performexpensive matrix-vector products.
Overall, our lower bounding method consists in computing a valid value of α that satisfies that theα -convexified
objective
g α is convex, following Eqs.
(4) and (5).
In particular, we use α = ρ ( L H ).ρ
( L H ) is computed via a power method, where the main operation L H v is evaluated without theneed to compute or store the L H matrix.
Provided this valid α , we perform PGD over g α and thisprovides a lower bound of the global minima of Eq.
Experiments
In this section we show the efficiency of our method by comparing against a simple Black-boxsolver.
Tightness of bounds is also analyzed in comparison with IBP and DeepT-Fast [Bonaert et al.
,2021], a zonotope based verification method able to handle multiplications tighter than IBP.
Finally,a study of the performance of our method in different scenarios is performed.
Unless otherwisespecified, every network is trained for 100 epochs with Stochastic Gradiend Descent (SGD), with alearning rate of 0 .
001 , which is divided by 10 at epochs [40 , 60 , 80] , momentum 0 .
9 , weight decay5 · 10 − 5 and batch size 128 .
We thoroughly evaluate our method over the popular image classification
[Table]
datasets MNIST [LeCun et al., 1998], CIFAR10 [Krizhevsky et al., 2014] and STL10 [Coates et al.
Every experiment is done over the first 1000 images of the test dataset, this is a commonpractice in verification [Singh et al., 2019].
For images that are correctly classified by the network,we sequentially verify robustness against the remaining classes in decreasing order of network output.
Each verification problem is given a maximum execution time of 60 seconds, we include experimentswith different time limits in Appendix E. Note that the execution time can be longer as execution iscut in an asynchronous way, i.e., after we finish the iteration of the BaB algorithm where the timelimit is reached.
All of our experiments were conducted on a single GPU node equipped with a32 GB NVIDIA V100 PCIe .
4.1 Comparison with Black-box solver
In this experiment, we compare the performance of our BaB verification algorithm with the Blackbox solver Gurobi [Gurobi Optimization, LLC, 2022].
Gurobi can globally solve Quadratically
Constrained Quadratic Programs whether they are convex or not.
As this solver cannot extend tohigher degree polynomial functions, we train 2 nd degree PNs with hidden size k = 16 to compare theverification time of our method with Gurobi.
In order to do so, we express the verification objective asa quadratic form g
( z ) =
f ( z ) t − f ( z ) a = z ⊤
Qz + q ⊤ z + c this together with the input constraintsz ∈ [ l , u ] is fed to Gurobi and optimized until convergence.
The black-box solver approach neither scales to higher-dimensional inputs nor to higher polynomialdegrees.
With this approach we need O ( d 2 ) memory to store the quadratic form, which makes itunfeasible for datasets with higher resolution images than CIFAR10.
On the contrary, as seen in
Table 1, our approach does not need so much memory and can scale to datasets with larger input sizeslike STL10.
4.2 Comparison with IBP and DeepT-Fast
In this experiment we compare the tightness of the lower bounds provided by IBP, DeepT-Fast andα -convexification and their effectiveness when employed for verification.
This is done by executingone upper bounding step with PGD and one lower bounding step for each lower bounding methodover
the initial feasible set provided by ϵ (see Eq.
We compare the average of the distance fromeach lower bound to the PGD upper bound over the first 1000 images of the MNIST dataset for PNswith hidden size k = 25 and degrees ranging from 2 to 7 .
We also evaluate verified accuracy of 2 nd(PN_Conv2) and 4 th (PN_Conv4) PNs with IBP, DeepT-Fast and α -convexification in the Uniformdiagonal shift setup.
We employ a maximum time of 120 seconds.
For details on the architecture ofthese networks, we refer to Appendix E.
When using IBP, we get a much looser lower bound than with α -convexification, see Fig.
Only forhigh-degree, high- ϵ combinations IBP lower bounds are closer to the PGD upper bound.
In practice,this is not a problem for verification, as for epsilons in the order of 0 .
1 , it is really easy to findadversarial examples with PGD and there will be no accuracy left to verify.
DeepT-Fast significantlyoutperforms IBP bounds across all degrees and ϵ values.
But, as observed in Fig.
3, except for big ϵ
7 Degree 10 −10 −
U − L ϵ = 0 .
7 Degree ϵ = 0 .     7 Degree ϵ = 0 .
7 Degree ϵ =
.α
-conv.
single α -conv.
multi
IBP DeepT-Fast
[Table]
values, its performance is still far from the one provided by both α -convexification methods.
Whencomparing both α -convexification methods (blue and green lines in Fig. 3), we observe that for smalldegree PNs ( N < 5 ), in the Uniform diagonal shift case we are able to obtain tighter bounds.
The looseness of the IBP lower bound is confirmed when comparing the verified accuracy with
IBP and the rest of lower bounding methods, see Table 2.
With α -convexification, we are able toverify the accuracy of 2 nd and 4 th order PNs almost exactly (almost no gap between the verifiedaccuracy and its upper bound) in every studied dataset, while with IBP, we are not able to verifyrobustness for a single image in any network- ϵ pair, confirming the fact that IBP cannot be usedfor PN verification.
The improvements in the bounds when utilizing DeepT-Fast instead of IBP isclearly seen in verification results in Table 2.
With DeepT-Fast, we are able to effectively verify PNsfaster than with α -convexification, but achieving a much lower verified accuracy ( Ver% ) than withα -convexification.
As a reference, for CIFAR10 PN_Conv2 at ϵ = 2 / 255 , with α -convexification weobtain 25 .
4% verified accuracy, while with DeepT-Fast, we can just obtain 1 .
4% verified accuracy.
Itis worth highlighting that DeepT-Fast can also scale to verify networks trained on STL10.
Conclusion
We propose a method to verify polynomial networks (PNs).
Our method, which can be categorizedas a α -BaB global optimization algorithm, is a novel approach to verification of PNs.
We believe canbe extended to cover other twice-differentiable networks in the future.
We exhibit that our methodoutperforms existing methods, such as black-box solvers, IBP and DeepT-Fast [Bonaert et al., 2021].
Our method enables verification in datasets such as STL10, which includes RGB images of 96 × 96resolution.
This is larger than the images typically used in previous verification methods , we note thatexisting methods like IBP and DeepT-Fast are also able to scale to STL10 buth with a lower verifiedaccuracy.
Our method can further encourage the community to extend verification to a broader classof functions as well as conduct experiments in datasets of higher resolution.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] In Appendix F we discuss the
limitations.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our
method aims at verifying PNs robustness to adversarial attacks, which has no direct
negative societal impact, see Appendix F
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] The complete proofof propositions, lemmas and theorems is present in the appendix, see Appendix G.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We use publicly
available datasets and our code and instructions are available in the supplementary
material.(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] We follow the standard practice in the literature forreporting the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] A single GPU was used in eachexperiment.
The single GPU is part of our internal cluster.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A] Standard image-based datasets are
used in this work.
All of them are publicly available and we cite their respecting papers,
where the licenses are mentioned.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]