Sound and Complete Verification of Polynomial Networks
Abstract
Polynomial Networks (PNs) have demonstrated promising performance on face and image recognition recently.
However, robustness of PNs is unclear and thus obtaining certificates becomes imperative for enabling their adoption in real-world applications.
Existing verification algorithms on ReLU neural networks (NNs) based on classical branch and bound (BaB) techniques cannot be trivially applied to PN verification.
In this work, we devise a new bounding method, equipped with BaB for global convergence guarantees, called Verification of Polynomial
Networks or VPN for short.
One key insight is that we obtain much tighter bounds than the interval bound propagation (IBP) and DeepT-Fast
[Bonaert et al., 2021] baselines.
This enables sound and complete PN verification with empirical validation on MNIST, CIFAR10 and STL10 datasets.
We believe our method has its own interest to NN verification.
The source code is publicly available at https://github.com/megaelius/PNVerification .
Introduction
Polynomial Networks (PNs) have demonstrated promising performance across image recognition and generation [Chrysos et al., 2021, Chrysos and Panagakis, 2020] being state-of-the-art on large-scale face recognition 2 .Unlike
the conventional Neural Networks (NNs), where non-linearitiy is introduced with the use of activation functions [LeCun et al., 2015], PNs are able to learn non-linear mappings without the need of activation functions by exploiting multiplicative interactions (Hadamard products).
Recent works have uncovered interesting properties of PNs in terms of model expressivity [Fan et al., 2021] and spectral bias [Choraria et al., 2022].
However, one critical issue before considering PNs for real-world applications is their robustness.
Neural networks are prone to small (often imperceptible to the human eye), but malicious perturbations in the input data points [Szegedy et al., 2014, Goodfellow et al., 2015].
Those perturbations can have a detrimental effect on image recognition systems, e.g., as illustrated in face recognition
[Goswami et al., 2019, Zhong and Deng, 2019, Dong et al., 2019, Li et al., 2020].
Guarding against such attacks has so far proven futile [Shafahi et al., 2019, Dou et al., 2018].
Instead, a flurry of research has been published on certifying robustness of NNs against this performance degradation [Katz et al., 2017, Ehlers, 2017, Tjeng et al., 2019, Bunel et al., 2020b, Wang et al., 2021, Ferrari et al., 2022].
However, most of the verification algorithms for NNs are developed for the ReLU activation function by exploiting its piecewise linearity property and might not trivially extend to other nonlinear activation functions [Wang et al., 2021]. Indeed, Zhu et al.
[2022] illustrate that guarding PNs against adversarial attacks is challenging.
Therefore, we pose the following question:
Can we obtain certifiable performance for PNs against adversarial attacks?
In this work, we answer affirmatively and provide a method for the verification of PNs.
Concretely, we take advantage of the twice-differentiable nature of PNs to build a lower bounding method based on α -convexification
[Adjiman and Floudas, 1996], which is integrated into a Branch and Bound (BaB)
algorithm
[Land and Doig, 1960] to guarantee completeness of our verification method.
In order to use α -convexification, a lower bound α of the minimum eigenvalue of the Hessian matrix over the possible perturbation set is needed.
We use interval bound propagation together with the theoretical properties of the lower bounding Hessian matrix [Adjiman et al., 1998], in order to develop an algorithm to efficiently compute α .
Our contributions can be summarized as follows: ( i )
We propose the first algorithm for the verification of PNs.
( ii ) We thoroughly analyze the performance of our method by comparing it with a black-box solver, with an interval bound propagation (IBP) BaB algorithm and with the zonotopebased abstraction method
DeepT-Fast [Bonaert et al., 2021].
( iii ) We empirically show that using α -convexification for lower bounding provides tighter bounds than IBP and DeepT-Fast for PN verification.
To encourage the community to improve the verification of PNs, we make our code publicly available in https://github.com/megaelius/PNVerification .
The proposed approach can practically verify PNs, while it could also theoretically be applied for sound and complete verification of any twice-differentiable network.
Notation: We use the shorthand [ n ] := { 1 , 2 , . . . , n } for a positive integer n .
We use bold capital (lowercase) letters, e.g., X ( x ) for representing matrices (vectors).
The j th column of a matrix X is given by x : j .
The element in the i th row and j th column is given by x ij , similarly, the i th element of a vector x is given by x i .
The element-wise (Hadamard) product, symbolized with ∗ , of two matrices (or vectors) in R d 1 × d 2 (or R d ) gives another matrix (or vector) in R d 1 × d 2 (or R d ).
The ℓ ∞ norm of a vector x ∈ R d is given by: || x || ∞ =
max i ∈ [ d ] | x i | .
Lastly, the operators L and U give the lower and upper bounds of a scalar, vector or matrix function by IBP, see Section 3.1.
Roadmap: We provide the necessary background by introducing the PN architecture and formalizing the Robustness Verification problem in Section 2.
Section 3 provides a sound and complete method called VPN to tackle PN verification problem.
Section 4 is devoted to experimental validation.
Additional experiments, details and proofs are deferred to the appendix.
Background
We give an overview of the PN architecture in Section 2.1 and the robustness verification problem in
Section 2.2.
2.1 Polynomial Networks (PNs)
Polynomial Networks (PNs) are inspired by the fact that any smooth function can be approximated via a polynomial expansion [Stone, 1948].
However, the number of parameters increases exponentially with the polynomial degree, which makes it intractable to use high degree polynomials for highdimensional data problems such as image classification where the input can be in the order of 10[Deng et al., 2009].
Chrysos et al.
[2021] introduce a joint factorization of polynomial coefficients in a low-rank manner, reducing the number of parameters to linear with the polynomial degree and allowing the expression as a neural network (NN).
We briefly recap one fundamental factorization below.
Let N be the polynomial degree,
z ∈ R
d be the input vector, d , k and o be the input, hidden and output sizes, respectively.
The recursive equation of PNs can be expressed as:
[Equation]
z
W [1] ∗ + W [2] ∗ + W [3] C + β f ( z ) x (1) x (2) x (3) where x (1) =
W ⊤ [1] z , f ( z ) =
Cx ( N ) + β and ∗ denotes the Hadamard product.
W [ n ] ∈ R d × k and
C ∈ R o × k are weight matrices, β ∈ R o is a bias vector.
A graphical representation of a third degree
PN architecture corresponding to Eq.
(1) can be found in Fig.
Further details on the factorization (as well as other factorizations) are deferred to the Appendix C.1 (Appendix C.2).
2.2 Robustness Verification
Robustness verification [Bastani et al., 2016, Liu et al., 2021] consists of verifying that a property regarding the input and output of a NN is satisfied, e.g. checking whether or not a small perturbation in the input will produce a change in the network output that makes it classify the input into another class.
Let f : [0 , 1] d → R o be a function, e.g., a NN or a PN, that classifies the input z into a class c , such that c = arg max f ( z ) .
Our target is to verify that for any input satisfying a set of constraints
C in , the output of the network will satisfy a set of output constraints C out .
Mathematically,
[Equation]
In this work we focus on adversarial robustness [Szegedy et al., 2014, Carlini and Wagner, 2017] in classification.
Given an observation z 0 , let t = arg max f
( z 0 ) be the correct class, our goal is to check whether every input in a neighbourhood of z 0 , is classified as t .
In this work, we focus on adversarial attacks restricted to neighbourhoods defined in terms of ℓ ∞ norm, which is a popular norm-bounded attack in the verification community [Liu et al., 2021].
Then, the constraint sets become:
[Equation]
In other words, we need an algorithm that given a function f , an input z 0 and an adversarial budget ϵ , checks whether Eq.
(2) is satisfied.
In the case of ReLU NNs, this has been proven to be an
NP-complete problem [Katz et al., 2017].
This can be reformulated as a constrained optimization problem.
For every adversarial class γ ̸ = t = arg max f ( z 0 ) , we can solve:
[Equation]
If the solution z ∗ with v ∗ = f ( z ∗ ) t − f ( z ∗ ) γ ≤ f ( z ) t − f
( z ) γ , ∀ z ∈ C in satisfies v ∗ > 0 then robustness is verified for the adversarial class γ .
There are two main properties that a verification algorithm admits: soundness and completeness .
An algorithm is sound ( complete ) if every time it verifies (falsifies) a property, it is guaranteed to be the correct answer.
In practice, when an algorithm is guaranteed to provide the exact global minima of Eq.
(4), i.e., v ∗ , it is said to be sound and complete (usually referred in the literature as simply complete [Ferrari et al., 2022]), whereas if a lower bound of it is provided ˆ v ∗ ≤ v ∗ , the algorithm is sound but not complete .
In our work, we will not consider just complete verification, which simply aims at looking for adversarial examples, e.g., Madry et al.
For a deeper discussion on soundness and completeness , we refer to Liu et al.
Method
Our method, called VPN, can be categorized in the the Branch and Bound (BaB) framework [Land and Doig, 1960], a well known approach to global optimization [Horst and Tuy, 1996] and NN verification [Bunel et al., 2020b].
This kind of algorithms ensures finding a global minima of the problem in Eq.
(4) by recursively splitting the original feasible set into smaller sets (branching) where upper and lower bounds of the global minima are computed (bounding).
This mechanism can be used to discard subsets where the global minima cannot be achieved (its lower bound is greater than the upper bound of another subset).
Our method is based on a variant of BaB algorithm, i.e., α -BaB [Adjiman et al., 1998], which is characterized for using α -convexification [Adjiman and Floudas, 1996] for computing a lower bound of the global minima of each subset.
To be specific, α -convexification aims to obtain a convex lower bounding function of any twice-differentiable function f : R d → R .
In Adjiman and Floudas [1996], they propose two methods:
• Uniform diagonal shift (single α )
[Equation]
is its α -convexified version, note that z i is the i th element of vector z .
Let
H g ( z ) =
∇ 2 zz g
( z ) be the Hessian matrix of g , g α is convex in z ∈
[ l , u ] for α ≥ max { 0 , − 12 min { λ min ( H g ( z )) :
z ∈ [ l , u ] }} , where λ min is the minimum eigenvalue.
Moreover, it holds that g α ( z ; α, l , u ) ≤ g ( z ) , ∀ z ∈
[ l , u ] .
• Non-uniform diagonal shift (multiple α ’s)
[Equation]
is its α -convexified version.
In Adjiman et al.
[1998], they show that for any vector d > 0 , setting
[Equation]
makes g α convex in z ∈ [ l , u ] .
The choice of the vector d is arbitrary, but affects the final result.
For example, taking d = u − l yields better results than d = 1 in Adjiman et al.
We need to remark that, though Eq.
(5) is a special case of Eq.
(6), the lower bound of α in Eq.
(5) via minimum eigenvalue of the lower bounding Hessian matrix cannot be regarded as a special case of Eq.
To make PN verification feasible via α -convexification, we need to study IBP for PNs and design an efficient estimate on the α ( α in the case of Non-uniform diagonal shift) parameter, which are our main technical contributions in the algorithmic aspect.
In our case, every feasible set, starting with the input set C in (Eq.
(3)), is split by taking the widest variable interval and dividing it in two by the middle point.
This is a rather simple, but theoretically powerful strategy, see Lemma 8 in
Appendix F. Then, the upper bound of each subproblem is given by applying standard Projected
Gradient Descent (PGD)
[Kelley, 1999] over the original objective function.
This is a common approach to find adversarial examples [Madry et al., 2018], but as the objective is non-convex, it is not sufficient for sound and complete verification.
The lower bound is given by applying PGD over the α -convexified objective g α ( g α ), as it is convex, PGD converges to the global minima and a lower bound of the original objective.
The α ( α ) parameter is computed only once per verification problem.
Further details on the algorithm and the proof of convergence of Eq.
(4) exist in Appendix D. A schematic of our method is available in Fig.
In Sections 3.1 to 3.3, we detail our method under the uniform diagonal shift case to compute a lower bound on the minimum eigenvalue of the Hessian matrix into three main components: interval propagation, lower bounding Hessians, and fast estimation on such lower bounding via power method.
To conclude the description of our method, in Section 3.4 we describe the α estimation for the non-uniform diagonal shift case.
PN f
Input set C in
True class t
Adversarial class γ
Subsets
Branch
Compute α (Sections 3.2 and 3.3) min z ∈ C g
( z ) min z ∈ C g
α
( z )
Check bounds
Verified
Falsified
PGD PGD
C in
C
C α
U
L
L ≤branch C
C 1 ,C
L >
Subsets is empty
U ≤
3.1 Interval Bound Propagation through a PN
Interval bound propagation (IBP) is a key ingredient of our verification algorithm.
Suppose we have an input set defined by an ℓ ∞ -norm ball like in Eq.
This set can be represented as a vector of intervals [ l , u ] =
([ l 1 , u 1 ] ⊤ , [ l 2 , u 2 ] ⊤ , · · · , [ l d , u d ] ⊤ )
∈
R
d × 2 , where [ l i , u i ] are the lower and upper bound for the i th coordinate.
Let L and U be the lower and upper bound IBP operators.
Given this input set, we would like to obtain bounds on the output of the network ( f ( z ) i ) , the gradient ( ∇ z f ( z ) i ) , and the Hessian ( ∇ 2 zz f
( z ) i ) for any z ∈ [ l , u ] .
The operators L ( g ( z )) and U ( g ( z )) of any function g : R d → R satisfy:
[Equation]
We will define these upper and lower bound operators in terms of the operations present in a PN.
Using interval propagation [Moore et al., 2009], denote the positive part w + i = max { 0 , w i } and the negative part w − i = min { 0 , w i } , h i ( z ) as a real-valued function of z , i ∈
[ d ] , we can define:
Identity
[Equation]
linear mapping
[Equation]
multiplication
[Equation]
where | · | is the set cardinality.
Note that the set S is equivalent to:
[Equation]
With these basic operations, one can define bounds on any intermediate output, gradient or Hessian of a PN.
For instance, the lower bound on the recursive formula from Eq.
(1) can be expressed as:
[Equation]
n ) i ) =
L (( w ⊤ [ n ]: i z ) x
[Equation]
n − 1) i
[Equation]
which only consists on a linear mapping and a multiplication of intervals.
We extend the upper and lower bound ( L ( · ) and U ( · ) ) operators to vectors and matrices in an entry-wise style:
[Equation]
Note that Eq.
(11) is not limited to squared matrices and can hold for arbitrary matrix dimensions.
One can directly use IBP to obtain bounds on the verification objective from Eq.
(4) with a single forward pass of the bounds through the network and obtaining L ( g ( z ))
=
L ( f ( z ) t ) − U ( f ( z ) γ ) .
In fact IBP is a common practice in NN verification to obtain fast bounds [Wang et al., 2018a].
3.2 Lower bound of the minimum eigenvalue of the Hessian
Here we describe our method to compute a lower bound on the minimum eigenvalue of the Hessian matrix in the feasible set.
Before deriving the lower bound, we need the first and second order partial derivatives of PNs.
Let g ( z ) =
f ( z ) t − f ( z ) a be the objective function for t = arg max f ( z 0 ) and any a ̸ = t .
In order to compute the parameter α for performing α -convexification, we need to know the structure of our objective function.
In this section we compute the first and second order partial derivatives of the PN.
The gradient and Hessian matrices of the objective function (see Eq.
(4)) are given by:
[Equation]
We now define the gradients ∇ z x ( n ) i and Hessians ∇ 2 zz x ( n ) i of Eq.
(1) in a recursive way:
[Equation]
with ∇ z x (1) i =
w [1]: i and ∇ 2 zz x
(1) i =
d × d being an all-zero matrix.
In the next, we are ready to compute a lower bound on the minimum eigenvalue of the Hessian matrix in the feasible set.
Firstly, for any z ∈ [ l , u ] and any polynomial degree N , we can express the set of possible Hessians
H =
{ H g ( z ) :
z ∈ [ l , u ] } as an interval matrix.
An interval matrix is a tensor [ M ] ∈ R d × d ×where every position [ m ] ij =
[ L ( m ij ) , U ( m ij )] is an interval.
Therefore, if H g ( z ) is bounded for z ∈ [ l , u ] , then we can represent H
= { H g ( z ) :
H g ( z ) ∈
[ M ] } =
{ H g ( z ) : L ( m ij ) ≤
H g ( z ) ij ≤ U ( m ij ) , ∀ i, j ∈ [ d ] } .
Let L ( M ) and U ( M ) be the element-wise lower and upper bounds of a Hessian matrix, the lower bounding Hessian is defined as follows:
[Equation]
where 1 is an all-one vector and diag ( v ) is a diagonal matrix with the vector v in the diagonal.
Described in Adjiman et al.
[1998], this matrix satisfies that λ min ( L H ) ≤ λ min ( H g ( z )) ,
∀ H g ( z ) ∈
H , z ∈ [ l , u ] .
Then, we can obtain the spectral radius ρ ( L H ) with a power method [Mises and Pollaczek-Geiringer, 1929].
As the spectral radius satisfies ρ ( L H ) ≥ | λ i ( L H ) | , ∀ i ∈ [ d ] , the following inequality holds:
[Equation]
allowing us to use α = ρ ( L H
[Equation] [Equation]
3.3 Efficient power method for spectral radius computation of the lower bounding Hessian
By using interval propagation, one can easily compute sound lower and upper bounds on each position of the Hessian matrix, compute the lower bounding Hessian and perform a power method with it to obtain the spectral radius ρ .
However, this method would not scale well to high dimensional scenarios.
For instance, in the STL10 dataset [Coates et al., 2011] (with input dimension d = 96 · 96 · 3 = 27 , 648 ) in each color image, our Hessian matrix would require an order of O ( d 2 ) =
O (10 9 ) real numbers to be stored.
This makes it intractable to perform a power method over such an humongous matrix, or even to compute the lower bounding Hessian.
Alternatively, we take advantage of the possibility of expressing the L H matrix as a sum of rank-1 matrices, to enable performing a power method over it.
Standard power method for spectral radius computation
Given any squared and real valued matrix M ∈ R d × d and an initial vector v 0 ∈ R d that is not an eigenvector of M , the sequence:
[Equation]
[Equation]
converges to the eigenvector with the largest eigenvalue in absolute value, i.e. the eigenvector where the spectral radius is attained, being the spectral radius ρ ( M ) =
(cid:112) ||
M ( Mv n − 1 ) ||
[Mises and
Pollaczek-Geiringer, 1929].
Power method over lower bounding Hessian of PNs
We can employ IBP (Section 3.1) in order to obtain an expression of the lower bounding Hessian ( L H ) and evaluate Eq.
(17) as:
[Equation]
Applying IBP on Eq.
(12) we obtain:
[Equation]
We can recursively evaluate L ( ∇ 2 zz x ( n ) i ) v
and U ( ∇ 2 zz x ( n ) i ) v efficiently as these matrices can be expressed as a sum of rank- 1 matrices as below
[Equation]
Proposition 1.
Let δ ∈
[ L ( δ ) , U ( δ )] be a real-valued weight, the matrix-vector products L ( δ · ∇ 2 zz x ( n ) i ) v
and U ( δ · ∇ 2 zz x ( n ) i ) v can be evaluated as:
[Equation]
where δ ′ ∈
[ L ( δ ) , U ( δ )] · [ L ( w ⊤ [ n ]: i z
+ 1) , U ( w ⊤ [ n ]: i z + 1)] and vectors L ( δ · ∇ z x ( n − 1) i ) and U ( δ · ∇ z x ( n − 1) i ) can be obtained through IBP on Eq.
Lastly, by applying recursively Proposition 1 from n = N to n = 1 , starting with δ = 1 , we can substitute the results on Eq.
(19) and then on Eq.
(18) to efficiently evaluate a step of the power method (Eq.
without needing to store the lower bounding Hessian matrix or needing to perform expensive matrix-vector products.
Overall, our lower bounding method consists in computing a valid value of α that satisfies that the α -convexified objective g α is convex, following Eqs.
(4) and (5).
In particular, we use α = ρ ( L H ).
ρ ( L H ) is computed via a power method, where the main operation L H v is evaluated without the need to compute or store the L H matrix.
Provided this valid α , we perform PGD over g α and this provides a lower bound of the global minima of Eq.
3.4 Non-uniform diagonal shift
In order to obtain an estimate of the α parameter as defined by Adjiman et al.
[1998] in Eq.
(7), we make use of the rank-1 matrices IBP rules defined in Appendix F.1.
We also define the operator
M ( · ) = max {| L ( . )
| , | U ( . ) |} and certain useful properties about it in Appendix F.2.
Thanks to
Lemmas 4 to 6, we can obtain an expression for M ( H z ( g )) .
This will be used to compute the vector α in the Non-uniform diagonal shift scenario for α -convexification.
Theorem 1.
Let f be a N -degree CCP PN defined as in Eq.
Let g ( z ) =
f ( z ) t − f ( z ) γ for any t ̸ = γ, t ∈
[ o ] , γ ∈ [ o ] .
Let H g
( z ) be the Hessian matrix of g, the operation M ( H g ( z )) results in:
[Equation]
[Equation]
where for n = 2 , ..., N , we can express:
[Equation]
Lastly, for n = 1 , M (cid:16) ∇ 2 zz x (1) i (cid:17) =
d × d is a d × d matrix full of zeros.
As one can observe, in Theorem 1, the matrix M ( H g ( z )) is expressed as a sum of rank-1 matrices.
This allows to efficiently compute (cid:80)
j ̸ = i M
( h g ( z ) ij )
d i d j , which is necessary for the right term in
Eq.
For the left term in Eq.
(7), we can efficiently compute the lower bound of the diagonal of the Hessian matrix by using the rules present in Section 3.1 and Appendix F.1.
Experiments
In this Section we show the efficiency of our method by comparing against a simple Black-box solver.
Tightness of bounds is also analyzed in comparison with IBP and DeepT-Fast [Bonaert et al., 2021], a zonotope based verification method able to handle multiplications tighter than IBP.
Finally, a study of the performance of our method in different scenarios is performed.
Unless otherwise specified, every network is trained for 100 epochs with Stochastic Gradiend Descent (SGD), with a learning rate of 0 .
001 , which is divided by 10 at epochs [40 , 60 , 80] , momentum 0 .
9 , weight decay 5 · 10 − 5 and batch size
We thoroughly evaluate our method over the popular image classification datasets MNIST [LeCun et al., 1998], CIFAR
[Krizhevsky et al., 2014] and STL
[Coates et al., 2011].
Every experiment is done over the first 1000 images of the test dataset, this is a common practice in verification [Singh et al., 2019].
For images that are correctly classified by the network, we sequentially verify robustness against the remaining classes in decreasing order of network output.
Each verification problem is given a maximum execution time of 60 seconds, we include experiments with different time limits in Appendix E. Note that the execution time can be longer as execution is cut in an asynchronous way, i.e., after we finish the iteration of the BaB algorithm where the time limit is reached.
All of our experiments were conducted on a single GPU node equipped with a
GB NVIDIA V100 PCIe.
4.1 Comparison with a Black-box solver
In this experiment, we compare the performance of our BaB verification algorithm with the Blackbox solver Gurobi [Gurobi Optimization, LLC, 2022].
Gurobi can globally solve Quadratically
Constrained Quadratic Programs whether they are convex or not.
As this solver cannot extend to higher degree polynomial functions, we train 2 nd degree PNs with hidden size k = 16 to compare the verification time of our method with Gurobi.
In order to do so, we express the verification objective as a quadratic form g ( z ) =
f ( z ) t − f ( z ) a = z ⊤
Qz + q ⊤ z + c this together with the input constraints z ∈ [ l , u ] is fed to Gurobi and optimized until convergence.
The black-box solver approach neither scales to higher-dimensional inputs nor to higher polynomial degrees.
With this approach we need O ( d 2 ) memory to store the quadratic form, which makes it unfeasible for datasets with higher resolution images than CIFAR10.
On the contrary, as seen in
Table 1, our approach does not need so much memory and can scale to datasets with larger input sizes like STL10.
[Table]
4.2 Comparison with IBP and DeepT-Fast
In this experiment we compare the tightness of the lower bounds provided by IBP, DeepT-Fast and α -convexification and their effectiveness when employed for verification.
This is done by executing one upper bounding step with PGD and one lower bounding step for each lower bounding method over the initial feasible set provided by ϵ (see Eq.
We compare the average of the distance from each lower bound to the PGD upper bound over the first 1000 images of the MNIST dataset for PNs with hidden size k = 25 and degrees ranging from 2 to 7 .
We also evaluate verified accuracy of 2 nd (PN_Conv2) and 4 th (PN_Conv4) PNs with IBP, DeepT-Fast and α -convexification in the Uniform diagonal shift setup.
We employ a maximum time of 120 seconds.
For details on the architecture of these networks, we refer to Appendix E.
[Figure]
When using IBP, we get a much looser lower bound than with α -convexification, see Fig.
Only for high-degree, high- ϵ combinations IBP lower bounds are closer to the PGD upper bound.
In practice, this is not a problem for verification, as for epsilons in the order of 0 .
1 , it is really easy to find adversarial examples with PGD and there will be no accuracy left to verify.
DeepT-Fast significantly outperforms IBP bounds across all degrees and ϵ values.
But, as observed in Fig. 3, except for big ϵ values, its performance is still far from the one provided by both α -convexification methods.
When comparing both α -convexification methods (blue and green lines in Fig. 3), we observe that for small degree PNs ( N < 5 ), in the Uniform diagonal shift case we are able to obtain tighter bounds.
The looseness of the IBP lower bound is confirmed when comparing the verified accuracy with
IBP and the rest of lower bounding methods, see Table 2.
With α -convexification, we are able to verify the accuracy of 2 nd and 4 th order PNs almost exactly (almost no gap between the verified accuracy and its upper bound) in every studied dataset, while with IBP, we are not able to verify robustness for a single image in any network- ϵ pair, confirming the fact that IBP cannot be used
[Table] [Table]
[Table] [Table]
for PN verification.
The improvements in the bounds when utilizing DeepT-Fast instead of IBP is clearly seen in verification results in Table 2.
With DeepT-Fast, we are able to effectively verify PNs faster than with α -convexification, but achieving a much lower verified accuracy ( Ver% ) than with α -convexification.
As a reference, for CIFAR10 PN_Conv2 at ϵ = 2 / 255 , with α -convexification we obtain 25 .
4% verified accuracy, while with DeepT-Fast, we can just obtain 1 .
4% verified accuracy.
It is worth highlighting that DeepT-Fast can also scale to verify networks trained on STL10.
Conclusion
We propose a novel α -BaB global optimization algorithm to verify polynomial networks (PNs).
We exhibit that our method outperforms existing methods, such as black-box solvers, IBP and
DeepT-Fast [Bonaert et al., 2021].
Our method enables verification in datasets such as STL10, which includes RGB images of 96 × 96 resolution.
This is larger than the images typically used in previous verification methods. Note that existing methods like IBP and DeepT-Fast are also able to scale to
STL10 buth with a lower verified accuracy.
Our method can further encourage the community to extend verification to a broader class of functions as well as conduct experiments in datasets of higher resolution.
We believe our method can be extended to cover other twice-differentiable networks in the future.
Limitations
As discussed in Appendix E.2, our verification method does not scale to high-degree PNs.
Even though we can verify high-accuracy PNs (see Table 2), we are still far from verifying the top performing deep PNs studied in Chrysos et al.
Another problem that we share with ReLU
NN verifiers is the scalability to networks with larger input size [Wang et al., 2021].
In this work we are able to verify networks trained in STL
[Coates et al., 2011], but these networks are shallow, yet their verification still takes a long time, see Table 2.
Acknowledgements
We are deeply thankful to the reviewers for providing constructive feedback.
Research was sponsored by the Army Research Office and was accomplished under Grant Number W911NF-19-1-0404.
This project has received funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation programme (grant agreement number 725594 - time-data).
This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021_178865.
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement n°- time-data).
This work was supported by Zeiss.
This work was supported by SNF project – Deep
Optimisation of the Swiss National Science Foundation (SNSF) under grant number 200021_205011.