U NDERSTANDING
R E LU
N ETWORK
R OBUSTNESS
T HROUGH T EST S ET C ERTIFICATION P ERFORMANCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural networks might exhibit weak robustness against input perturbations within the learning distribution and become more severe for distributional shifts or data outside the distribution.
For their safer use, robustness certificates provide formal guarantees to the stability of the prediction in the vicinity of the input.
However, the relationship between correctness and certified robustness remains unclear.
In this work, we investigate the unexpected outcomes of verification methods applied to piecewise linear classifiers for clean, perturbed, in- and out-of-distribution samples.
In our experiments, we conduct a thorough analysis for image classification tasks and show that robustness certificates are strongly correlated with prediction correctness for in-distribution data.
In addition, we provide a theoretical demonstration that formal verification methods robustly certify samples sufficiently far from the training distribution.
These results are integrated with an experimental analysis and demonstrate their weakness compared to standard out-of-distribution detection methods.
I NTRODUCTION
Building reliable artificial intelligence systems requires systematic methods for assessing their quality to gain confidence in their correctness or to identify possible failures.
In general, neural networks are non-robust against geometric perturbations and are easily fooled by precisely calculated adversarial attacks (Biggio et al., 2013; Szegedy et al., 2014).
For these reasons, relying solely on model’s prediction is not sufficient to ensure safe results.
The problem of adversarial attacks has been addressed in the literature with a variety of defense mechanisms, divided into empirical and provable defenses.
Empirical defenses aim to improve the robustness of the model through training with adversarial samples (Goodfellow et al., 2015; Carlini & Wagner, 2017;
Madry et al., 2017; Andriushchenko et al., 2020).
However, robustness comes at the expense of accuracy (Yang et al., 2020;
Jovanovic et al., 2021), and there is absolutely no guarantee that the model will behave correctly in the event of new, unseen attacks.
To overcome this problem, formal verification methods, e.g. Reluplex (Katz et al., 2017), are proposed to increase the trustworthiness of a prediction by assuring its stability in the vicinity of the input. Formal verification methods are subsequently divided into exact or complete (Katz et al., 2017; Bunel et al., 2018; Lu & Kumar, 2020; Xu et al., 2020; Wang et al., 2021) and approximate or incomplete (Zhang et al., 2018; Dvijotham et al., 2018; Gehr et al., 2018; M¨uller et al., 2021; 2022; Wang et al., 2021).
To give an intuition of how incomplete verification works, consider a convex space constructed around the input and then propagate it through the nonlinearity of the network.
At the output layer, the resulting shape is certified as robust if it is entirely contained inside the same predicted class, without crossing any decision boundary.
Motivation. The aim of this work is to understand how robustness certificates can be used at operational time, i.e., when the labels are not given. In this context, robustness certificates (or formal verification methods) ensure the stability of the prediction in the vicinity of the input for a predefined perturbation. But what does this mean in practice?
Can we trust the prediction or not?
If not, how much does this increase our confidence in correctness?
Can we use certified robustness as an additional safety metric?
Why should we use robustness certificates, given that a misclassified or out-of-distribution sample can be robustly certified?
In our analysis we want to explore how to relate certifiable robustness to correctness at operational time.
For this reason, we evaluate robustness certificates on both in- and out-of-distribution cases.
Previous literature has mainly focused on improving robustness verification in qualitative or quantitative terms.
For example, increasing the number of certified samples within the correctly classified ones, or speeding up the verification process (Wang et al., 2021; M¨uller et al., 2021; 2022).
Another line explores the tension between adversarial robustness and accuracy from an empirical (Yang et al., 2020;
Liu et al., 2020a) or provable (Jovanovic et al., 2021; M¨uller et al., 2022) training perspective.
The purpose is to emphasize how these training methods increase robustness at the expense of network accuracy and how to mitigate this tradeoff.
In this line, common benchmarks (Hendrycks & Dietterich, 2018; Croce et al., 2021; Wu et al., 2022) relate correctness with robustness only for correctly classified samples.
[Figure]
In this paper, we examine it through another perspective.
We evaluate certified robustness independently of the classification results.
Specifically, we address the following questions.
The first is certified and correctly classified ( • , • in Figure 1). Can we guarantee that a certified test sample has been classified correctly?
This question wants to clarify if it possible to verify whether a test sample is classified correctly or not.
And if there exists an optimal adversarial budget for achieving a good trade-off between accuracy and robustness.
The second is In- or Out-Of-Distribution (ID or OOD) ( • in Figure 1). Given an OOD sample that gets high confidence (i.e., is not detected by standard OOD detectors), there are two possible outcomes: certified or not certified. If the number of certified OOD samples is greater than the number of certified ID samples, then we cannot safely use formal verification methods. So, the question arise: can we detect if a test sample is from a different distribution with respect to the training distribution?
Is it possible to verify whether a test sample is ID or OOD?
Approach. To answer these questions, we evaluate robustness certificates with two metrics: the
Area Under the Receiver Operating characteristic curve (AUROC or AUC) Davis & Goadrich (2006) and the False Positive Rate computed at 95% of true positives (FPR95).
In our study, the Receiver Operating Characteristic (ROC) curve is constructed by varying the size of the convex set computed around the input sample.
To give an example, if we consider an ℓ ∞ -norm certificate computed around an input x , the verification will search through an adversarial example ˜ x where the difference ∥ x − ˜ x ∥ ∞ is at maximum ϵ , with ϵ as predefined condition.
Thus, increasing ϵ will likely produce an adversarial example and fail the robustness verification.
The ROC curve is than constructed by varying ϵ and considering different definitions of the True Positive Rate (TPR) and False Positive
Rate (FPR) differently according to the different scenarios: in- and out-of-distribution.
Since verification methods solve a complex optimization problem, they turn out to be relatively slow.
For this reason, our results are constrained to ReLU networks with convolutional or fully connected layers.
On the one hand, ReLU activation allows for more accurate verification, i.e. the convex relaxation is relatively more precise (Gehr et al., 2018).
On the other hand, the small size of the evaluated models allows for shorter verification times.
Main contribution. In this work, we conduct an in-depth analysis on in- and out-of-distribution data for various networks and certificate types, e.g. geometric or norm-based.
To the best of our knowledge, this is the first work that investigates the practical utility of robustness certificates on ID and OOD data.
Our core contributions are summarized as follows:
• First evaluation on the relationship between correctness and certified robustness for clean and perturbed ID and OOD samples.
We empirically show that the number of certified samples is directly related to the accuracy of the network and that robustness certificates are a powerful safety metric for ID data.
• Extensive evaluation of verification methods on various data sets, networks, training procedures and certification types.
• Formal proof that robustness certificates are valid for samples sufficiently far from the training distribution in case of piecewise linear classifiers, e.g. ReLU networks. In the task
of OOD detection, we show that the performance of verification methods is relatively lower than standard OOD detection approaches on normally and adversarially trained networks, and significantly lower for networks trained with OOD samples.
B ACKGROUND & R ELATED W ORK
We define a neural network by a function f : R d → R |K| which maps input samples x ∈ R d to output y ∈ R |K| , where K = { 1 , . . .
, K } is the set of K classes.
We assume a feedforward architecture composed by affine transformations, f ( l ) ( x ) =
W l σ ( l − 1) ( x ) + b ( l ) , for l = 1 , . . .
, L , and followed by ReLU activation functions, σ ( k ) ( x ) = max { 0 , f ( k ) ( x ) } , for k = 1 , . . .
, L − 1 .
In the end, the resulting classifier is obtained as composition of pre- and post-activations, i.e. f ( L ) ( x ) =
W ( L ) σ L − 1 ( x ) + b ( L ) .
In addition, we define all network parameters ( W ( l ) , b ( l ) ) as θ .
Adversarial robustness. Adversarial robustness refers to a model’s ability to resist being fooled.
Formally, given an input x ∈ R d , an adversary is allowed to choose any point from a convex set S ( x ) ⊆
R d .
The classifier is certifiably robust for this input x if the predicted class remains unchanged, i.e. arg max j f j (˜ x ) =
arg max j f j ( x ) , ∀ ˜ x ∈ S ( x ) .
The set S ( x ) can be defined for different specifications, e.g. ℓ p -norm perturbation (Wong & Kolter, 2018), geometric transformations (Balunovic et al., 2019), randomized smoothing (Cohen et al., 2019) and others.
To reduce the vulnerability of the network against adversarially perturbed inputs, the common approach is to train it according to the following min-max optimization problem:
[Equation]
In the other minimization, we consider training f on an ID dataset D in , while in the inner maximization we look for the maximum value of the loss function L that may give us an adversarial sample.
As the inner maximization problem results intractable, most of the existing methods rely on approximations.
For example, Projected Gradient Descent (PGD) (Madry et al., 2017) and Fast Gradient
Sign Method (FGSM) (Goodfellow et al., 2015) are commonly used techniques for improving robustness of a neural network, accomplished by generating adversarial examples and retraining the network with corrected labels.
Robustness certificate. As previously mentioned, a neural network f is certifiably robust for the input x ∈ R d if the prediction for all perturbed versions remains unchanged.
Although an adversarially-trained network is robust to attacks created during training, it may still be vulnerable to unseen new attacks.
To overcome this problem, formal verification methods model the previous statement as a mathematical optimization problem:
[Equation]
where we denote by k the predicted class for x , i.e. k = arg max j f j ( x ) .
We observe that optimization explores the difference by comparing the outputs of the neural network to predict any class other than the initially predicted one.
If the result is positive, we certify the input sample as robust in S ( x ) .
Otherwise, there exists an input that misleads the prediction of the network and the certification fails.
Convex relaxation. To reduce the runtime of the verification process, convex relaxation propagates the input set S ( x ) through the network producing lower and upper bounds at every layer.
This speeds up the entire verification but sacrifices exactness, resulting in a lower bound at the output layer:
[Equation]
where f ∗ denotes the optimal result of the verification, and f , f the lower and upper bounds, respectively.
Current state-of-the-art methods, e.g. GPUPoly (M ¨ uller et al., 2021) or β -CROWN
(Wang et al., 2021), parallelize the computation and propagation of boundaries on the GPU.
Robust OOD detection. OOD detection means to determine whether a sample comes from a learned distribution or not.
Formal robustness guarantees for low network confidence on OOD samples has been recently investigated.
This involves verifying that a predictor assigns low values to all labels for OOD inputs within a defined neighborhood.
This challenge may seem straightforward when using current methods such as softmax calibration (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), or Mahalanobis (Lee et al., 2018), but as highlighted in Hein et al.
(2019), ReLU networks, i.e. networks with fully connected, convolutional, or residual layers, produce arbitrarily high confidence predictions far from the training data.
To overcome this problem, recent contributions have proposed mathematical guarantees to reduce the confidence far from the training distribution.
An early approach of Meinke & Hein (2020) integrates the softmax layer with density estimators based on Gaussian mixture models to distinguish between ID and OOD.
Although they achieve similar OOD detection performance as previous approaches, e.g. Outlier Exposure (OE) (Hendrycks et al., 2019), they can guarantee decreasing confidence far from the training distribution.
In this direction, Bitterwolf et al.
(2020) propose a training approach which uses interval bound propagation (IBP) to derive a provable upper bound on the maximal confidence of the network in a ℓ ∞ -norm of ϵ around a given point.
This procedure leads to classifiers with pointwise guarantees even for near-OOD samples, but IBP produces loose bounds that cause a drop in network accuracy.
Recently, Meinke et al.
(2021) combine a binary discriminator to distinguish between ID and OOD together with previous approaches to preserve high clean accuracy while providing adversarial OOD guarantees.
Although they achieve state-of-the-art performance in different OOD metrics and test distributions, the results are not yet useful in practice as most are still below 50%.
Most of the existing literature focuses on improving empirical robustness to adversarial attacks inside (Goodfellow et al., 2015; Carlini & Wagner, 2017; Madry et al., 2017) and outside (Hein et al., 2019; Bitterwolf et al., 2020; Meinke & Hein, 2020; Meinke et al., 2021) the distribution or on formally demonstrating network stability in the input neighborhood (Gopinath et al., 2018; Balunovic et al., 2019; Wang et al., 2021; M¨uller et al., 2022; 2021).
Another line of work deals with the tradeoff between accuracy and robustness from a training perspective (Yang et al., 2020; Jovanovic et al., 2021;
Liu et al., 2020a) or on specific benchmarks (Hendrycks & Dietterich, 2018; Croce et al., 2021; Wu et al., 2022).
Unlike these, in this work we evaluate how robustness certificates relate to accuracy on ID samples and how they perform on OOD samples.
I N -D ISTRIBUTION A NALYSIS
Here, we evaluate formal verification methods for distinct networks, perturbation types and trainings.
For this purpose, a benchmark analysis is conducted on clean and perturbed in-distribution data to determine the amount of samples that were successfully Certified and Correctly classified (CC) and those that were Certified but Incorrectly classified (CI).
In this analysis, we want to determine if incorrectly classified samples will be robustly certified or not.
In the end, if the ratio of false positives is sufficiently lower or close to zero, we can confidently rely on the robustness verification process as an indicator for correct classification, otherwise not.
[Table]
Assuming that we do not process inputs that cannot be verified and instead rely on a fallback strategy - such as a query to an expert - we can calculate the remaining performance and error rate for the samples actually processed by the model.
Following Henne et al.
(2020), in Table 1 we present similar evaluation metrics.
We define the Certified Correct Ratio (CCR) and Certified Incorrect Ratio (CIR) as the number of CC and CI over the total number of samples N , respectively.
Similarly, we call the ratio of CC over the total number of correctly classified samples C as TPR, and the ratio of CI over the total number of incorrectly classified samples I as FPR.
Thus, with CC and CI, we graphically show the performance of robustness certificates for increasing certification range, while with TPR and FPR, we calculate
AUC as an evaluation metric.
To certify the robustness for geometric and norm-based perturbations we select the convex verifier GPUPoly (M ¨ uller et al., 2021).
3. A SSESSMENT ON UNPERTURBED SAMPLES
Geometric robustness.
In this context, we utilize DeepG (Balunovic et al., 2019) to compute the linear inequality constraints around the set of geometrically transformed images.
In this experiment, we consider clean (unperturbed) test samples and three geometric perturbations: rotation, shearing and scaling.
[Figure]
i [Figure] In Figure 2, we show a comparison between architectures in terms of percentage of CCR and
CIR for increasing rotation values.
In the results, we obtain a comparable difference between CCR and CIR for small perturbation intervals, which remains relatively constant by increasing the range.
The long computation time to generate the convex set for each rotation interval did not allow for a wide range of points.
Here, we graphically show the relationship between robustly certified samples and accuracy.
In the end, the goal is to find a certified interval value that decreases CIR while maintaining a high CCR and thus reliability in certification.
The non-rectilinear behavior of the curves should be attributed to the sampling procedure in DeepG (Balunovic et al., 2019).
This is inherent in the way of how boundaries are constructed around the geometric perturbation.
Before computing the linear inequalities, DeepG operates Monte Carlo sampling to produce geometrically manipulated images inside the given range.
In the case of affine transformations, an interpolation method is needed to reinsert the manipulated pixel into the image grid, and due to its non-linear behavior, sampling is required.
In the context of DeepG, the number of samples (1000) used for the LP solver and the tolerance (0.01) in Lipschitz optimization were constant for increased perturbation values.
Additional experiments are presented in Appendix A.1.
[Figure]
Norm-based robustness.
Here, we evaluate ℓ ∞ -norm robustness certificates for clean test samples, i.e. S ( x ) =
{ ˜ x ∈ R d | ∥ x − ˜ x ∥ ∞ ≤ ϵ, ϵ ≥ 0 } .
In Figure 3, we plot the ROC curve for each network with 400 ϵ values between zero and 0.2, where ϵ is defined as adversarial perturbation budget.
The curve starts from the right hand side for ϵ equal to zero, and increasingly moves to the left hand side.
We see that for small CIR ∼ 0.02, the CCR ∼ 0.
remains surprisingly high.
Within this range, the ROC of
Conv-Plain stays mostly higher than that of Conv-PGD.
We can associate this result to the fact that the plain model has higher accuracy (92.7) with respect to the adversarially trained one (90.8).
In contrast, MLP6x[100], while having a slightly higher accuracy, leads to lower ROC than MLP4x[50].
This highlights that larger fully connected models are less likely to be certified and reduce the performance of robustness certificates.
We show CCR and CIR curves and ROC curves of TPR & FPR in Figure 7 in Appendix A.2.
3. A SSESSMENT ON PERTURBED SAMPLES ( DISTRIBUTIONAL SHIFT )
Here, we test geometrically manipulated in-distribution samples (or distributional shifts).
To this end, we run each network on perturbed samples and evaluate the verification results for the predicted class.
Testing neural networks for distributional shifts, practically assesses their use for real-world applications.
In Table 2, we show the results of AUCs for different networks and perturbations.
The ROC curves are approximately estimated with 400 ϵ values between 0 and 0.02, i.e. 20% of maximum adversarial budget, which push the amount of certified samples to zero for all tests and models.
Similarly to the previous section, TPR and FPR (used to generate ROC curves) are calculated with ℓ ∞ -norm robustness certificates.
In our analysis, we observe that the accuracy on perturbed test sets has decreased and similarly AUC decreased by the same amount.
Convolutional networks maintains higher AUCs
[Table]
with respect to fully connected models and Conv-PGD achieves the best results besides its lower accuracy. Since adversarial training of ℓ ∞ -norm samples helps the same kind of verification, the more certified true positives, the higher the AUC will be.
Both metrics followed a similar trend, enhancing the relationship between accuracy and certified robustness. This result is intended to state that distributional shifts (or perturbed ID samples) are as difficult to verify as the network’s generalization ability is lower. This is independent of the training procedure.
Despite the fact that adversarially-trained networks obtain higher AUCs than plain models (mirroring the results for unperturbed samples), the AUC of adversarially-trained networks decreases analogously with respect to accuracy. Validating the fact that accuracy is correlated with robustness.
Discussion. We summarize the analysis on clean and perturbed ID samples by pointing out that robustness is strongly related to accuracy.
This is visible in Figure 3, where increasing the certification range reduces both correctly and incorrectly classified samples.
Luckily, we obtain more CC than CI samples for small perturbation budgets, showing that robustness certificates are a powerful safety metric for ID data.
To give a numerical example, if the accuracy decreases by ∼ 11% (from 90% to 80%), the error rate drops of ∼ 75% (from 8% to 2%).
We obtained similar results for both types of certification (geometric and norm-based).
Therefore, similar conclusion can be derived for others verification and training methods as well, e.g. randomised smoothing.
In short, the more certification-inclined a network is, the better we can use verification methods as metric to distinguish between correctly and incorrectly classified samples.
O UT -O F -D ISTRIBUTION A NALYSIS
Motivation Hein et al.
(2019) demonstrated that piecewise linear classifiers held high confidence for samples outside the training distribution and post-processing techniques for softmax scores are not able to reduce this confidence.
This is an inherent problem of the network architecture that further leads to the incorrect use of formal verifiers.
A crucial issue on the adoption of such methods is that
OOD samples not only yield high confidence, but are easily verifiable and get certified as correct.
In this section, we theoretically show that robustness certificates hold for samples sufficiently far from the training distribution.
Then, we support our findings with numerical results on OOD test samples.
4. T HEORETICAL A NALYSIS
Here, we formally show that robustness certificates are always valid for piecewise linear classifiers and for samples sufficiently far from the training distribution.
This finding is a derivation of the more general result demonstrated in Hein et al.
, so let us introduce some definitions necessary for the main proof.
We briefly recall the definition of continuous piecewise affine classifiers (Arora et al., 2018), which applies to feedforward neural networks with piecewise affine activation functions, e.g.
ReLU, and linear at the output layer.
Definition 4.1.
A function f : R d → R is called piecewise affine if there exists a finite set of polytopes { Q r }
Mr =1 (referred to as linear regions of f ) such that ∪ Mr =1 Q r = R d and f is an affine function when restricted to every Q r .
This definition applies to all layers performing linear mappings, e.g. fully connected, convolutional, residual layers, skip connections and further maximum and average pooling.
Specifically, given a classifier f : R d → R K , where K is the number of classes, Definition 4.1 applies to each component f
i : R d → R and all K components ( f i )
Ki =1 have the same set of linear regions.
We further extend the definition of ReLU networks as piecewise linear classifiers with the fact that all linear regions are polytopes and thus convex sets (Hein et al., 2019).
Lemma 4.1 (Hein et al.
Let { Q r } Rr =1 be the set of convex polytopes where the ReLUclassifier f : R d → R K is an affine function, meaning for every k ∈ { 1 , . . .
, R } and x ∈ Q k there exists V k ∈ R K × d and c k ∈ R K such that f ( x ) =
V k x + c k .
Thus, for any x ∈ R d \ { 0 } there exists α ∈ R with α > 0 and r ∈ { 1 , . . .
, R } such that βx ∈ Q r for all β ≥ α .
Given Lemma 4.1, we can state our result.
Theorem 4.1.
Let ∪ Mr =1 Q r = R d and f ( x ) =
V r x + a r be the piecewise affine representation of the output of a ReLU network on Q r .
If V r does not contain identical rows for all r = 1 , . .
, R , then for almost any x ∈ R d \ { 0 } , there exists α ∈ R with α > 0 and a predicted class k ∈ K such that:
[Equation]
holds for S ( αx ) ⊂ Q r .
The proof is given in Appendix B.
The main consequence of this theorem consists in the fact that the surrounding of infinitely many samples far enough from the training distribution would be easily certified as robust.
As already noted in Hein et al.
(2019), the constraint on V r is very weak.
On the other side, the fact that S ( αx ) ⊂ Q r is not straightforward, since the definition of S ( x ) may vary depending on the type of certificate one is interested in.
[Figure]
In Figure 4, we show the relationship between α and ϵ for ℓ ∞ -norm robustness certificates, where ϵ is the adversarial budget, i.e. S ( αx ) =
[Equation]
We display the maximum ϵ value for which the certificate holds for increasing α .
One can note that this settings is unlikely in practice as all the images are normalized to be inside the interval [0 , 1] d and therefore ϵ ∈ [0 , 1] .
Besides this, we theoretically demonstrate that samples far enough from the training distribution are expected to be certified and that the certification range augments with the distance.
This is a major problem for the practical use of robustness certificates, since samples are likely to be certified quite far from the training distribution.
Hence, the use of formal verification methods suggests integration with OOD detectors.
4. E XPERIMENTAL A NALYSIS
In this section, we conduct experiments on OOD samples for different datasets, networks and training methods.
The aim is to evaluate the performance of robustness certificates in detecting whether a sample is in- or out-of-distribution.
To this end, we compare the convex verifier GPUPoly (M¨uller et al., 2021) against standard OOD detection methods: MaxSoftmax (Hendrycks & Gimpel, 2017), ODIN (Liang et al., 2018), Mahalanobis (Lee et al., 2018) and Energy (Liu et al., 2020b).
We ran all methods on the entire test set except GPUPoly, which was executed only on the first 1000 test samples.
This is due to the incredibly long run time of validating a large amount of samples for a large range of ϵ values.
The ROC curve for GPUPoly has been computed by varying the adversarial budget ϵ .
Here, we consider as true positives all certified samples from the ID test set, and as false positives all certified samples from the OOD test set.
We define 4 000 ϵ values equidistant between 0 and 0.2, i.e. 20% of maximum adversarial budget, which push the amount of certified samples to zero for all tests and models.
As an example, verifying 1 000 images on our largest network ( neurons) with GPUPoly takes about 20 minutes per single ϵ .
We conduct our experiments on a
Nvidia GPU RTX 3090.
[-In the RGB, we have RGB images of size 32x32: CIFAR10/
-]
Grayscale Category.
In Table 3, we show the results on grayscale datasets, where we use MNIST as ID dataset.
Given the limited size of the models, PGD and FGSM attacks prevent convergence during training, so we evaluate only Plain, OE and Randomized trained networks in this analysis.
In case of OE, we consider two datasets: OrganAMNIST and FMNIST.
As might be expected, [Table] networks trained with OE perform significantly better than those trained with Plain or Randomized.
The results with FMNIST as OOD training set compared to OrganAMNIST are surprisingly close to optimum in KMNIST for all standard OOD detection methods.
In the context of GPUPoly, we observe better results compared to other methods for convolutional networks in the KMNIST dataset, and definitely lower results for fully connected models.
On the one hand, GPUPoly struggles to certify samples in distribution, leading to inferior results than standard OOD detection methods.
On the other hand, for FPR at 95% of true positives, we obtain more certified OOD samples, empirically validating the hypothesis that verification methods easily certified samples far enough from the training distribution.
The hardness of verifying OE trained networks should be related to the slightly thinner decision boundaries induced during the training procedure.
Surprisingly, the randomized trained convolutional network performed slightly better than its plain counterpart.
In appendix C.1, we report the ROC curves for convolutional networks.
[Table]
Mahalanobis (
[Table]
[Table]
[Table]
Energy
[Table]
[Table]
GPUPoly
[Table]
[Table]
[Table]
Randomized (
[Table]
=
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] RGB Category. Here, we test two convolutional networks of different sizes: ConvSmall and ConvMed, trained on two ID datasets: GTSRB and CIFAR10.
In general, the clean accuracy is remarkably low compared to state-of-the-art models, and slightly lower for adversarially trained networks than for plain models, but it is aligned with related work on verification methods (see, e.g., (M¨uller et al., 2021; 2022)).
In Table 4, we show the results for the ConvMed model.
In this setting, we trained each network on all 43 classes of the GTSRB dataset.
As a consequence, we obtain lower accuracy with respect to the models of section 3 trained on just the first 10 classes.
Similarly to the grayscale category, standard OOD detection methods perform likewise.
In the case of OE, GPUPoly certifies more
OOD than ID samples, drawing the AUC below the random guess value of 0.5.
On the one side, adversarial training procedures, such as PGD, FGSM and randomized, do not seem to help the verification process, resulting in substandard performance for GPUPoly.
On the other side, standard
OOD detection methods are slightly affected.
[Figure]
In Figure 5, we show the ROC curve of ConvMed trained with OE on the GTSRB as ID and ImageNet cropped as OOD training sets and tested on SVHN.
As noted,
GPUPoly certifies more OOD than ID samples.
We discuss this behavior and affiliate it with a couple of reasons.
First, OE induces an irregular gradient that leads the verification process to fail in the case of both ID and OOD samples.
This decreases the number of robustness certified samples, and affects TPR and FPR equally.
Second, OE reduces the accuracy on ID samples, leaving more stable gradients and a larger prediction space for
OOD samples.
This augments the number of OOD certified samples, and increases the FPR.
In the end, this experiment empirically validates the theoretical results discussed above.
Additional results are shown in Appendix C.
Discussion.
In this analysis, we have shown that robustness certificates perform similarly than standard OOD detection methods for adversarially-trained networks, both on grayscale and RGB images.
The results are completely different for networks trained with OE, where the ROC is lower than the random guess.
This highlights the problematic nature of formal verification methods in easily certifying OOD samples for networks trained to be OOD aware.
Hence, the use of robustness certificates on piecewise linear classifiers needs to be complemented with additional safety measures.
C ONCLUSION
In this paper, we explored the robustness of ReLU networks through an in-depth analysis of clean and perturbed samples inside and outside the distribution.
We used convex verification methods to robustly certify the network prediction.
By varying the adversarial perturbation budget ϵ , we constructed ROC curves.
In our in-distribution analysis, we showed that there is a strong correlation between certified robustness and accuracy for clean and perturbed samples.
In this context, formal verification methods prove to be a useful error-reduction metric leading to an overall increase in reliability.
Completely different are the results for the OOD analysis, where robustness certificates demonstrate their unreliability compared to standard OOD detection methods.
In such analysis, we proved theoretically that samples far enough from the training distribution are easily certified for
ReLU classifiers.
In our experiments, we validate the theoretical findings through an extensive analysis.
The results suggest the addition of OOD detection measures for the practical use of robustness certificates in real applications.
In the end, verification methods can be a piece of the puzzle toward trustworthy AI.
As a future perspective, it would be interesting to combine these methods with standard OOD detection methods in a useful way.
However, it is still unclear how to distinguish between the two for a given sample.
A E XTENSION TO THE IN - DISTRIBUTION ANALYSIS .
[Table] Architectures & Training
We train a total of four neural networks on the first ten classes of the GTSRB dataset (Houben et al., 2013). Two Fully Connected (FC) Multilayer Perceptron (MLP): MLP 4 x [50] and MLP 6 x [100] normally trained (plain), and two convolutional neural networks: one trained with PGD (Madry et al., 2017) attacks ( ℓ ∞ -norm attacks with ϵ = 0 . 01 for a maximum of 40 steps), and the other normally trained, which are denoted as Conv-PGD and Conv-Plain, respectively.
The clean accuracy (ACC) and other parameters are reported in Table 5.
To achieve higher accuracy with such small networks we reduced the number of classes to 10 to decrease the amount of features to be learned by the network.
A. G EOMETRIC R OBUSTNESS
Here, we report further geometric robustness certificates of the GTSRB test set.
In Figure 6, we show results for shearing and scaling.
In this experiment, we fed each network with clean test samples.
As previously discussed, the ratio between CCR and CIR reflects the network’s accuracy.
Differently from rotations, shearing and scaling result to be less prone to be certified.
The lower part of the graph highlights the erratic behavior of FPR, attributable to DeepG’s sampling procedure. [Figure] e [Figure]
i
[Figure]
A. N ORM - BASED R OBUSTNESS
In Figure 7a, we show the percentage of verified samples for increasing ϵ of the ℓ ∞ -norm, where ϵ is defined as adversarial perturbation budget.
As noted, convolutional neural networks yield better results with respect to fully connected ones and Conv-PGD turns out to be the best.
In addition, we see that for very small ϵ ∼ 0 .
003 , the ∼ 1% of CI is comparatively very small respect to the ∼ 75% of CC for Conv-Plain.
Instead, at ∼ 2% of CI we have ∼ 50% of CC for fully connected models.
In Figure 7b
[Figure]
we plot the ROC curves for TPR and FPR.
We observe that the ROC of Conv-PGD remains mostly higher than that of Conv-Plain.
We attribute this result to the fact that adversarially trained networks are more easily certifiable than simple models, which leads to generally higher
TPR results.
Similarly to the case of CCR & CIR curves, MLP6x[100] demonstrates to be less prone to be certified and results in lower AUC with respect to MLP4x[50].
B P ROOF OF T HEOREM 4.
Theorem 4.1 Let ∪ Mr =1 Q r = R d and f ( x ) =
V l x + a l be the piecewise affine representation of the output of a ReLU network, then for almost any x ∈ R d , there exists α ∈ R with α > 0 and a [Figure] predicted class k ∈ K such that:
[Equation]
holds for S ( αx ) ⊂ Q r .
Proof.
By Lemma 4.1, there exists a region Q r , with r ∈ { 1 , . . .
, R } and β > 0 such that for all α ≥ β
we have αx ∈ Q r .
Given that S ( αx ) ⊂ Q r and since z ∈ S ( αx ) we have that z ∈ Q r .
Let f ( z ) =
V r z + a r be the affine form of the ReLU classifier f on Q r .
Let k ∗ = arg max k ⟨ v rk , z ⟩ , where v rk is the k -th row of V r .
Given the fact that V r does not has identical rows, i.e. v rl ̸ = v rm for l ̸ = m , the maximum is unique up to zero.
If the maximum is unique, it holds for sufficiently large α [Equation]
[Equation]
C E XTENSION TO THE OUT - OF - DISTRIBUTION EXPERIMENTAL ANALYSIS .
Datasets.
In order to obtain a more comprehensive and fair evaluation, we consider two categories of image datasets: grayscale and RGB .
In the grayscale category we place datasets with grayscale images of size 28x28: MNIST (LeCun, 1998), EMNIST (Cohen et al., 2017), KMNIST (i.e. KuzushijiMNIST Clanuwat et al.
and FMNIST (i.e., Fashion-MNIST Xiao et al.
{+In the RGB, we have RGB images of size 32x32: CIFAR10/
+}
(Krizhevsky et al., 2009), SVHN (Sermanet et al., 2012) and GTSRB (Houben et al., 2013). In addition, we utilize OrganAMNIST from MedMNIST (Yang et al., 2021) and ImageNet Cropped (C) (Deng et al., 2009) for training OOD aware models. For each category, we normalise all datasets by the same mean and standard deviation of the ID training set. [Table]
Architectures & Training.
In Table 6, we describe the network architectures, activation type and number of neurons for each dataset category. Evaluation is carried out on different training procedures. Networks trained only with clean training data are called Plain . Adversarially trained networks are
PGD (Madry et al., 2017) or FGSM (Goodfellow et al., 2015), where ϵ is the adversarial perturbation budget. Randomized are networks trained with randomized smoothing (Cohen et al., 2019) where σ is the standard deviation. Lastly, OE stands for Outlier Exposure (Hendrycks et al., 2019), where we insert the OOD training set in parentheses. Analysis.
Here, we extend our evaluation of robustness certificates for OOD detection.
Below, we show ROC curves and other results for different trainings, networks and datasets.
As previously mentioned, GPUPoly is used with ℓ ∞ -norm robustness certificates and a range of 4 000 values of ϵ between 0 and 0.2.
Instead, all other methods uses a range of 10 e 5 for the threshold.
C. G RAYSCALE C ATEGORY E XTENSION
[Figure]
In Figure 8, we report the ROC curves for convolutional networks trained normally and with randomized smoothing on the MNIST dataset.
We observe that GPUPoly( ℓ ∞ ) perform relatively better than standard OOD detection methods on EMNIST and KMNIST.
C. C OLORED C ATEGORY E XTENSION
In Figure
[Table]
we show the ROC curves for the ConvMed model trained with OE (ImageNet cropped),
PGD and randomized smoothing.
In the context of OE [Table] we clearly see that the ROC curves of
GPUPoly are below the average value of 0.5.
Besides his average performance on PGD and randomized trained networks, the results graphically demonstrate that OOD samples are more likely to be certified than ID samples for OOD aware networks [Table] [Table]
ConvMed/FGSM( ϵ = 1 / 255 ) 57.
Mahalanobis 57.
55.2 Softmax 64.
90.2 ODIN 65. 78.5 Energy 64. 96.2 GPUPoly( ℓ ∞ ) 58. ConvMed/PGD( ϵ = 1 / 255 ) 56.
Mahalanobis 57.
41.3 Softmax 64.
92.1 ODIN 64.
73.9 Energy 64.
98.1 GPUPoly( ℓ ∞ ) 58.
[Figure]
In Table
[Figure]
we report the results for the RGB category with CIFAR10 as ID dataset.
In this evaluation, we compare PGD, FGSM and normally trained convolutional networks.
Similarly to the networks trained on the GTSRB dataset, we observe similar performances as standard OOD detection methods.
As previously mentioned, the accuracy is generally low compared to state-of-the-art networks, but is aligned with related work on verification methods (M ¨ uller et al., 2021; 2022) [Figure]
MahalanobisSoftmaxODINEnergyGPUPoly
[Figure] In Figure 10 and Figure [Figure] we report the ROC curves for ConvSmall and ConvMed, respectively.
All networks are trained with CIFAR10 as ID dataset. [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]