Certified defences hurt generalisation
Abstract
In recent years, much work has been devoted to designing certified defences for neural networks, i.e., methods for learning neural networks that are provably robust to certain adversarial perturbations.
Due to the non-convexity of the problem, dominant approaches in this area rely on convex approximations, which are inherently loose.
In this paper, we question the effectiveness of such approaches for realistic computer vision tasks.
First, we provide extensive empirical evidence to show that certified defences suffer not only worse accuracy but also worse robustness and fairness than empirical defences.
We hypothesise that the reason for why certified defences suffer in generalisation is (i) the large number of relaxed non-convex constraints and (ii) the strong alignment between the adversarial perturbations and the "signal" direction.
We provide a combination of theoretical and experimental evidence to support these hypotheses.
Introduction
Deep neural networks have been shown to be vulnerable to adversarial attacks: imperceptible perturbations to the input can fool state-of-the-art classifiers [13, 27, 21].
The existence of adversarial examples raises serious security concerns in many safety-critical applications [29, 10] and so robustness to adversarial attacks is becoming a crucial design goal for modern deep learning architectures.
In practice, robustness against many different types of perturbations may be desired depending on the domain of application.
Hence, in order to build robust models, we need to first define a threat model for the adversary, i e., a set of perturbations B .
The most commonly used threat models in the literature are norm-bounded perturbations, where B ϵ,p = { δ : ∥
δ ∥ p ≤ ϵ } is the ℓ p -ball with radius ϵ centred around the origin.
In this paper, we focus on B ϵ, 2 which, for ease of notation, we will represent as B ϵ .
For any distribution D , neural network model f θ parameterised by θ , and loss function L , our goal is to find a model which solves the following robust optimisation problem:
[Equation]
We call R ϵ ( θ ) the robust error when L is the 0-1 loss function.
In practice, as the distribution D is unknown, we instead minimise the empirical robust error on a finite dataset D sampled from D .
Unfortunately, in the case of neural networks, the inner maximisation is a non-convex optimisation problem and prohibitively hard to solve from a computational perspective [16, 30].
Therefore, two kinds of approaches are widely used to efficiently solve it: empirical defences that provide a lower bound on the solution and certified defences that provide an upper bound.
Among empirical defences, Adversarial Training (AT) [13, 20] is one of the few that has stood the test of time.
AT minimises the worst-case empirical loss in Equation (1) by approximately solving the inner-maximisation problem with first-order gradient-based optimisation methods.
However, despite its simplicity and computational efficiency, owing to its heuristic nature, AT does not provide any robustness guarantee. In many safety critical domains, such guarantees are of immense importance.
COAP AT
CROWN-IBP
Perturbation budget ( ϵ /255)
S t a n d a r d e rr o r (a) Standard error 9 18  0.
COAP AT
CROWN-IBP
Perturbation budget ( ϵ /255)
R o b u s t e rr o r (b) Robust error 9 18  0.
COAP AT
CROWN-IBP
Perturbation budget ( ϵ /255)
A cc .
d i s c .
(c) Unfairness
To address this limitation, recently, there has been significant interest in designing certified defences, i.e., methods for learning neural networks that are provably robust to norm-bounded perturbations on the training data.
Many recent works [31, 23, 7, 37] have proposed to solve a convex relaxation of the inner-maximisation problem by relaxing the non-convex ReLU constraint sets with convex ones.
Despite all of these progresses, certified defences based on convex relaxations suffer from an inherent flaw: the upper bound they provide on the robust error is far from being tight due to the looseness of the convex relaxation [25].
In this paper, we argue that the fundamental looseness of convex relaxations hinders the practical effectiveness of current certified defences.
In particular, as shown in Figure 1, certified defences suffer significantly worse accuracy, robustness, and fairness on the test data compared to adversarial training.
Our contributions are as follows:
• In Section 2, we show that current certified defences hurt accuracy, robustness, and fairness across a range of ℓ 2 -ball perturbations on real-world vision datasets like MNIST and CIFAR-10.
• In Section 3, we provide experimental evidence that certified defences hurt generalisation because of (i) the large number of relaxed non-convex constraints and (ii) the strong alignment between the adversarial perturbations and the signal direction.
In Section 4, we prove, in a simplified high-dimensional classification setting, that certified defences yield higher robust risk than adversarial training when the adversarial perturbation aligns with the signal direction.
Certified defences hurt generalisation on real-world data
In this section, we show that certified defences hurt standard error, robust error, and fairness on two common computer vision datasets: MNIST [19] and CIFAR-10 [18].
Among certified defences, we consider the convex outer adversarial polytope (COAP) approach [32, 31], which achieves state-ofthe-art certified robustness under ℓ 2 -ball perturbations.
Additionally, we consider CROWN-IBP [36, 34], which uses the tightest convex relaxation CROWN [36] and achieves state-of-the-art certified robustness under ℓ ∞ -ball perturbations.
Among empirical defences, we consider adversarial training (AT) [20, 13], which is one of the most popular and effective defences to date.
Models and robust evaluation We consider the ℓ 2 -ball perturbations threat model.
To reliably evaluate the robust error, we use the strongest version of AutoAttack (AA+) [4].
For CIFAR-10, we train a residual network (ResNet) and for MNIST we train a large convolutional neural network (CNN).
Both architectures were introduced in Wong et al.
[32] as standard benchmarks for certified defences.
We refer the reader to Appendix D.3 for experimental details.
Certified defences hurt standard and robust error Several studies have shown that adversarial training may lead to an increase in standard error when compared with standard training [24, 28, 35].
Here, we observe the same phenomenon to a much higher degree in certified defences.
Specifically, our experimental results show that certified defences not only suffer worse standard error but also worse robust error than adversarial training.
First, we observe on both MNIST and CIFAR-10 in
Figures 2a and 2c, respectively, that for increasing perturbation budget, the standard error gap between
[Figure]
Perturbation budget
[Figure]
S t a n d a r d e rr o r (a)
[Figure]
COAPATCROWN
[Figure]
IBP
Perturbation budget
[Figure]
[Table]
[Figure]
u
[Figure]
t e rr
[Figure] [Figure]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
r
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
cc
[Figure]
(e)
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
budget
[Table]
[Figure]
[Table]
[Figure]
(f) MNISTRobustunfairness
[Table] [Table]
[Figure]
[Table] [Table]
[Table] [Table]
COAPATCROWN
[Table]
Perturbation budget ( ϵ /255)
R o b u s t a cc .
d i s c .
(h) CIFAR-10Robustunfairness certified (CROWN-IBP, COAP) and empirical defences (AT) increases.
In particular, the gap reaches almost 90% for CROWN-IBP on MNIST when ϵ
= 1 . 5 .
Secondly, we observe that the robust error gap increases with increasing perturbation budget for both MNIST and CIFAR-10 in Figures 2b and 2d, respectively.
In particular, the gap reaches almost 40% for the largest perturbation budgets.
Certified defences hurt fairness Previously, we showed that certified defences hurt both standard and robust generalisation.
Taking it one step further, we show that certified defences (CROWN-IBP,
COAP) suffer significantly worse fairness than empirical defences (AT).
Let R ( θ ) be the standard error of the classifier f θ and R k ( θ ) the standard error conditioned on the class label k .
We measure the degree of unfairness as:
[Equation]
Using the terminology in Sanyal et al.
[26], we refer to this metric as accuracy discrepancy .
However, we expect our results to translate to other related fairness metrics as well [6, 8, 14, 15].
Additionally, we consider the discrepancy in robust accuracy, as it was observed in Xu et al.
[33] that adversarial defences may induce a large discrepancy of robustness among different classes.
We refer to this metric as robust accuracy discrepancy and it corresponds to replacing standard error with the robust error in Equation (2).
We present our experimental results comparing the fairness of certified and empirical defences.
For
MNIST, we observe in Figure 2e and 2f that COAP and CROWN-IBP have a significant discrepancy for both standard and robust accuracy.
For large perturbations, these methods obtain 100% discrepancy, indicating that their accuracy on the worst class can be as low as 0% .
By contrast, AT preserves fairness for both standard and robust accuracy much better.
In particular, the discrepancy for standard accuracy is always less than 2% for all perturbation budgets considered.
Similarly, for [-CIFAR-
 AT-]CIFARAT maintains a constant accuracy discrepancy around 20% for all perturbation budgets considered, whereas for certified defences it steadily increases with the perturbation budget, reaching above 80% .
Additionally, for robust accuracy, we observe a discrepancy gap of 35% between the best certified and empirical defences for the largest perturbation budget considered.
Developing intuition on synthetic datasets
In this section, we hypothesise that certified defences hurt robust and standard generalisation because of (i) the large number of relaxed non-convex constraints and (ii) the strong alignment between the
COAPAT
ST
Perturbation budget ϵ
R o b u s t e rr o r (a) Linearly separable distribution 0.
COAPATST
Perturbation budget ϵ
R o b u s t e rr o r (b) Concentric spheres distribution 0.
Linear Spheres
Perturbation budget ϵ
A c t i v e n e u r o n s ( % ) (c) COAP active neurons adversarial perturbations and the signal direction.
We investigate these hypotheses on more controlled settings.
In particular, we consider two synthetic data distributions: a linearly separable distribution as in Clarysse et al.
[3], which is similar to distributions studied in Nagarajan and Kolter [22], Tsipras et al.
[28], and the concentric spheres distribution studied in Gilmer et al.
[12], Nagarajan and Kolter [22].
Data and threat models Similarly to the previous section, we focus on ℓ 2 -ball perturbations of size ϵ .
As for distributions, we consider the linearly separable distribution where first, the label y ∈ { +1 , − 1 } is drawn with equal probability.
Then, for some γ > 0 , the covariate vector is created as x = [ γ sgn( y ); ˜ x ] , where ˜ x ∈ R d − 1 is a random vector drawn from a standard normal distribution ˜ x ∼ N (cid:0) 0 , σ 2 I d − 1 (cid:1) and [; ] represents concatenation.
We sample the concentric spheres dataset as follows; for 0 < R 1 < R − 1 , we first draw a binary label y ∈ { +1 , − 1 } with equal probability and then the covariate vector x ∈ R d is distributed uniformly on the sphere of radius R y .
Observe that achieving a low test error on the concentric spheres distribution requires a non-linear classifier.
In Figures 3a and 3b, we plot the robust error of standard training (ST), adversarial training (AT), and certified training (COAP) on the linear and concentric spheres distributions respectively.
We see that in contrast to the linear setting, COAP has a much higher robust error on the concentric spheres distribution than AT and ST, where the gap increases with increasing perturbation budget ϵ .
Below, we provide intuition as to why COAP has a much higher robust error than AT on the concentric spheres distribution.
The intuition is two-fold: first of all COAP relaxes the non-convex ReLU constraints for all neurons that activate within the perturbation set, i.e., there exists δ ∈ B ϵ for which the input to the neuron equals 0 .
Hence, the larger the percentage of relaxed neurons, the worse the approximation.
This is formally captured by Theorem A.2 in Appendix A.
Secondly, the ℓ 2 -ball perturbations are significantly aligned with the signal direction, meaning that they effectively reduce the information about the label in the data.
Applying an approximation in this direction yields poor generalisation.
We prove this in
Theorem 4.1 for the linearly separable distribution.
COAP relaxes many constraints on the concentric spheres In Figure 3c we empirically show that COAP convexly approximates a large number of constraints when training on the concentric spheres distribution.
We plot the percentage of active neurons on the concentric spheres and linear distributions against increasing perturbation budgets: the percentage is much higher for the concentric spheres than for the linearly separable distribution and increases with perturbation budget ϵ .
Indeed, the complex spherical decision boundary requires much more active neurons compared to the linear decision boundary which only needs 1 active neuron.
ℓ 2 -ball perturbations align with the signal direction We empirically show that ℓ 2 -ball perturbations align with the signal direction on the concentric spheres distribution. Note that for a point x drawn from the concentric spheres distribution, the signal direction is given by y x ∥ x ∥ 2 (see Figure 4a for a 2D visualization).
In Figure 4b, we plot the cosine distance between the ℓ 2 -perturbations
[Figure]
s i n e s i m i l a r i t y
[Figure]
[Figure] [Figure]
[Figure]
COAPAT
[Figure]
S t a n d a r d e rr o r
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
COAPAT
[Figure]
R o b u s t
e rr o r
[Figure]
computed on the training set, and the signal direction.
Comparing Figures 4b to 4d, we see that during the early stages of training, the ℓ 2 -ball perturbations are not aligned with the signal direction and the robust and standard errors for COAP are similar to AT.
However, after some epochs, when the ℓ 2 -ball perturbations start to align with the signal direction, both the robust and standard error gaps between
COAP and AT increase.
This provides evidence that, as training progresses, ℓ 2 -ball perturbations become significantly aligned with the signal direction and the generalisation gap worsens.
Approximations along the signal direction hurt generalisation
In this section, we further investigate our hypothesis that certified defences hurt generalisation when adversarial perturbations are aligned with the signal direction.
In particular, we study the linearly separable distribution from the previous section and assume that the adversarial attacks concentrate all of their perturbation budget along the signal direction.
In Theorem 4.1, we prove for a simple neural network that, in high dimensions, certified defences (COAP) yield higher robust error than empirical defences (AT) for large perturbation budgets.
We then corroborate our theoretical results with extensive experimental evidence on synthetic data.
Data and threat models We consider the linearly separable distribution described in Section 3.
As for the threat model, we consider signal-directed attacks that efficiently concentrate their attack budget on the signal in the input.
Since the signal direction corresponds to the first component of the data, we define the set of allowed perturbations as:
[Equation]
where e 1 is the standard basis vector of the first coordinate.
Further, as the original formulation of
COAP only allows ℓ p -adversaries, we provide an extension of COAP that covers our theoretical and experimental setting in Appendix A.
One gradient step training We consider the hypothesis class to be the set of one-neuron shallow neural networks f θ : R d −→ R , defined by:
[Equation]
where x ∈ R d , θ ∈ R d , a ∈ R , b ∈ R and the only trainable parameter is θ 1 .
Note that as our distribution is linearly separable, our hypothesis class includes the ground truth.
We study the early phase of neural network optimisation.
Under structural assumptions on the data, it has been proved that one gradient step with sufficiently large learning rate can drastically decrease the training loss [2] and extract task-relevant features [11, 5].
A similar setting was also studied recently in Ba et al.
[1] for the MSE loss in the high-dimensional asymptotic limit.
Here, we focus on the classification setting with binary cross-entropy loss.
Below we state our main theorem.
[Figure]
Theorem 4.1.
Let ¯ θ
and ˜ θ be the network parameters after one step of gradient descent with respect to AT and COAP objectives.
Let,
[Equation]
where θ are the network parameters at initialization.
Then, COAP yields higher robust risk than AT
[Equation]
Theorem 4.1 relies on two main assumptions.
The first is an assumption on the data dimensionality and the initialisation of the network parameters θ .
For instance, if t he network parameters are initialised sampling from a standard multivariate gaussian θ ∼ N (0 , I d ) , then we know that ∥ θ ∥concentrates around √ d with high probability.
Hence, the assumption is satisfied when the data dimensionality d is sufficiently high.
Further, the second assumption requires that the perturbation budget ϵ is sufficiently close to the separation margin γ .
This is consistent with the experimental evidence we presented so far, as the generalisation of certified defences significantly worsen for large perturbation budgets.
Synthetic experiments We corroborate our theory with experimental evidence using a one-hidden layer neural network with 100 neurons.
In particular, we investigate the effect of perturbation budget ϵ on generalisation for three different models: standard training (ST), adversarial training (AT) [20, 13] and convex outer adversarial polytope (COAP) [31, 32].
In Figure 5, we plot robust and standard errors for both small and large sample size regimes as the perturbation budget ϵ increases.
The generalisation gap in the small sample size regime between standard and adversarial training was already observed in Clarysse et al.
[3] for linear classifiers.
Here, we observe a further generalisation gap between AT and COAP in both small and large sam ple size regimes, which surprisingly worsens in the large sample regime.
Conclusions
In this paper, we show that certified defences can hurt accuracy, robustness and fairness for realistic datasets and adversarial perturbations.
Further, we develop intuition on synthetic datasets for why certified defences hurt generalisation, combining both theoretical and experimental evidence.
We believe that shedding light on the performance gap between empirical and certified defences will not only provide us with a clearer picture of the trade-offs observed in practice but also lead to better approaches for adversarial robustness.
A Certified defences for signal-directed adversaries
We now formulate the convex outer adversarial polytope (COAP) [31] for adversaries that concentrate all their budget along the signal direction in the input.
Our derivation can be seen as an extension of Wong and Kolter [31], Erdemir et al. Data and threat model We consider the linearly separable distribution described in Section 3.
First, the label y ∈ { +1 , − 1 } is drawn with equal probability. Then, for some γ > 0 , the covariate vector is x = [ γ sgn( y ); ˜ x ] , where ˜ x ∈ R d − 1 is a random vector drawn from a standard normal distribution ˜ x ∼ N (cid:0) 0 , σ 2 I d − 1 (cid:1) and [; ] represents concatenation.
As for the threat model, we consider signal-directed attacks that efficiently concentrate their attack budget on the signal in the input.
Since the signal direction corresponds to the first component of the data, we define the set of allowed perturbations as:
[Equation]
) where e 1 is the standard basis vector of the first coordinate.
Network structure For the rest of this section we consider a 2 layer feed-forward ReLU network. In particular, we define f θ ( x ) : R d → R 2 as follows:
[Equation]
where x ∈ R d , z 1 ∈ B ϵ ( x ) , W 1 and W 2 , are linear operators and we define the set of network parameters as θ
= { W i , b i } i =1 , 2 .
Constructing the convex outer bound We define the adversarial polytope Z ϵ ( x ) as the set of all final-layer activations attainable by perturbing x with some ˜ x ∈ B ϵ ( x ) :
[Equation]
Our approach will be to construct a convex outer bound on this adversarial polytope: if no adversarial example exists in this outer approximation, then we are guaranteed that no adversarial example exists in the original polytope.
We relax the ReLU activations z = ReLU (ˆ z ) with their convex envelopes:
[Equation]
where u and ℓ are respectively the pre-activations ˆ z upper and lower bounds.
First, we address the problem of obtaining the upper and lower bounds u and ℓ for the pre-activations ˆ z .
Specifically, the following proposition gives a closed form solution for ℓ and u .
Proposition A.1.
Consider the neural network f θ defined in Equation (8) .
Let w 1 be the first column of W 1 .
Then, for a data point x and perturbation budget ϵ , we have the following element-wise bounds on the pre-activation vector ˆ z 2 :
[Equation]
where
[Equation]
Proof.
Given a data point x and perturbation budget ϵ , let ˜ x
= x + δ be the perturbed input to the network.
First, we find an upper bound the pre-activations values ˆ z 2 :
[Equation]
In particular, we want to solve the following optimisation problem for each component of the pre-activation vector:
[Equation]
(14) where u will be the vector containing element-wise upper bounds.
Note that δ = βe 1 , thus the optimisation problem can be rewritten as:
[Equation]
where w 1 is the first column of W 1 .
The vector of upper bounds will then be:
[Equation]
Along the same lines, we can derive the vector of lower bounds ℓ :
[Equation]
Next, we define the outer bound on the adversarial polytope we get from relaxing ReLU constraints as ˜ Z ϵ ( x ) .
Given a data point x with known label y , we can formulate the problem of finding an adversarial example with a linear program as follows:
[Equation]
where ¯ y is the binary negation of y . Note that if we solve this linear program and find that the objective is positive, then we know that no input perturbation within the threat model can misclassify the example. However, solving the linear program in Equation (18) for every example in the dataset is intractable. Therefore, we consider the dual formulation and take a feasible solution.
In the following theorem, we state the dual problem formulation of the linear program in Equation (18).
v Theorem A.2.
The dual of the linear program (18) can be written as
[Equation]
where (cid:101) J ϵ
( x, ν 1 , ν 2 , ν 3 ) is equal to −
[Equation]
(20) and g θ is a one-hidden layer neural network given by the equations
[Equation]
where I − , I +
and I denote the sets of activations in the hidden layer where ℓ and u are both negative, both positive or span zero, respectively [Equation]
Proof.
Consider a data point x and let ˜ x
= x + δ be the adversarial perturbed data point.
First [Equation] we explicit all the constraints for the linear program defined in (18): [Equation]
t
[Equation]
where I − , I +
and I denote the sets of activations in the hidden layer where ℓ and u are both negative, both positive [Equation] or span zero respectively.
In order to compute the dual of this problem, we associate the following Lagrangian variables with each of the constraints:
[Equation]
note that we do not define explicit dual variables for [ z 2 ] j = 0 and [ z 2 ]
j = [ˆ z 2 ]
j as we can easily eliminate them.
We write the Lagrangian as follows:
[Equation]
and we take the infimum
[Equation]
[Equation]
Now
[Equation]
we can compute the infimum for the ψ ⊤ δ term:
[Equation]
(26) and since for all the other terms the infimum of a linear function is −∞ , except in the special case when it is identically zero
[Equation]
the infimum of L ( · ) becomes:
[Equation]
if conditions −∞ else (27) where the conditions to satisfy are:
[Equation]
Thus, we can rewrite the dual problem as follows:
[Equation]
Note that the dual variable λ corresponds to the upper bounds in the convex ReLU relaxation, while µ and τ correspond to the lower bounds.
By the complementarity property, we know that at the optimal solution, these variables will be zero if the ReLU constraint is non-tight, or non-zero if the ReLU constraint is tight.
Since the upper and lower bounds cannot be tight simultaneously, either λ or µ + τ must be zero.
This means that at the optimal solution to the dual problem we can decompose [ W ⊤ 2 ν 3 ] j into positive and negative parts since ( u j − ℓ j ) λ j ≥ 0 and τ j + µ j ≥ 0 :
[Equation]
30) combining this with the constraint [ ν 2 ] j = u
j λ j − τ j leads to
[Equation]
ν 2 ] j = [Equation] 31) for j ∈ I and 0 ≤ α j ≤ 1 .
Hence, we have that:
[Equation]
Now, we denote ˆ ν 1 = − ψ to make our notation consistent, and putting all of this together the dual objective becomes:
[Equation]
and the final dual problem:
[Equation]
[Equation]
Most importantly, the theorem states that we can represent the dual problem as a linear back propagation network, which provides a tractable solution for a lower bound on the primal objective.
In practice, rather than solving the exact dual problem, we choose a fixed dual feasible solution:
[Equation]
B Theoretical results for signal-directed adversaries
In this section, we study a simplified one-neuron neural network. We consider the linearly separable distribution from the previous section and assume that the adversarial attacks concentrate all of their perturbation budget along the signal direction.
For a formal definition of the data and threat models we refer the reader to Appendix A.
One-neuron neural network We consider the hypothesis class to be the set of one-neuron shallow neural networks f θ : R d −→ R , defined by:
[Equation]
where x ∈ R d , θ ∈ R d , a ∈ R , b ∈ R and the only trainable parameter is θ 1 .
Note that as our distribution is linearly separable, our hypothesis class includes the ground truth.
Further, we focus on the classification setting with binary cross-entropy loss:
[Equation]
where σ ( · ) is the sigmoid function.
B.1 Adversarial training
The basic idea behind adversarial training is to update the network parameters according to the following rule:
[Equation] This is usually done by applying some first-order approximation to the maximisation problem. However, for our simplified network we can analytically compute the gradient. First of all, note that when L is the binary cross-entropy loss function we can rewrite the maximisation problem as follows:
[Equation]
In particular, if J ϵ is strictly positive then no adversarial example exists that fools the network.
Further, note that this formulation is closely related to the objective considered in Appendix A for the convex outer adversarial polytope.
This will be useful when comparing COAP and AT gradients.
Below we provide the gradient of the adversarial training objective w.r.t.
the network parameters θ .
Proposition B.1.
Consider the neural network f θ defined in Equation (36) and the threat model B ϵ defined in Equation (7) .
Let L be the binary cross-entropy loss function, as defined in Equation (37) .
Then, we have:
[Equation]
where ℓ = θ ⊤ x − ϵ | θ 1 | and u = θ ⊤ x + ϵ | θ 1 | are respectively lower and upper bounds on the ReLU inputs.
Proof.
Given a data point x with known label y ∈ {− 1 , 1 } , when L is the binary cross-entropy loss function we have:
[Equation]
For our simplified network we can analytically compute a closed form solution of the minimisation problem:
[Equation]
where ℓ = θ ⊤ x − ϵ | θ 1 | and u = θ ⊤ x + ϵ | θ 1 | are respectively lower and upper bounds on the pre-activations.
Thus
[Equation]
we can compute the gradients for adversarial training w.r.t the signal parameter:
[Equation]
and applying the chain-rule we have:
[Equation]
where in the last equality we use a known property of the sigmoid function, σ ( x ) =
1 − σ ( − x ) .
B.2 Convex outer adversarial polytope
We now consider the dual approximation (cid:101)
J ϵ to the optimisation problem in Equation (39). Note that, for a binary classification problem, we have c = sgn( y ) and the dual objective in Theorem A. becomes: [Equation] where we set α to the dual feasible solution and for the sake of clarity we omit the dependence on the network parameters θ .
We are particularly interested in the data points for which J ϵ ( x, y ) ̸ = (cid:101)
J ϵ ( x, y ) , i.e., when the certified and adversarial training objectives differ.
Below, we provide a necessary and sufficient condition to have a mismatch between the two objectives.
Proposition B.2.
Consider the neural network f θ defined in Equation (36) and the threat model B ϵ defined in Equation (7) .
Let L be the binary cross-entropy loss function, as defined in Equation (37) .
Further, we define ℓ = θ ⊤ x − ϵ | θ 1 | and u = θ ⊤ x + ϵ | θ 1 | respectively as lower and upper bounds on the ReLU inputs. Let I ⋆ = { ( x, y ) : 0 ∈ [ ℓ, u ] ∧ a sgn( y ) > 0 } . Then, for data points in I ⋆ , we have that AT and
COAP gradients differ:
[Equation]
and COAP gradient is given by:
[Equation]
Further, for data points that are not in I ⋆ we have that AT and COAP gradients are equivalent:
[Equation]
Proof. For the sake of clarity, we report here the definition of COAP objective from appendix A. [Equation] Further, recall that the dual variables ν are given by the following equations:
[Equation]
where I − , I + and I denote the sets of activations in the hidden layer where ℓ and u are both negative, both positive and span zero, respectively. First, we consider the case when the neuron is always dead, i.e., ℓ < u < 0 . The dual variables are:
[Equation]
Hence, AT and COAP objectives are equal in this case:
[Equation]
where the last equality follows from Equation (41).
Next, we consider the case when the neuron is always active, i.e., 0 < ℓ < u .
The dual variables are:
[Equation]
and the dual objective becomes:
[Equation]
where the last equality follows from the fact that 0 < ℓ < u .
Finally, we consider the case when the neuron is in the activation set I , i.e., ℓ < 0 < u .
The dual variables are:
[Equation]
Here we have two cases, when ˆ ν 2 > 0 we can rewrite the dual objective as:
[Equation]
and the two objectives coincide.
When ν 2 < 0 we can rewrite the dual objective as:
[Equation]
Hence, the only case when COAP gradient differs from AT gradient is when ν 2 < 0 and the neuron belongs to the activation set I .
We compute the partial derivative w.r.t.
the signal parameter θ 1 in this case, by the chain rule we have:
[Equation]
In this section, we prove that convex relaxations along the signal direction hurt robust generalisation.
First, in Lemma B.1 we relate the robust error of the classifier f θ to the ℓ 2 -norm of the signal parameter θ 1 .
Specifically, we show that robust error monotonically decreases in ∥ θ 1 ∥ 2 .
Lemma B.1.
Let f θ be the neural network defined in Equation (36) and B ϵ the threat model defined in Equation (7) .
We define the robust risk R ϵ of f θ as follows:
[Equation]
Then, R ϵ ( θ ) is monotonically decreasing in ∥ θ 1 ∥ 2 .
Proof.
[Equation]
hence R ϵ ( θ ) is monotonically decreasing in ∥ θ 1 ∥ 2 and the statement follows. Below we present our main result. Theorem 4.1. Let ¯ θ
and ˜ θ be the network parameters after one step of gradient descent with respect to AT and COAP objectives. Let, [Equation] where θ are the network parameters at initialization.
Then, COAP yields higher robust risk than AT:
[Equation]
Proof.
First we assume, without loss of generality, that at initialisation θ 1 > 0 , and since a and b are not trainable parameters we must have a > 0 and b < 0 to include the ground truth in our hypothesis class.
Let J ϵ be the adversarial training inner maximisation as defined in Equation (39).
Then, AT solves the following optimisation problem:
[Equation]
Similarly, let (cid:101)
J ϵ be the COAP dual approximation to the inner maximization described in Equation (47).
Then, COAP solves the following optimisation problem:
[Equation]
Since we are only training the signal parameter θ 1 , after one gradient descent step, we have:
[Equation]
Further, from Lemma B.1 we know that AT yields smaller robust risk than COAP if the following holds:
[Equation]
which, after one step of gradient descent, is equivalent to:
[Equation]
Now recall from Theorems B.1 and B.2 that the gradients of AT and COAP differ only on the set I ⋆ .
In particular, we have that:
[Equation]
and
[Equation]
Hence
[Equation]
for our purpose we need to show that:
[Equation]
Our strategy will be to lower-bound the expectation in Equation (69) with some strictly positive quantity.
We define
[Equation]
and plug-in the gradient computed in Theorem B.2:
[Equation]
Now
[Equation]
we observe that Z is always negative on the set I ⋆ , since we need to satisfy the constraint ℓ < 0 < u :
[Equation]
Further, from Equation (59) we have:
[Equation]
Combining these two observations we can lower-bound the expectation:
[Equation]
Now
[Equation]
we need to show that this lower-bound is strictly positive:
[Equation]
Note that
[Equation]
we can further expand this expression:
[Equation]
Further
[Equation]
Z | ( x, y ) ∈
I ⋆ is distributed as a truncated normal with:
[Equation]
Hence, we can plug in the expectations of the truncated normal distribution to obtain the following:
[Equation]
where we define r = ∥ θ 2: d ∥ 2 ∥ θ 1 ∥ 2 .
Now, under our assumptions, from Lemma C.3 we have:
[Equation]
which concludes the proof.
C Auxiliary lemmas C.1 Upper bound on the exponential function
Lemma C.1.
Let f : R → R be the function defined by f ( x ) = exp( x ) .
When x ≤ 0 and n is even we have:
[Equation] [Equation]
Proof.
Let g : ( −∞ , 0] → R be the function defined by
[Equation]
[Equation] Since g ( x ) → ∞ as x → −∞ , g must attain an absolute minimum somewhere on the interval ( −∞ , 0] .
Now, differentiating we have:
• If f has an absolute minimum at 0 , then for all x, f ( x ) ≥ f (0) = 1 − exp(0) = 0 , so we are done.
• If f has an absolute minimum at y for some y < 0 , then f ′ ( y ) = 0 .
But differentiating,
[Equation]
Therefore, for any x ,
[Equation]
since n is even
C.2 Lower bound on the difference of Gaussian CDFs
Lemma C.2.
Let f : R 2 → R be the function defined by f ( x, y ) =
Φ( y ) − Φ( x ) .
When x < y <we have:
[Equation]
where Φ and ϕ are respectively the CDF and PDF of the standard Gaussian distribution.
Proof.
First, we want to prove that 2 x √ π is a lower bound for the error function erf( x ) when x ≤ 0 .
That is, we want to show that f ( x ) ≥ 0 where f : ( −∞ , 0]
→ R is the function defined by:
[Equation]
Since f is continuous and f ( x ) → ∞ as x → −∞ , f must attain an absolute minimimum on the interval ( −∞ , 0] .
Now, differentiating we have:
[Equation]
hence f attains an absolute minimum at 0 and we have f ( x ) ≥ f (0) = 0 .
Next, we show that 2 √ π ( x − x 3 / 3) is an upper bound for erf( x ) when x ≤ 0 .
Let g : ( −∞ , 0] → R the function defined by:
[Equation]
Similarly, since g is continuous and g ( x ) → ∞ as x → −∞ , g must attain an absolute minimimum on the interval ( −∞ , 0] .
Now, differentiating we have:
[Equation]
hence g attains an absolute minimum at 0 and we have g ( x ) ≥ g (0) = 0 .
Now, since a < b < 0 we can use the erf bounds derived above:
[Equation]
which concludes the proof.
C.3 Upper bound on the ratio of Gaussian PDFs and CDFs
Lemma C.3.
Suppose f : R → R is defined as follows:
[Equation]
where α := − γ + ϵ rσ , β := − γ − ϵ rσ , Φ and ϕ are respectively the standard Gaussian CDF and PDF.
Assume that:
[Equation]
Then, we have:
[Equation]
Proof.
We begin by providing a lower bound on the difference of gaussian cdfs.
Applying Lemma C. with x = α and y =
β we have:
[Equation]
Next, we can upper-bound f :
[Equation]
Now, we use the upper-bound for the exponential function from Lemma C.1 with n = 2 :
[Equation]
and substituting it back into our upper-bound for f we get:
[Equation]
which can be further simplified:
[Equation]
and we have that for ϵ > 5+2 √ 3 13 γ and r > (cid:114) max (cid:16)
[Equation]
10 γϵ
[Equation] [Equation] [Equation]
12 σ 2 ϵ (cid:17) the upper bound is negative, i.e. u ( r ) < 0 .
Finally, for the sake of clarity we can further simplify the condition on r :
[Equation]
which concludes the proof.
D Experimental details D.1 Synthetic experiments with signal-directed adversaries
Below we provide detailed experimental details to reproduce Figure 5.
Data generation For the linearly separable distribution we set d = 1000 , n test = 10 5 , γ = 6 .
Model and hyper-parameters For all the experiments, we use the one hidden layer architecture defined in Equation (8) with 100 neurons.
We use PyTorch SGD optimiser and train all networks for 100 epochs.
We sweep over the learning rate η ∈ { 0 . 1 , 0 . 01 , 0 . 001 } and for each perturbation budget, we choose the one that interpolates the training set and minimises robust error on the test set.
Robust evaluation We perform all the attacks to evaluate robust risk at test-time using exact line search; this is computationally tractable since the attacks are directed along one dimension.
Training paradigms For standard training (ST), wee train the network to minimise the crossentropy loss. For adversarial training (AT) [20, 13], we train the network to minimise the robust binary cross-entropy loss.
At each epoch, we compute an exact adversarial example using line search and update the weights using a gradient with respect to this example.
For convex outer adversarial polytope (COAP) [31, 32], at each epoch, we compute upper and lower bounds u and ℓ as described in Theorem A.1. We then train the network to minimise the upper bound on robust error from Theorem A.2.
Standard training.
We train the network to minimise the cross-entropy loss.
Adversarial training [20, 13].
We train the network to minimise the robust binary cross-entropy loss.
At each epoch, we compute an exact adversarial example using line search and update the weights using a gradient with respect to this example.
Certified training [31, 32].
At each epoch, we compute upper and lower bounds u and ℓ as described in Proposition A.1.
We then train the network to minimize the upper-bound on robust error derived in
Theorem A.2.
D.2 Synthetic experiments with ℓ 2 adversaries
Below we provide complete experimental details to reproduce Figures 3 and 4.
Data generation For the spheres dataset, we generate a random x ∈ R d where ∥ x ∥ 2 is either R 1 or R − 1 , with equal probability assigned to each norm.
We associate with each x a label y such that y = −
if ∥ x ∥ 2 = R − 1 and y = [-
-]1 if ∥ x ∥ 2 = R 1 .
We can sample uniformly from this distribution by sampling z ∼ N (0 , I d ) and then setting x = z ∥ z ∥ 2 R − 1 or x = z ∥ z ∥ 2 R 1 .
For the linearly separable distribution we set d = 1000 , n = 50 , n test = 10 5 , γ = 6 .
For the concentric spheres distribution we set d = 100 , , n = 50 , n test = 10 5 , γ min = 1 and γ max = 12 .
Model and hyper-parameters For all the experiments, we use a MLP architecture with W =neurons in each hidden layer and ReLU ( · ) activation functions.
We use PyTorch SGD optimiser with a momentum of 0 .
95 and train the network for 150 epochs.
We sweep over the learning rate η ∈ { 0 . 1 , 0 . 01 , 0 . 001 } and for each perturbation budget, we choose the one that minimises robust error on the test set and interpolates the training set. Robust evaluation We consider ℓ 2 -ball perturbations.
We evaluate robust error at test-time using Auto-PGD [4] with 100 iterations and 5 random restarts.
We use both the cross-entropy and difference of logits loss to prevent gradient masking.
We use the implementation provided in AutoAttack [4] with minor adjustments to allow for non-image inputs.
Training paradigms For standard training (ST), we train the network to minimise the cross-entropy loss.
For adversarial training (AT) [20, 13] we train the network to minimise the robust cross-entropy loss.
At each epoch, we search for adversarial examples using Auto-PGD [4] with a budget of 10 steps and 1 random restart.
Then, we update the weights using a gradient with respect to the adversarial examples.
For convex outer adversarial polytope (COAP) [31, 32].
We train the network to minimise the upper-bound on the robust error.
Our implementation is based on the code released by the authors.
D.3 Image experiments
Below we provide complete experimental details to reproduce Figure 2.
Model architectures For MNIST, we train the CNN architecture with four convolutional layer and two fully connected layers of 512 units introduced in Wong et al.
We report the architectural details in Table 1.
For CIFAR-10, we train the residual network (ResNet) with the same structure used in Wong et al.
[32]; we use 1 residual block with 16, 16, 32, and 64 filters.
For Tiny ImageNet, we train a WideResNet. Following Xu et al.
[34] we use 3 wide basic blocks with a widen factor of 10. [Table]
Dataset preprocessing For MNIST, we use full 28 × 28 images without any augmentations and normalisation.
For CIFAR-10, we use random horizontal flips and random crops as data augmentation, and normalise images according to per-channel statistics.
For Tiny ImageNet, we use random crops of 56 × 56 and random flips during training. During testing, we use a central 56 times 56 crop. We also normalise images according to per-channel statistics. Robust evaluation We consider ℓ 2 -ball perturbations.
We evaluate the robust error using the most expensive version of AutoAttack (AA+) [4].
Specifically, we include the following attacks: untargeted APGD-CE (5 restarts), untargeted APGD-DLR (5 restarts), untargeted APGD-DLR ( restarts), Square Attack (5000 queries), targeted APGD-DLR (9 target classes) and targeted FAB ( target classes).
AT training details For MNIST, we train 100 epochs using Adam optimiser [17] with a learning rate of 0 .
001 , momentum of 0 .
9 and a batch size of 128; we reduce the learning rate by a factor 0. at epochs 40 and 80.
For CIFAR-10 with ResNet, we train 150 epochs using SGD with a learning rate of 0 .
05 and a batch size of 128; we reduce the learning rate by a factor 0.1 at epochs 80 and 120.
For Tiny Imagenet and CIFAR-10 with Wide-Resnet we train 200 epochs using SGD with a learning rate of 0.1 and a batch size of 512; we reduce the learning rate by a factor 0.1 at epochs 100 and 150.
For the inner optimisation of all models and datasets, adversarial examples are generated withiterations of Auto-PGD [4].
COAP training details We follow the settings proposed by the authors and report them here.
For
MNIST, we use the Adam optimiser [17] with a learning rate of 0.001 and a batch size of 50.
We schedule ϵ starting from 0.01 to the desired value over the first 20 epochs, after which we decay the learning rate by a factor of 0.5 every 10 epochs for a total of 60 epochs.
For CIFAR-10, we use the SGD optimiser with a learning rate of 0.05 and a batch size of 50.
We schedule ϵ starting from 0. to the desired value over the first 20 epochs, after which we decay the learning rate by a factor of 0. every 10 epochs for a total of 60 epochs.
For all datasets and models, we use random projection ofdimensions.
For all experiments, we use the implementation provided in Wong et al.
CROWN-IBP training details We follow the settings proposed by the authors and report them here.
For MNIST, we train 200 epochs with a batch size of 256.
We use Adam optimiser [17] and set the learning rate to 5 × 10 − 4 .
We warm up with 10 epochs of regular training, and gradually ramp up ϵ train from 0 to ϵ in 50 epochs.
We reduce the learning rate by a factor 0.1 at epoch 130 and 190. For
CIFAR-10, we train 2000 epochs with a batch size of 256, and a learning rate of 5 × 10 − 4 .
We warm up for 100 epochs, and ramp-up ϵ for 800 epochs.
Learning rate is reduced by a factor 0 .
1 at epoch 1400 and 1700.
For Tiny ImageNet, we train 600 epochs with batch size 128.
The first 100 epochs are clean training, then we gradually increase ϵ train with a schedule length of 400.
For all datasets, an hyper-parameter β to balance LiRPA bounds and IBP bounds for the output layer is gradually decreased from 1 to 0 (1 for only using LiRPA bounds and 0 for only using IBP bounds), with the same schedule of ϵ .
For all experiments, we use the implementation provided in the auto LiRPA library [34].