ANCER: A NISOTROPIC C ERTIFICATION VIA S AMPLEWISE V OLUME M AXIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Randomized smoothing has recently emerged as an effective tool that enables certiﬁcation of deep neural network classiﬁers at scale.
All prior art on randomized smoothing has focused on isotropic (cid:96) p certiﬁcation, which has the advantage of yielding certiﬁcates that can be easily compared among isotropic methods via (cid:96) p -norm radius.
However, isotropic certiﬁcation limits the region that can be certiﬁed around an input to worst-case adversaries, i.e. it cannot reason about other “close”, potentially large, constant prediction safe regions.
To alleviate this issue, ( i ) we theoretically extend the isotropic randomized smoothing (cid:96) 1 and (cid:96)certiﬁcates to their generalized anisotropic counterparts following a simpliﬁed analysis.
Moreover, ( ii ) we propose evaluation metrics allowing for the comparison of general certiﬁcates – a certiﬁcate is superior to another if it certiﬁes a superset region – with the quantiﬁcation of each certiﬁcate through the volume of the certiﬁed region.
We introduce A N C ER , a framework for obtaining anisotropic certiﬁcates for a given test set sample via volume maximization.
We achieve it by generalizing memory-based certiﬁcation of data-dependent classiﬁers.
Our empirical results demonstrate that A N C ER achieves state-of-the-art (cid:96) 1 and (cid:96)certiﬁed accuracy on CIFAR-10 and ImageNet, while certifying larger regions in terms of volume, highlighting the beneﬁts of moving away from isotropic analysis.
I NTRODUCTION
The well-studied fact that Deep Neural Networks (DNNs) are vulnerable to additive imperceptible noise perturbations has led to a growing interest in developing robust classiﬁers (Goodfellow et al., 2015; Szegedy et al., 2014).
A recent promising approach to achieve state-of-the-art provable robustness ( i.e. a theoretical bound on the output around every input) at the scale of ImageNet (Deng et al., 2009) is randomized smoothing (Lecuyer et al., 2019;
Cohen et al., 2019).
Given an input x and a network f , randomized smoothing constructs g ( x ) = E (cid:15) ∼D [ f ( x + (cid:15) )] such that g ( x ) =
g ( x + δ ) ∀ δ ∈ R , where the certiﬁcation region R is characterized by x , f , and the smoothing distribution D .
For instance, Cohen et al.
(2019) showed that if D = N (0 , σ 2 I ) , then R is an (cid:96) 2 -ball whose radius is determined by x , f and σ .
Since then, there has been signiﬁcant progress towards the design of D leading to the largest R for all inputs x .
The interplay between R characterized by (cid:96) 1 , (cid:96)and (cid:96) ∞ -balls, and a notion of optimal distribution D has been previously studied Yang et al.
Despite this progress, current randomized smoothing approaches provide certiﬁcation regions that are isotropic in nature, limiting their capacity to certifying smaller and worst-case regions.
We provide an intuitive example of this behavior in Figure 1.
The isotropic nature of R in prior art is due to the common assumption that the smoothing distribution D is identically distributed (Yang et al., 2020;
Kumar et al., 2020; Levine & Feizi, 2021).
Moreover, comparisons between various randomized smoothing approaches were limited to methods that produce the same (cid:96) p certiﬁcate, with no clear metrics for comparing with other certiﬁcates.
In this paper, we address both concerns and present new state-of-the-art certiﬁed accuracy results on both CIFAR-10 and ImageNet datasets.
Our contributions are threefold.
(i)
We provide a general and simpler analysis compared to prior art (Cohen et al., 2019; Yang et al., 2020) that paves the way for the certiﬁcation of anisotropic regions characterized by any norm, holding prior art as special cases.
We then specialize our result to regions that, for a positive deﬁnite A , are ellipsoids, i.e. (cid:107) A δ (cid:107) 2 ≤ c , and generalized cross-polytopes, i.e. (cid:107) A δ (cid:107) 1 ≤ c , generalizing both (cid:96) 2 (Cohen et al., 2019) and (cid:96)
(Lecuyer et al., 2019;
Yang
[Figure]
A discussion is presented in et al., 2020) certiﬁcation (Section 4).
(ii)
We introduce a new evaluation framework to compare methods that certify general (isotropic or anisotropic) regions.
We compare two general certiﬁcates by deﬁning that a method certifying R 1 is superior to another certifying R 2 , if R 1 is a strict superset to R 2 .
Further, we deﬁne a standalone quantitative metric as the volume of the certiﬁed region, and specialize it for the cases of ellipsoids and generalized cross-polytopes (Section 5).
(iii)
We propose A N C ER , an anisotropic certiﬁcation method that performs sample-wise ( i.e. per sample in the test set) region volume maximization (Section 6), generalizing the data-depedent, memory-based solution from Alfarra et al.
Through experiments on CIFAR-10 (Krizhevsky, 2009) and
ImageNet (Deng et al., 2009), we show that restricting A N C ER ’s certiﬁcation region to (cid:96) 1 and (cid:96) 2 -balls outperforms state-of-the-art (cid:96) 1 and (cid:96) 2 results from previous works (Yang et al., 2020; Alfarra et al., 2020).
Further, we show that the volume of the certiﬁed regions are signiﬁcantly larger than all existing methods, thus setting a new state-of-the-art in certiﬁed accuracy.
Notation.
We consider a base classiﬁer f : R n → P ( K ) , where P ( K ) is a probability simplex over K classes, i.e. f i ≥ 0 and 1 (cid:62) f
= 1 , for i ∈ { 1 , . . .
, K } .
Further, we use ( x, y ) to be a sample input x and its corresponding true label y drawn from a test set D t , and f y to be the output of f at the correct class.
We use (cid:96) p to be the typically deﬁned (cid:107) · (cid:107) p norm ( p ≥ 1 ), and (cid:96) A p or (cid:107) · (cid:107) A ,p for p = { 1 , 2 } to be a composite norm deﬁned with respect to a positive deﬁnite matrix A as (cid:107)
A − 1 /p v (cid:107) p .
R ELATED W ORK
Veriﬁed Defenses.
Since the discovery that DNNs are vulnerable against input perturbations (Goodfellow et al., 2015; Szegedy et al., 2014), a range of methods have been proposed to build classiﬁers that are veriﬁably robust (Huang et al., 2017; Gowal et al., 2019; Bunel et al., 2018;
Salman et al., 2019b).
Despite this progress, these methods do not yet scale to the networks the community is interested in certifying (Tjeng et al., 2019; Weng et al., 2018).
Randomized Smoothing.
The ﬁrst works on randomized smoothing used Laplacian (Lecuyer et al., 2019; Li et al., 2019) and Gaussian Cohen et al.
(2019) distributions to obtain (cid:96) 1 and (cid:96) 2 -ball certiﬁcates, respectively.
Several subsequent works improved the performance of smooth classiﬁers by training the base classiﬁer using adversarial augmentation (Salman et al., 2019a), regularization (Zhai et al., 2019), or general adjustments to training routines (Jeong & Shin, 2020).
Recent work derived (cid:96) p -norm certiﬁcates for other isotropic smoothing distributions (Yang et al., 2020; Levine & Feizi, 2020; Zhang et al., 2019).
Concurrently, Dvijotham et al.
(2020) developed a framework to handle arbitrary smoothing measures in any (cid:96) p -norm; however, the certiﬁcation process requires signiﬁcant hyperparameter tuning.
Similarly, Mohapatra et al.
(2020) introduces larger certiﬁcates that require higher-order information, yet do not provide a closed-form solution.
This was followed by a complementary data-dependent smoothing approach, where the parameters of the smoothing distribution were optimized per test set sample to maximize the certiﬁed radius at an individual input (Alfarra et al., 2020).
All prior works considered smoothing with isotropic distributions and hence certiﬁed isotropic (cid:96) p -ball regions.
In this paper, we extend randomized smoothing to certify anisotropic regions, by pairing it with a generalization of the data-dependent framework (Alfarra et al., 2020) to maximize the certiﬁed region at each input point.
M OTIVATING A NISOTROPIC C ERTIFICATES
Certiﬁcation approaches aim to ﬁnd the safe region R , where arg max i f i ( x ) = arg max i f i ( x + δ ) ∀ δ ∈ R .
Recent randomized smoothing techniques perform this certiﬁcation by explicitly optimizing the isotropic (cid:96) p certiﬁed region around each input (Alfarra et al., 2020), obtaining state-ofthe-art performance as a result.
Despite this (cid:96) p optimality, we note that any (cid:96) p -norm certiﬁcate is worst-case from the perspective of that norm, as it avoids adversary regions by limiting its certiﬁcate to the (cid:96) p -closest adversary.
This means that it can only enjoy a radius that is at most equal to the distance to the closest decision boundary.
However, decision boundaries of general classiﬁers are complex, non-linear, and non-radially distributed with respect to a generic input sample (Karimi et al., 2019).
This is evidenced by the fact that, within a reasonably small (cid:96) p -ball around an input, there are often only a small set of adversary directions (Tramèr et al., 2017; 2018)
( e.g. see the decision boundaries in Figure 1).
As such, while (cid:96) p -norm certiﬁcates are useful to reason about worst-case performance and are simple to obtain given previous works (Cohen et al., 2019;
Yang et al., 2020;
Lee et al., 2019), they are otherwise uninformative in terms of the shape of decision boundaries, i.e. which regions around the input are safe.
To visualize these concepts, we illustrate the decision boundaries of a base classiﬁer f trained on a toy 2-dimensional, radially separable (with respect to the origin) binary classiﬁcation dataset, and consider two different input test samples (see Figure 1).
We compare the optimal isotropic and anisotropic certiﬁed regions of different shapes at these points.
In Figures 1a and 1b, we compare an isotropic cross-polytope (of the form (cid:107) δ (cid:107) 1 ≤ r ) with an anisotropic generalized cross-polytope (of the form (cid:107)
A δ (cid:107) 1 ≤ r ),
while in Figures 1c and 1d we compare an isotropic (cid:96) 2 ball (of the form (cid:107) δ (cid:107) 2 ≤ r ) with an anisotropic ellipsoid (of the form (cid:107)
A δ (cid:107) 2 ≤ r ).
Notice that in Figures 1a and 1c, due to the curvature of the classiﬁcation boundary (shown in white), the optimal certiﬁcation region is isotropic in nature, which is evidenced by the similarities of the optimal isotropic and anisotropic certiﬁcates.
On the other hand, in Figures 1b and 1d, the location of the decision boundary allows for the anisotropic certiﬁed regions to be considerably larger than their isotropic counterparts, as they are not as constrained by the closest decision boundary, i.e. the worst-case performance.
We note that these differences are further highlighted in higher dimensions, and we study them for a single
CIFAR-10 test set sample in Appendix A.1.
Further, we also showcase how anisotropic certiﬁcation allows for further insights into constant prediction (safe) directions in Appendix A.2.
A NISOTROPIC C ERTIFICATION
One of the main obstacles in enabling anisotropic certiﬁcation is the complexity of the analysis required.
To alleviate this, we follow a Lipschitz argument ﬁrst observed by Salman et al. (2019a) and Jordan & Dimakis (2020) and propose a simple and general certiﬁcation analysis.
We start with the following two observations.
All proofs are in Appendix B.
Proposition 1.
Consider a differentiable function g : R n → R .
If sup x (cid:107)∇ g ( x ) (cid:107) ∗ ≤ L where (cid:107) · (cid:107) ∗ has a dual norm (cid:107) z (cid:107) =
max x z (cid:62) x s.t.
(cid:107) x (cid:107) ∗ ≤ 1 , then g is L -Lipschitz under norm (cid:107) · (cid:107) ∗ , that is | g ( x ) − g
( y )
| ≤ L (cid:107) x − y
(cid:107) .
Given the previous proposition, we formalize (cid:107) · (cid:107) certiﬁcation as follows:
Theorem 1.
Let g : R n → R K , g
i be L -Lipschitz continuous under norm (cid:107) · (cid:107) ∗ ∀ i ∈ { 1 , . . .
, K } , and c A = arg max i g i ( x ) .
Then, we have arg max
i g i ( x + δ ) =
c A for all δ satisfying:
[Equation]
Theorem 1 provides an (cid:107) · (cid:107) norm robustness certiﬁcate for any L -Lipschitz classiﬁer g under (cid:107) · (cid:107) ∗ .
The certiﬁcate is only informative when one can attain a tight non-trivial estimate of L , ideally sup x (cid:107)∇ g ( x ) (cid:107) ∗ , which is generally difﬁcult when g is an arbitrary neural network.
Framework Recipe.
In light of Theorem 1, randomized smoothing can be viewed differently as an instance of Theorem 1 with the favorable property that the constructed smooth classiﬁer g enjoys an analytical form for L = sup x (cid:107)∇ g ( x ) (cid:107) ∗ by design.
As such, to obtain an informative (cid:107) · (cid:107) certiﬁcate, one must, for an arbitrary choice of a smoothing distribution, compute the analytic Lipschitz constant L under (cid:107) · (cid:107) ∗ for the smooth g .
While there can exist a notion of “optimal” smoothing distribution for a given choice of (cid:107) · (cid:107) certiﬁcate, as in part addressed earlier for the isotropic (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ certiﬁcates (Yang et al., 2020), this is not the focus of this paper.
The choice of the smoothing distribution in later sections is inspired by previous work for the purpose of granting anisotropic certiﬁcates.
This recipe complements randomized smoothing works based on Neyman-Pearson’s lemma (Cohen et al., 2019) or the Level-Set and Differential Method (Yang et al., 2020).
We will deploy this framework recipe to show two specializations for anisotropic certiﬁcation, namely ellipsoids (Section 4.1) and generalized cross-polytopes (Section 4.2).
4. C ERTIFYING E LLIPSOIDS
In this section, we consider the certiﬁcation under (cid:96) Σ2 norm, or (cid:107) δ (cid:107) Σ , 2 = √ δ (cid:62) Σ − 1 δ , that has a dual norm (cid:107) δ (cid:107) Σ − 1 , 2 .
Note that both (cid:107) δ (cid:107) Σ , 2 ≤ r and (cid:107) δ (cid:107) Σ − 1 , 2 ≤ r deﬁne an ellipsoid.
Despite that the following results hold for any positive deﬁnite Σ , we assume for efﬁciency reasons that Σ is diagonal throughout.
First, we consider the anisotropic Gaussian smoothing distribution N (0 , Σ) with the smooth classiﬁer deﬁned as g Σ ( x ) =
E (cid:15) ∼N (0 , Σ)
[ f ( x + (cid:15) )] .
Considering the classiﬁer Φ − 1 ( g Σ ( x )) , where Φ is the standard Gaussian CDF, and following Theorem 1 to grant an (cid:96) Σ2 certiﬁcate for Φ − 1 ( g Σ ( x )) , we derive the Lipschitz constant L under (cid:107) · (cid:107) Σ − 1 , 2 , in the following proposition.
Proposition 2.
Φ − 1 ( g Σ ( x )) is 1 -Lipschitz (i.e. L = 1 ) under the (cid:107) · (cid:107)
Σ − 1 , 2 norm.
Since Φ − 1 is a strictly increasing function, by combining Proposition 2 with Theorem 1, we have:
Corollary 1.
Let c A = arg max i g
i Σ
( x ) , then arg max i g i Σ ( x + δ )
=
c A for all δ satisfying:
[Equation]
Corollary 1 holds the (cid:96) 2 certiﬁcation from Zhai et al.
as a special case for when Σ = σ 2 I .
4. C ERTIFYING G ENERALIZED C ROSS -P OLYTOPES
Here we consider certiﬁcation under the (cid:96) Λ1 norm deﬁning a generalized cross-polytope, i.e. the set { δ : (cid:107) δ (cid:107) Λ , 1 = (cid:107) Λ − 1 δ (cid:107) 1 ≤ r } , as opposed to the (cid:96) 1 -bounded set that deﬁnes a cross-polytope, i.e. { δ : (cid:107) δ (cid:107) 1 ≤ r } .
As with the ellipsoid case and despite that the following results hold for any positive deﬁnite Λ , for the sake of efﬁciency, we assume Λ to be diagonal throughout.
For generalized cross-polytope certiﬁcation, we consider an anisotropic Uniform smoothing distribution U , which deﬁnes the smooth classiﬁer g Λ ( x ) =
E (cid:15) ∼U [ − 1 , 1] n [ f ( x + Λ (cid:15) )] .
Following Theorem 1 and to certify under the (cid:96) Λ1 norm, we compute the Lipschitz constant of g Λ under the (cid:107) Λ x (cid:107) ∞ norm, which is the dual norm of (cid:107) · (cid:107) Λ , 1 (see Appendix B), in the next proposition.
Proposition 3.
The classiﬁer g Λ is 1 / 2 -Lipschitz (i.e. L = 1 / 2 ) under the (cid:107) Λ x (cid:107) ∞ norm.
Similar to Corollary 1, by combining Proposition 3 with Theorem 1, we have that:
Corollary 2.
Let c A = arg max i g i Λ ( x ) , then arg max i g i Λ ( x + δ ) =
c A for all δ satisfying:
[Equation]
Corollary 2 holds the (cid:96) 1 certiﬁcation from Yang et al.
as a special case for when Λ = λI .
E VALUATING A NISOTROPIC C ERTIFICATES
With the anisotropic certiﬁcation framework presented in the previous section, the question arises: “Given two general (isotropic or anisotropic) certiﬁcation regions R 1 and R 2 , how can one effectively compare them?”.
We propose the following deﬁnition to address this issue.
Deﬁnition 1.
For a given input point x , consider the two certiﬁcation regions R 1 and R 2 obtained for two classiﬁers f 1 and f 2 , i.e. A 1 = { δ : arg
max c f c 1 ( x ) =
arg max c f c 1 ( x + δ ) , ∀ δ ∈ R 1 } and A 2 = { δ : arg max c f c 2 ( x ) =
arg max c f c 2 ( x + δ ) , ∀ δ ∈ R 2 } where arg max c f c 1 ( x ) = arg max c f c 2 ( x ) .
We say A 1 is a "superior certiﬁcate" to A 2 (i.e. A 1 (cid:31)
A 2 ), if and only if, A 1 ⊃ A 2 .
This deﬁnition is a natural extension from the radius-based comparison of (cid:96) p -ball certiﬁcates, providing a basis for evaluating anisotropic certiﬁcation.
To compare an anisotropic to an isotropic region of certiﬁcation, it is not immediately clear how to (i) check that an anisotropic region is a superset to the isotropic region, and (ii) if it were a superset, how to quantify the improvement of the anisotropic region over the isotropic counterpart.
In Sections 5.1 and 5.2, we tackle these issues for the particular cases of ellipsoid and generalized cross-polytope certiﬁcates.
5. E VALUATING E LLIPSOID C ERTIFICATES
Comparing (cid:96) 2 − Balls to (cid:96) Σ2 − Ellipsoids (Specialization of Deﬁnition 1).
Recall that if Σ = σ 2 I , our ellipsoid certiﬁcation in Corollary 1 recovers as a special case the isotropic (cid:96) 2 -ball certiﬁcation of Cohen et al.
Salman et al. (2019a); Zhai et al.
Consider the certiﬁed regions
R 1 = { δ : (cid:107) δ (cid:107) 2 ≤ ˜ σr 1 } and R 2 = { δ : (cid:107) δ (cid:107) Σ , 2 = √ δ (cid:62) Σ − 1 δ ≤ r 2 } for given r 1 , r 2 > 0 .
Since we take Σ = diag ( { σ 2 i } ni =1 ) , the maximum enclosed (cid:96) 2 -ball for the ellipsoid R 2 is given by the set R 3 = { δ : (cid:107) δ (cid:107) 2 ≤ min i σ i r 2 } , and thus R 2 ⊇ R 3 .
Therefore, it sufﬁces that R 3 ⊇ R 1 ( i.e. min i σ i r 2 ≥ ˜ σr 1 ), to say that R 2 is a superior certiﬁcate to the isotropic R 1 as per Deﬁnition 1.
Quantifying (cid:96) Σ2 Certiﬁcates.
The aforementioned specialization is only concerned with whether our ellipsoid certiﬁed region R 2 is “superior” to the isotropic (cid:96) 2 -ball without quantifying it.
A natural solution is to directly compare the volumes of the certiﬁed regions.
Since the volume of an ellipsoid given by R 2 is V ( R 2 ) =
r n 2 √ π n / Γ( n / 2 +1) (cid:81) ni
=
σ i (Kendall, 2004), we directly compare the proxy radius ˜ R deﬁned for R 2 as ˜ R =
r 2 n (cid:112)(cid:81) ni σ i , since larger ˜ R correspond to certiﬁed regions with larger volumes.
Note that ˜ R , which is the n th root of the volume up to a constant factor, can be seen as a generalization to the certiﬁed radius in the case when σ i = σ ∀ i .
5. E VALUATING G ENERALIZED C ROSS -P OLYTOPE C ERTIFICATES
Comparing (cid:96) 1 − Balls to (cid:96) Λ1 − Generalized Cross-Polytopes
(Specialization of Deﬁnition 1).
Consider the certiﬁcates S 1 = { δ : (cid:107) δ (cid:107) 1 ≤ ˜ λr 1 } , S 2 = { δ : (cid:107) δ (cid:107) Λ , 1 = (cid:107) Λ − 1 δ (cid:107) 1 ≤ r 2 } , and
S 3 = { δ : (cid:107) δ (cid:107) 1 ≤ min
i
λ i r 2 } , where we take Λ = diag ( { λ i } ni =1 ) .
Note that since S 2 ⊇ S 3 , then as per Deﬁnition 1, it sufﬁces that S 3 ⊇ S 1 ( i.e. min
i
λ i r 2 ≥ ˜ λr 1 ) to say that the anisotropic generalized cross-polytope S 2 is superior to the isotropic (cid:96) 1 -ball S 1 .
Quantifying (cid:96) Λ1 Certiﬁcates.
Following the approach proposed in the (cid:96) Σ2 case, we quantitatively compare the generalized cross-polytope certiﬁcation of Corollary 2 to the (cid:96) 1 certiﬁcate through the volumes of the two regions.
We ﬁrst present the volume of the generalized cross-polytope.
Proposition 4.
V (cid:0) { δ :
[Equation]
Λ
[Equation]
≤
[Equation]
=
[Equation]
i .
Following this deﬁnition, we deﬁne the proxy radius for S 2 in this case to be ˜ R =
r 2 n (cid:112)(cid:81) ni
=
λ i .
As with the (cid:96) 2 case, larger ˜ R correspond certiﬁed regions with larger volumes.
As in the ellipsoid case, ˜
R can be seen as a generalization to the certiﬁed radius when λ i = λ ∀ i .
A N C ER : S AMPLE - WISE V OLUME M AXIMIZATION FOR A NISOTROPIC C ERTIFICATION
Given the results from the previous sections, we are now equipped to certify anisotropic regions, in particular ellipsoids and generalized cross-polytopes.
As mentioned in Section 4, these regions are generally deﬁned as R =
{ δ : (cid:107) δ (cid:107) Θ ,p ≤ r p } for a given parameter of the smoothing distribution Θ = diag ( { θ i } ni =1 ) , an (cid:96) p -norm ( p ∈ { 1 , 2 } ), and a gap value of r p ∈ R + .
At this point, one could simply take an anisotropic distribution with arbitrarily chosen parameters Θ and certify a trained network at any input point x , in the style of what was done in the previous randomized smoothing literature with isotropic distributions.
However, the choice of Θ is more complex in the anisotropic case.
A ﬁxed choice of anisotropic Θ could severely underperform the isotropic case – take, for example, the anisotropic distribution of Figure 1d applied to the input of Figure 1c.
Instead of taking a ﬁxed Θ , we generalize the framework introduced in Alfarra et al.
(2020), where parameters of the smoothing distribution are optimized per input test point ( i.e. in a sample-wise fashion) so as to maximize the resulting certiﬁcate.
The goal of the optimization in Alfarra et al.
(2020) is, at a point x , to maximize the isotropic (cid:96) 2 region described in Section 4.1 ( i.e. { δ : (cid:107) δ (cid:107) 2 ≤ σ x r p ( x, σ x )) } ), where r p is the gap and a function of x and σ x ∈ R + .
In the isotropic (cid:96) p case, this generalizes to maximizing the region { δ : (cid:107) δ (cid:107) p ≤ θ x
r p ( x, θ x ) } , which can be achieved by maximizing radius θ x r p ( x, θ x ) through θ x ∈ R + , obtaining r ∗ iso (Alfarra et al., 2020).
For the general anisotropic case, we propose A N C ER , whose objective is to maximize the volume of the certiﬁed region through the proxy radius , while satisfying the superset condition with respect to the maximum isotropic (cid:96) 2 radius, r ∗ iso .
In the case of the ellipsoids and generalized cross-polytopes as presented in Sections 5.1 and 5.2, respectively, A N C ER ’s optimization problem can be written as:
[Equation]
where r p ( x, Θ x ) is the gap value under the anisotropic smoothing distribution.
We iteratively solve a relaxed version of Equation (1) , with further details presented in Appendix C.
Memory-based Anisotropic Certiﬁcation.
While each of the classiﬁers induced by the parameter Θ x , i.e. g Θ x , is robust by deﬁnition as presented in Section 4, the certiﬁcation of the overall datadependent classiﬁer is not necessarily sound due to the optimization procedure for each x .
This is a known issue in certifying data-dependent classiﬁers, and is addressed by Alfarra et al.
(2020) through the use of a memory-based procedure.
In Appendix D, we present an adapted version of this algorithm to A N C ER , and in Appendix E we discuss the limitations of this approach.
All subsequent results are obtained following this procedure.
E XPERIMENTS
We now study the empirical performance of A N C ER to obtain (cid:96) Σ2 , (cid:96) Λ1 , (cid:96) 2 and (cid:96) 1 certiﬁcates on networks trained using randomized smoothing methods found in the literature.
In this section, we show that A N C ER is able to achieve (i) improved performance on those networks in terms of (cid:96)and (cid:96) 1 certiﬁcation when compared to certiﬁcation baselines that smooth using a ﬁxed isotropic σ (Fixed σ ) (Cohen et al., 2019;
Yang et al., 2020;
Salman et al., 2019a; Zhai et al., 2019) or a data-dependent and memory-based isotropic one (Isotropic DD) (Alfarra et al., 2020); and (ii) a signiﬁcant improvement in terms of the (cid:96) Σ2 and (cid:96) Λ1 -norm certiﬁed region obtained by the same methods – compared by computing the proxy radius of the certiﬁed regions – thus generally satisfying the conditions of a superior certiﬁcate proposed in Deﬁnition 1.
We follow an evaluation procedure as similar as possible to the ones described in Cohen et al.
; Yang et al.
Salman et al.
(2019a)
; Zhai et al.
by using code and pre-trained networks whenever available and by performing experiments on CIFAR-10 (Krizhevsky, 2009) and
ImageNet (Deng et al., 2009), certifying the entire CIFAR-10 test set and a subset of 500 examples from the ImageNet test set.
For the implementation of A N C ER , we solve Equation (1) with Adam for 100 iterations, where the certiﬁcation gap r p ( x, Θ x ) is estimated at each iteration using 100 noise samples per test point (see Appendix C) and Θ x in Equation (1) is initialized with the Isotropic DD solution from Alfarra et al.
Further details of the setup can be found in Appendix F.
As in previous works, (cid:96) p certiﬁed accuracy at radius R is deﬁned as the portion of the test set
D t for which the smooth classiﬁer correctly classiﬁes with an (cid:96) p certiﬁcation radius of at least R .
In a similar fashion, we deﬁne the anisotropic (cid:96) Σ2 / (cid:96) Λ1 certiﬁed accuracy at a proxy radius of ˜ R (as deﬁned in Section 5) to be the portion of D t in which the smooth classiﬁer classiﬁes correctly with an (cid:96) Σ2 / (cid:96) Λ1 -norm certiﬁcate of an n th root volume of at least ˜ R .
We also report average certiﬁed radius ( ACR ) deﬁned as E x,
y ∼D t
[ R x 1 ( g ( x ) =
y )]
(Alfarra et al., 2020; Zhai et al., 2019) as well as average certiﬁed proxy radius ( AC ˜ R ) deﬁned as E x,y ∼D t
[ ˜ R x 1 ( g ( x ) = y )] , where R x and ˜ R x
[Table]
[Figure]
[Figure]
C e r t
i f i ed A [Figure]
[Figure]
C e r t
i f i ed A cc u r a cy
[Figure]
denote the radius and proxy radius at x with a true label y for a smooth classiﬁer g .
Recall that in the isotropic case, the proxy radius is, by deﬁnition, the same as the radius for a given (cid:96) p -norm.
For each classiﬁer, we ran experiments on the σ values reported in the original work (with the exception of Yang et al.
(2020), see Section 7.2).
For the sake of brevity, we report in this section the topcertiﬁed accuracy plots, ACR and AC ˜
R per radius across σ , as in Salman et al. (2019a); Zhai et al.
; Alfarra et al.
The performance of each method per σ is presented in Appendix H.
7. E LLIPSOID CERTIFICATION ( (cid:96) 2 AND (cid:96) Σ2 - NORM CERTIFICATES )
We perform the comparison of (cid:96) 2 -ball vs. (cid:96) Σ2 -ellipsoid certiﬁcates via Gaussian smoothing using networks trained following the procedures deﬁned in Cohen et al.
(2019), Salman et al.
(2019a), and Zhai et al.
For each of these, we report results on ResNet18 trained using σ ∈ { 0 .
5 , 1 . 0 } for CIFAR-10, and ResNet50 using σ ∈ { 0 . 25 , 0 . 5 , 1 . 0 } for ImageNet.
For details of the training procedures, see Appendix F.1.
Figure 2 plots top-1 certiﬁed accuracy as a function of the (cid:96) 2 radius (top) and of the (cid:96) Σ2 -norm proxy radius (bottom) per trained network and dataset, while Tablepresents an overview of the certiﬁed accuracy at various (cid:96) 2 radii, as well as (cid:96) 2 ACR and (cid:96) Σ2 -norm
AC ˜ R .
Recall that, following the considerations in Section 5.1, the (cid:96) 2 certiﬁcate obtained through
A N C ER is the maximum enclosed isotropic (cid:96) 2 -ball in the (cid:96) Σ2 ellipsoid.
First, we note that sample-wise certiﬁcation (Isotropic DD and A N C ER ) achieves higher certiﬁed accuracy than ﬁxed σ across the board.
This mirrors the ﬁndings in Alfarra et al.
(2020), since certifying with a ﬁxed σ for all samples struggles with the robustness/accuracy trade-off ﬁrst mentioned in Cohen et al.
(2019), whereas the data-dependent solutions explicitly optimize σ per sample to avoid it.
More importantly, A N C ER achieves new state-of-the-art (cid:96) 2 certiﬁed accuracy at most radii in Table 1, e.g. at radius 0.5 A N C ER brings certiﬁed accuracy to 77% (from 66%) and 70% (from 62%) on CIFAR-10 and ImageNet, respectively, yielding relative percentage improvements in ACR between 13% and 47% when compared to Isotropic DD.
While the results are signiﬁcant, it might not be immediately clear why maximizing the volume of an ellipsoid with A N C ER results in a larger maximum enclosed (cid:96) 2 -ball certiﬁcate in (cid:96) Σ2 ellipsoid when compared to optimizing the (cid:96) 2 -ball with
Isotropic DD.
We explore this phenomenon in Section 7.3.
As expected, A N C ER substantially improves (cid:96) Σ2 AC ˜ R compared to Isotropic DD in all cases – with relative improvements in AC ˜ R between 38% and 63% over both datasets.
The joint results, certiﬁcation with (cid:96) 2 and (cid:96) Σ2 , establish that A N C ER certiﬁes the (cid:96) 2 -ball region obtained by previous approaches, in addition to a much larger region captured by the (cid:96) Σ2 certiﬁed accuracy and AC ˜ R , and therefore is, according to Deﬁnition 1, generally superior to the Isotropic DD one.
7. G ENERALIZED C ROSS -P OLYTOPE CERTIFICATION ( (cid:96) 1 AND (cid:96) Λ1 - NORM CERTIFICATES )
To investigate (cid:96) 1 -ball vs. (cid:96) Λ1 -generalized cross-polytope certiﬁcation via Uniform smoothing, we compare A N C ER to the (cid:96) 1 state-of-the-art results from RS4A (Yang et al., 2020).
While the authors of the original work report best certiﬁed accuracy based on 15 networks trained at different σ
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] levels between 0 . 15 and 3 . 5 on CIFAR-10 (WideResNet40) and ImageNet (ResNet50) and due to limited computational resources, we perform the analysis on a subset of those networks with σ = { 0 . 25 , 0 .
We reproduce the results in Yang et al.
(2020) as closely as possible, with details of the training procedure presented in Appendix F.2.
Figure 3 shows the top-1 certiﬁed accuracy as a function of the (cid:96) 1 radius (top) and of the (cid:96) Λ1 -norm proxy radius (bottom) for RS4A, and Table 2 shows an overview of the certiﬁed accuracy at various (cid:96) 1 radii, as well as (cid:96) ACR and (cid:96) Λ1 AC ˜ R .
As with the ellipsoid case, we notice that A N C ER outperforms both Fixed σ and Istropic DD for most (cid:96) 1 radii, establishing new state-of-the-art results in CIFAR-10 at radii 0.5 and 1.0, and ImageNet at radii 0.5 (compared to previous results reported in Yang et al.
Once more and as expected, A N C ER signiﬁcantly improves the (cid:96) Λ1 AC ˜ R for all radii, pointing to substantially larger cerﬁcates than the isotropic case.
These results also establish that A N C ER certiﬁes the (cid:96) 1 -ball region obtained by previous work, in addition to the larger region obtained by the (cid:96) Λ1 certiﬁcate, and thus we can consider it superior (with respect to Deﬁnition 1) to Isotropic DD.
7. W HY DOES A N C ER IMPROVE UPON I SOTROPIC DD’ S (cid:96) p CERTIFICATES ?
As observed in Sections 7.1 and 7.2, A N C ER ’s (cid:96) 2 and (cid:96) 1 certiﬁcates outperform the corresponding certiﬁcates obtained by Isotropic DD.
To explain this, we compare the (cid:96) 2 certiﬁed region obtained by A N C ER , deﬁned in Section 6 as { δ : (cid:107) δ (cid:107) 2 ≤ min i σ xi r ( x, Σ x ) } , to the one by Isotropic DD deﬁned as { δ : (cid:107) δ (cid:107) 2 ≤ σ x r ( x, σ x ) } .
We observe that the radius of both of these certiﬁcates can be separated into a σ -factor ( σ x vs. σ x min
= min
i σ xi ) and a gap -factor ( r ( x, σ x ) vs. r ( x, Σ x ) ).
We posit the seemingly surprising result can be attributed to the computation of the gap-factor r using an anisotropic, optimized distribution.
However, another potential explanation would be that A N C ER [Table] beneﬁts from a prematurely stopped initialization provided by Isotropic DD, thus achieving a better σ x min than the isotropic σ x when given further optimization iterations
[Table]
[Figure]
and gap r right) obtained by A N C ER initialized
Isotropic Isotropic DD allowed To investigate this, we take the optimized parameters from the Isotropic DD experiments on S MOOTH A DV for an initial σ = 0 .
25 on CIFAR-10, and run the optimization step of Isotropic DD for 100 iterations more than its default number of iterations from Alfarra et al.
(2020), so as to match the total number of optimization steps between
Isotropic DD and A N C ER .
The histograms of σ x or σ x min and the gap-factor r , i.e. the two factors from the (cid:96) 2 certiﬁcation results, are presented in Figure 4.
While σ x for
Isotropic DD is similar in distribution to A N C ER ’s σ x min , the distribution of the two gaps, r ( x, σ x ) and r ( x, Σ x ) , are quite different.
In particular, the A N C ER certiﬁcation gap is signiﬁcantly larger when compared to Isotropic DD, and is the main contributor to the improvement in the (cid:96) 2 -ball certiﬁcate of A N C ER .
That is to say, A N C ER generates Σ x that is better aligned with the decision boundaries, and hence increases the conﬁdence of the smooth classiﬁer.
7. C ERTIFICATION R UNTIME
[Table]
The certiﬁcation procedures of Isotropic DD and A N C ER tradeoff improved certiﬁed accuracy for runtime, since they require a sample-wise optimization to be run prior to the C ERTIFY step described in Cohen et al.
(2019), and a memory-based step as per Alfarra et al.
The runtime of the optimization and certiﬁcation procedures is roughly equal for (cid:96) 1 , (cid:96) 2 , (cid:96) Σ2 and (cid:96) Λ1 certiﬁcation, and mostly depends on network architecture.
As such, we report the average certiﬁcation runtime for a test set sample on an NVIDIA Quadro RTX 6000 GPU for Fixed σ , Isotropic DD and A N C ER (including the isotropic initialization step) in Table 3.
We observe that the overall run time overhead for A N C ER is not signiﬁcant as compared to its certiﬁcation gains.
C ONCLUSION
In this work, we lay the theoretical foundations for anisotropic certiﬁcation through a simple analysis, propose a metric for comparing general robustness certiﬁcates, and introduce A N C ER , a certiﬁcation procedure that estimates the parameters of the anisotropic smoothing distribution to maximize the certiﬁcate.
Our experiments show that A N C ER achieves state-of-the-art (cid:96) 1 and (cid:96) 2 certiﬁed accuracy on CIFAR-10 and ImageNet.
Our anisotropic analysis enables further insights about the boundary of the safe region around an input sample, as compared to its isotropic counterpart.
R EPRODUCIBILITY S TATEMENT
We provide complete proofs for each of the theoretical results presented in Section 4 in Appendix B.
Details on the practical implementation of the A N C ER optimization algorithm is presented in
Appendix C, while the memory-based procedure is detailed in Appendix D. An overview of the experimental setup used to obtain the results in Section 7 can be found at the top of that section, while details on the hyperparameters and network training are presented in Appendix F. We include source code in Python and instructions to reproduce our results as part of the supplementary material.
E THICS S TATEMENT
We conﬁrm that no results in this paper involved studies on human subjects, and all experiments used open-source datasets.
A Q UALITATIVE M OTIVATION OF A NISOTROPIC C ERTIFICATION A. V
ISUALIZING CIFAR-10 O PTIMIZED I SOTROPIC VS . A NISOTROPIC C ERTIFICATES
[Figure]
[Figure] To extend the illustration in Figure 1 to a higher dimensional input, we now analyze an example of the isotropic (cid:96) 2 certiﬁcation of randomized smoothing with N (0 , σ 2 I ) , where σ is optimized per input Alfarra et al.
(2020), against
A N C ER , certifying an anisotropic region characterized by a diagonal (cid:96) Σ2 -norm.
To do so, we consider a CIFAR-10 Krizhevsky (2009) dataset point x , where the input is of size (32x32x3).
We perform the 2D analysis by considering the regions closest to a decision boundary.
To do so, and following Moosavi-Dezfooli et al.
(2019), we compute the Hessian of f y ( x ) with respect to x where y is the true label for x with f classifying x correctly, i.e. y = arg max i f i ( x ) .
In addition to the Hessian, we also compute its eigenvector decomposition, yielding the eigenvectors { ν i } , i ∈ { 1 , . . .
, 3072 } ordered in descending order of the absolute value of the respective eigenvalues.
In Figure 5a, we show the projection of the landscape of f y in the highest curvature directions, i.e. ν 1 and ν 2 .
Note that the isotropic certiﬁcation, much as in Figure 1c, in these 2 dimensions is nearly optimal when compared to the anisotropic region.
However, if we take the same projection with respect to the eigenvectors with the lowest and highest eigenvalues, i.e. ν 1 and ν 3072 , the advantages of the anisotropic certiﬁcation become clear as shown in Figure 5b. A. V ISUALIZING S AFE I MAGES IN O PTIMIZED A NISOTROPIC C ERTIFICATES x x +
As observed from the examples in Section 3 and Appendix A.1, anisotropic certiﬁcation reasons more closely about the shape of the decision boundaries, allowing for further insights into constant prediction (safe) directions.
In Figure 6, we present a series of test set images x , as well as practically indistinguishable x + δ images which are not inside the optimal certiﬁed isotropic (cid:96) 2 -balls for each input sample, yet are within the anisotropic certiﬁed regions.
This showcases the merits of using anisotropic certiﬁcation for characterizing larger safe regions.
B A NISOTROPIC C ERTIFICATION AND E VALUATION P ROOFS
Proposition 1 (restatement) .
Consider a differentiable function g : R n → R .
If sup x (cid:107)∇ g ( x ) (cid:107) ∗ ≤ L where (cid:107) · (cid:107) ∗ has a dual norm (cid:107) z (cid:107) =
max x z (cid:62) x s.t.
(cid:107) x (cid:107) ∗ ≤ 1 , then g is L -Lipschitz under norm (cid:107) · (cid:107) ∗ , that is | g ( x ) − g
( y )
| ≤ L (cid:107) x − y
(cid:107) .
Proof.
Consider some x, y ∈ R n and a parameterization in t as γ ( t ) =
(1 − t ) x
+ ty ∀ t ∈ [0 , 1] .
Note that γ (0) =
x and γ (1) =
y .
By the Fundamental Theorem of Calculus we have:
[Equation]
Theorem 1 (restatement) .
Let g : R n → R K , g
i be L -Lipschitz continuous under norm (cid:107) · (cid:107) ∗ ∀ i ∈ { 1 , . . .
, K } , and c A = arg max i g i ( x ) .
Then, we have arg max
i g i ( x + δ ) =
c A for all δ satisfying:
[Equation]
Proof.
Take c
B
=
arg
max
c g c (cid:54) =
c A ( x ) .
By Proposition 1, we get:
[Equation]
By subtracting the inequalities and re-arranging terms, we have that as long as g c A ( x ) − L (cid:107) δ (cid:107) > g c B ( x ) + L (cid:107) δ (cid:107) , i.e. the bound in the Theorem, then
g c A ( x + δ ) > g c B ( x + δ ) , completing the proof.
Proposition 2 (restatement) .
Consider g Σ ( x ) =
E (cid:15) ∼N (0 , Σ)
[ f ( x + (cid:15) )] .
Φ − 1 ( g Σ ( x )) is 1 -Lipschitz (i.e. L = 1 ) under the (cid:107) · (cid:107)
Σ − 1 , 2 norm.
Proof.
To prove Proposition 2, one needs to show that Φ − 1 ( g i Σ ( x )) ∀ i is 1-Lipschitz under the (cid:107) · (cid:107)
Σ − 1 , 2 norm.
For ease of notation, we drop the superscript g i Σ and use only g .
We want to show that (cid:107)∇
[Equation]
g
[Equation]
( x )) (cid:107) 2 ≤ 1 .
Following the argument presented in Salman et al.
(2019a), it sufﬁces to show that, for any unit norm direction u and p = g Σ ( x ) , we have:
[Equation]
We start by noticing that:
[Equation]
We now need to ﬁnd the optimal f ∗ : R n → [0 , 1] that satisﬁes g Σ ( x ) =
E v ∼N (0 , Σ) [ f ( x + v )]
= p while maximizing the left hand size E v ∼N (0 , Σ)
[ f ( x + v ) u (cid:62) Σ − 12 v ] . We argue that the maximizer is the following function:
[Equation]
To prove that f ∗ is indeed the optimal maximizer, we ﬁrst show feasibility.
(i) :
It is clear that f ∗ : R n → [0 , 1] . ( ii ) Note that:
[Equation]
To show the optimality of f ∗ , we show that it attains the right upper bound:
[Equation]
obtaining the bound from Equation (2), and thus completing the proof.
Proposition 3 (restatement) .
Consider g Λ ( x ) =
E (cid:15) ∼U [ − 1 , 1] n [ f ( x + Λ (cid:15) )] .
The classiﬁer g i Λ ∀ i is 1 / 2 -Lipschitz
(i.e. L = 1 / 2 ) under the (cid:107) Λ x (cid:107) ∞ norm.
Proof.
We begin by observing that the dual norm of (cid:107) x (cid:107) Λ , 1 = (cid:107) Λ − 1 x (cid:107) 1 is (cid:107) x (cid:107) ∗ =
(cid:107) Λ x (cid:107) ∞ , since:
[Equation]
Without loss of generality, we analyze ∂g
i / ∂x 1 .
Let ˆ x =
[ x 2 , . .
, x n ] ∈ R n − 1 , then:
[Equation]
Thus,
[Equation]
=2 λ j (cid:90) [ − 1 , 1] n −(cid:12)(cid:12)(cid:12)
f i ( x 1 + 1 , ˆ x + ˆΛˆ (cid:15) )
− f i ( x 1 − 1 , ˆ x + ˆΛˆ (cid:15) )
(cid:12)(cid:12)(cid:12) d n − 1 ˆ (cid:15) ≤ 1 2 .
The second and last steps follow by the change of variable t = x 1 + λ 1 (cid:15) 1 and Leibniz rule.
Following a symmetric argument, (cid:12)(cid:12) λ
j∂g i / ∂x j (cid:12)(cid:12) ≤ 1 / 2 ∀ i resulting in having (cid:107) Λ ∇ g i ( x ) (cid:107) ∞ =
max i λ i (cid:12)(cid:12) ∂g
i / ∂x i (cid:12)(cid:12) ≤ 1 / 2 ∀ i concluding the proof.
Proposition 4 (restatement) .
V (cid:0) { δ : (cid:107) Λ − 1 δ (cid:107) 1 ≤ r } (cid:1) =
(2 r ) n n ! (cid:81)
i λ i .
Proof.
Take A = r Λ − 1 = diag ( 1 / rλ 1 , . .
, 1 / rλ n ) = diag ( a 1 , . .
. , a n ) .
We can re-write the region as { x : (cid:80) i a i | x i | ≤ 1 } , from which it is clear to see that this region is an origin centered, axis-aligned simplex with the set of vertices V = {± 1 / a i e i } ni =1 , where e i is the standard basis vector i .
Deﬁne the sets of vertices V t =
V \ {− 1 / a n e n } and V b =
V \ { 1 / a n e n } .
Given the symmetry around the origin, each of these sets deﬁnes an n -dimensional hyperpyramid with a shared base
B n − 1 given by the n − 1 -dimensional hyperplane deﬁned by all vertices where x n = 0 , and an apex at the vertex 1 / a n e n (or − 1 / a n e n in the case of V b ).
The volume of each of these n − 1dimensional hyperpyramids is given by V ( B n − 1 ) / na n (Kendall (2004)), yielding a total volume of
V n = 2 n 1 a n V ( B n − 1 ) .
The same argument can be applied to compute V ( B n − 1 ) which is a union of two n − 1 -dimensional hyperpyramids.
This forms a recursion that completes the proof.
Proof.
( Alternative Proof .)
We consider the case that Λ − 1 is a general positive deﬁnite matrix that is not necessarily diagonal.
Note that V (cid:0) { δ : (cid:107) Λ − 1 δ (cid:107) 1 ≤ r } (cid:1)
=
V (cid:0) { δ : (cid:107) ( r Λ) − 1 δ (cid:107) 1 ≤ 1 } (cid:1) =
r n | Λ |V ( { δ : (cid:107) δ (cid:107) 1 ≤ 1 } ) where | r Λ | denotes the determinant.
The last equality follows by the volume of a set under a linear map and noting that { δ : (cid:107) ( r Λ) − 1 δ (cid:107) 1 ≤ 1 } = { r Λ δ : (cid:107) δ (cid:107) 1 ≤ r } .
At last, { δ : (cid:107) δ (cid:107) 1 ≤ 1 } can be expressed as the disjoint union of 2 n simplexes.
Thus, we have
V (cid:0) { δ : (cid:107) Λ − 1 δ (cid:107) 1 ≤ r } (cid:1) =
(2 r ) n / n !
| Λ | since the volume of a simplex is 1 / n ! completing the proof.
For completeness, we supplement the previous result with bounds on the volume that may be useful for future readers.
Proposition 5.
For any positive deﬁnite Λ − 1 ∈ R n × n , we have the following:
[Equation]
Z (Λ) (cid:17) ≤ V (cid:16) { δ : (cid:107) Λ − 1 δ (cid:107) 1 ≤ r } (cid:17) ≤ (2 r ) n V ( Z (Λ)) where V ( Z ( Λ ))
= (cid:112) | Λ (cid:62) Λ | which is the volume of the zonotope with a generator matrix Λ .
Proof.
Let S 1 = { δ : (cid:107) Λ − 1 δ (cid:107) 1 ≤ r } , S ∞ =
{ δ : (cid:107) Λ − 1 δ (cid:107) ∞ ≤ r } and S n ∞ =
{ δ : n (cid:107) Λ − 1 δ (cid:107) ∞ ≤ r } .
Since (cid:107) Λ − 1 δ (cid:107) ∞ ≤ (cid:107) Λ − 1 δ (cid:107) 1 ≤ n (cid:107) Λ − 1 δ (cid:107) ∞ , then S ∞ ⊇ S 1 ⊇ S n ∞ .
Therefore, we have
V ( S ∞ ) ≥ V ( S 1 ) ≥ V ( S n ∞ ) .
At last note that, S n ∞ = { rn Λ δ : (cid:107) δ (cid:107) ∞ ≤ 1 } and that with the change of variables δ = 2 u − 1 n where 1 n is a vector of all ones, we have S n ∞ =
Z (cid:0) 2 rn Λ (cid:1) ⊕ − rn Λ1 n where ⊕ is
a
Minkowski sum and noting that rn Λ1 n is a single point in R n .
Therefore, V (cid:16) Z (cid:16) 2 rn Λ (cid:17) ⊕ − rn Λ1 n (cid:17) =
( 2 r / n ) n V ( Z (Λ)) .
The upper bound follows with a similar argument completing the proof.
B. C ERTIFICATION UNDER G AUSSIAN M IXTURE S MOOTHING D ISTRIBUTION
We consider a general, K -component, zero-mean Gaussian mixture smoothing distribution G such that:
[Equation]
Given f and as per the recipe described in Section 4, we are interested in the Lipschitz constant of the smooth classiﬁer g G
[Equation]
where g Σ i is deﬁned as in the Gaussian case.
Note the weaker bound when compared to Proposition 2, for each of the Gaussian components presented in the following proposition.
Proposition 6.
g Σ is (cid:112) 2 / π -Lipschitz under (cid:107) .
(cid:107) Σ − 1 , 2 norm.
Proof.
Following a similar argument to the proof of Proposition 2, we get:
[Equation]
With Proposition 6, we obtain a Lipschitz constant for a Gaussian mixture smoothing distribution as:
Proposition 7.
g G is (cid:112) π / 2 -Lipschitz under (cid:107) δ (cid:107) B − 1 , 2 norm, where B − 1 = (cid:80)
Ki α i Σ − 1 i .
Proof.
[Equation]
Obtained by ﬁrst applying the triangle inequality, then Proposition 2 followed by Jensen’s inequality.
Thus yielding the following certiﬁcate by combining Proposition 7 and Theorem 1.
Corollary 3.
Let c A = arg max
i g G ( x ) , then arg max
i g i G ( x + δ ) =
c A for all δ satisfying:
[Equation]
where B − 1 = (cid:80)
Ki α i Σ − 1 i .
C A N C ER O PTIMIZATION
In this section we detail the implementation choices required to solving Equation (1).
For ease of presentation, we restate the A N C ER optimization problem (with Θ x = diag ( { θ xi } ni =1 ) ):
[Equation]
where r p ( x, Θ x ) is the gap value under the anisotropic smoothing distribution, and r ∗ iso is the optimal isotropic radius, i.e. ¯ θ x r p ( x, ¯ θ x ) for ¯ θ x ∈ R + .
This is a nonlinear constrained optimization problem that is challenging to solve.
As such, we relax it, and solve instead:
[Equation]
given a hyperparameter κ ∈ R + .
While the constraint θ xi ≥ ¯ θ x is not explicitly required to enforce the superset condition over the isotropic case, it proved itself beneﬁcial from an empirical perspective.
To sample from the distribution parameterized by Θ x (in our case, either a Gaussian or Uniform), we make use of the reparameterization trick , as in Alfarra et al.
The solution of this optimization problem can be found iteratively by performing projected gradient ascent.
A standalone implementation for the A N C ER optimization stage is presented in Listing 1, whereas the full code integrated in our code base is available as supplementary material.
To perform certiﬁcation, we simply feed the output of this optimization to the certiﬁcation procedure from Cohen et al.
[Table]
batch: torch.
Tensor, repeated_theta: torch.
Tensor): raise NotImplementedError def compute_gap(self, pABar: float): raise NotImplementedError class L2Certificate(Certificate): def __init__(self, batch_size: int, device: str = "cuda:0"): self.m = Normal(torch.zeros(batch_size).to(device), torch.ones(batch_size).to(device)) self.device = device self.norm = "l2" def compute_proxy_gap(self, logits: torch.
Tensor): return self.m.icdf(logits[:, 0].clamp_(0.001, 0.999)) - \ self.m.icdf(logits[:, 1].clamp_(0.001, 0.999))
def sample_noise(self, batch: torch.
Tensor, repeated_theta: torch.
Tensor): return torch.randn_like(batch, device=self.device) *
repeated_theta def compute_gap(self, pABar: float): return norm.ppf(pABar) class L1Certificate(Certificate): def __init__(self, device="cuda:0"): self.device = device self.norm = "l1" def compute_proxy_gap(self, logits: torch.
Tensor): return logits[:, 0] - logits
[:, 1] def sample_noise(self, batch: torch.
Tensor, repeated_theta: torch.
Tensor): return 2 * (torch.rand_like(batch, device=self.device) - 0.5) *
repeated_theta def compute_gap(self, pABar: float): return 2 * (pABar - 0.5)
[Table]
D M EMORY - BASED C ERTIFICATION FOR A N C ER
To guarantee the soundness of the A N C ER classiﬁer, we use an adapted version of the data-dependent memory-based solution presented in Alfarra et al.
The modiﬁed algorithm involves a post-processing certiﬁcation step that obtains adjusted certiﬁcation statistics based on the memory procedure from Alfarra et al.
(2020) (see the original paper for more details).
We present an adapted version to A N C ER of this post-processing memory-based step in Algorithm 1.
Input: input point x N +1 , certiﬁed region R N +1 , prediction C N +1 , and memory M
Result:
Prediction for x N +1 and certiﬁed region at x N +1 that does not intersect with any certiﬁed region in M .
for ( x i , C i , R i ) ∈
M do if C N +1 (cid:54) =
C i then if x N
1 ∈ R i then return A BSTAIN ,else if MaxIntersect ( R N +1 , R i ) and Intersect ( R N +1 , R i ) then end add
Note that the proposed certiﬁed region R N +1 emerges from our certiﬁcation bounds presented in Sections 4.1 and 4.2.
There are a few differences between our proposed Algorithm 1 with respect to the original variant presented in Alfarra et al.
The ﬁrst is that we remove the computation of the largest certiﬁable subset of a certiﬁed region R N +1 when there exists an i such that x N +1 ∈ R i with a different class prediction, i.e. ( LargestInSubset in Alfarra et al.
due to the complexity of the operation in the anisotropic case. As an example, it is generally difﬁcult to ﬁnd the largest volume ellipsoid contained in another ellipsoid. Due to this complexity, we choose to simply A BSTAIN instead.
Given the high dimensionality of the data, empirically, we never found a certiﬁcate in this situation within our experiments.
Further, to ease the computational burden of the
Intersect function, we introduce and instantiate the function MaxIntersect ﬁrst which checks whether the (cid:96) p -ball over-approximation of the region R N +1 intersects with a (cid:96) p over-approximation of R i .
This follows since when the (cid:96) p balls over-approximation to the anisotropic regions R N + and R i do not intersect, then R N +1 and R i do not intersect either.
Only in cases in which those over-approximation regions intersect, we run the more expensive Intersect procedure.
We present practical implementations for MaxIntersect , Intersect and LargestOutSubset for the ellipsoids and generalized cross-polytopes considered in this paper.
D. I MPLEMENTING M AX I NTERSECT ( R A , R B ) IN THE E LLIPSOID AND G ENERALIZED C ROSS -P OLYTOPE C ASES
Given the two regions R A and R B , consider (cid:96) p -ball approximations of those regions,
R ˜ B
= { x ∈ R n : (cid:107) x − a (cid:107) p ≤ r a } and R ˜ B = { x ∈ R n : (cid:107) x − b (cid:107) p ≤ r b } such that R A ⊆ R ˜
A and R B ⊆ R ˜ B .
Lemma 1.
If (cid:107) a − b (cid:107) p > r a + r b , then R A ∩ R B = ∅ .
Proof.
For the sake of contradiction, let (cid:107) a − b (cid:107) p > r a + r b and x ∈ R ˜
A ∩ R ˜ B .
Then, we have that (cid:107) x − a (cid:107) ≤ r a and (cid:107) x − b (cid:107) ≤ r b .
However:
[Equation]
forming a contradiction.
Thus, R ˜ A ∩ R ˜ B = ∅ , which in turn implies R A ∩ R B = ∅ since R A and
R B are subsets of R ˜
A and R ˜ B , respectively.
This forms a fast, maximum intersection check for ellipsoids, i.e. p = 2 , and generalized crosspolytopes, i.e. p = 1 .
The MaxIntersect function returns False if (cid:107) a − b (cid:107) p > r a + r b , and
True otherwise.
D. I MPLEMENTING I NTERSECT ( R A , R B ) IN THE E LLIPSOID C ASE
The problem of efﬁciently checking if two ellipsoids intersect is not trivial.
We rely on the work of Ros et al.
; Gilitschenski & Hanebeck (2012) with missing proofs from Gilitschenski &
Hanebeck (2012) for completeness.
Lemma 2.
Let R A
= { x ∈ R n : ( x − a )
(cid:62) A ( x − a ) ≤ 1 } and R B = { x ∈ R n : ( x − b ) (cid:62) B ( x − b ) ≤ 1 } deﬁne two ellipsoids centered at a and b , respectively.
We have that R = { x : t ( x − a ) (cid:62) A ( x − a ) + (1 − t )( x − b ) (cid:62) B ( x − b ) ≤ 1 } for any t ∈ [0 , 1]
satisﬁes R A ∩ R B ⊆ R ⊆
R A ∪ R B .
Proof.
By considering the convex combination of the left-hand side of the inequalities deﬁning the regions R A and R B , it becomes obvious that x ∈ R A ∩ R B = ⇒ x ∈ R , concluding the left side of the property.
As for the right side, it sufﬁces to show that if x / ∈ R A and x ∈ R then x ∈ R B and, similarly,
that if x / ∈ R B and x ∈ R then x ∈ R A .
We show the ﬁrst case since the second follows by symmetry.
Without loss of generality, we assume that a = b = 0 n .
Now, let x be such that x (cid:62)
A x > 1 and tx (cid:62)
A x + (1 − t ) x (cid:62) B x ≤ 1 since x / ∈ R A and x ∈ R .
Then, since x ∈ R , we have that (1 − t ) x (cid:62) B x ≤ 1 − tx (cid:62) A x ≤ 1 since x (cid:62)
A x > 1 which implies that x ∈ R B .
Note that the previous result holds without loss of generality when for the radius 1 as the radius can be absorbed in A and B .
As the following Lemma was shown by Gilitschenski & Hanebeck (2012) without proof, we complement it below for completeness.
Lemma 3.
The set R is equivalent to the following ellipsoid R = { x : ( x − m ) (cid:62) E t ( x − m ) ≤ K ( t ) } where E t =
t
A +(1 − t ) B , m = E − 1 t
( t A a + (1 − t ) B b ) , and K ( t ) = 1 − ta (cid:62)
A a − (1 − t ) b (cid:62) B b + m (cid:62) E t m .
Proof.
[Equation]
The last equality follows by adding and subtracting m (cid:62) E t m and concluding the proof.
Proposition 8.
The set of points satisfying R for t ∈ (0 , 1) is either an empty set, a single point, or the ellipsoid R .
Proof.
We ﬁrst observe that since A and B are positive deﬁnite, then E t is positive deﬁnite.
Then observe that for a choice of t ∈ (0 , 1) such that K ( t ) < 0 , the set R is an empty set, and since
R ⊇ R A ∩ R B , the two sets do not intersect.
If K ( t ) = 0 , then the only point satisfying R is the center at m .
Following a similar argument, then the two ellipsoids intersect at a point.
At last for a choice of t such that K ( t ) > 0 , then R deﬁnes an ellipsoid.
As per Theorem 8, it sufﬁces to ﬁnd some t ∈ [0 , 1] under which K ( t ) < 0 to guarantee that the ellipsoids do not intersect.
To that end, we solve the following convex optimization problem: t ∗ = argmin t ∈ [0 , 1] K ( t ) and check the condition if K ( t ∗ ) < 0 .
Moreover, as shown by Ros et al.
; Gilitschenski & Hanebeck (2012) K ( t ) is convex in the domain t ∈ (0 , 1) .
With several algebraic manipulations, one can show that K ( t ) has the following equivalent forms:
[Equation]
Observe that for ANCER, we have that both A and B to be diagonals with diagonal elements { A ii } n i =1 and { B ii } n i =1 , respectively, resulting in the following simple form for K ( t ) :
[Equation]
The Intersect function in the ellipsoid case returns False if there exists a t ∈ (0 , 1) such that K ( t ) < 0 , i.e. ellipsoids do not intersect, and True otherwise.
D. I MPLEMENTING I NTERSECT ( R A , R B ) IN THE G ENERALIZED C ROSS -P OLYTOPE C ASE
Let R A and R B be two generalized cross-polytopes R A
= { x ∈ R n : (cid:107) A ( x − a ) (cid:107) 1 ≤ 1 } and
R B = { x ∈ R n : (cid:107) B ( x − b ) (cid:107) 1 ≤ 1 } , where A and B are positive deﬁnite diagonal matrices with elements { A ii } ni =1 and { B ii } ni =1 , respectively.
We are interested in deciding whether R A and R B intersect.
However, given the conservative context in which Intersect is used in Algorithm 1, we only need to make sure that the function only returns False if it is guaranteed that R A ∩ R B = ∅ .
As such, we are able to simplify the complex problem of generalized cross-polytope intersection to the much simpler one of ellipsoid over-approximation intersection.
We do this by considering the over-approximation, i.e. superset, ellipsoids R ˜
A
= { x ∈ R n : (cid:107) A ( x − a ) (cid:107) 2 ≤ 1 } and R ˜ B = { x ∈ R n : (cid:107) B ( x − b ) (cid:107) 2 ≤ 1 } , and perform the ellipsoid intersection check presented in Appendix D.2.
If R ˜ A ∩ R ˜ B = ∅ , then this implies that R A ∩ R B = ∅
and we can safely return False .
Otherwise, we conservatively assume the generalized cross-polytopes intersect, and return True , triggering the reduction procedure detailed in Appendix D.5.
D. I MPLEMENTING L ARGEST O UT S UBSET ( R A , R B ) IN THE E LLIPSOID C ASE
Given two ellipsoids R A = { x ∈ R n : ( x − a ) (cid:62) A
( x − a ) ≤ 1 } and R B = { x ∈ R n : ( x − b ) (cid:62) B ( x − b ) ≤ 1 } that do intersect where A and B are positive deﬁnite diagonal matrices
, the task is to ﬁnd the largest possible ellipsoid R ˜ B centered at b such that R ˜ B ⊆ R B where R A ∩ R ˜ B = ∅ .
Finding a maximum ellipsoid that satisﬁes those conditions is not trivial, so instead we consider a maximum enclosing (cid:96) 2 -ball of R B ,
R ˜ B
= { x ∈ R n : (cid:107) x − b (cid:107) 2 ≤ r } , that does not intersect R A .
To obtain this ball, we project the center of R B , b , to the ellipsoid R A .
Particularly, we formulate the problem as the projection of a vector y =
b − a onto an ellipsoid with the same shape as R A centered at 0 n .
This is equivalent to solving the following optimization problem for a symmetric positive deﬁnite matrix A :
[Equation]
Note that the objective function is convex, and the constraint forms a convex set.
Forming the
Lagrangian to this problem, we obtain:
[Equation]
where λ > 0 . Therefore, the global optimal solution must satisfy the KKT conditions below:
[Equation]
Thus, to project the vector y on our region the ellipsoid characterized by A , one needs to solve the scalar optimization f ( λ ) = 0 then substitute back in the formula of x ∗ .
Further, given A = diag ( A 11 , . . .
A nn ) , we can simplify the problem to:
[Equation]
Once x ∗ is obtained, we can deﬁne the maximum radius of the (cid:96) 2 -ball centered at b that does not intersect R A as:
[Equation]
for an arbitrarily small (cid:15) .
Finally, we obtain R ˜ B as the maximum ball contained within R B that has a radius smaller than r ∗ , that is:
[Equation]
Note that while choosing the radius of R ˜ B to be r ∗ guarantees that R ˜ B ∩ R A = ∅ , this does not guarantee that R ˜ B ⊆ R B .
To guarantee both properties, we take the minimum of both r ∗ and min i B ii .
This approach ﬁnds the solution to the projection of the point to the ellipsoid { x ∈ R n : x (cid:62) A x ≤ 1 } ; it does not work for the case in which b ∈ R A , since the problem would be trivially solved by setting x ∗ =
y .
Thus, our classiﬁer must abstain in that situation.
D. I MPLEMENTING L ARGEST O UT S UBSET ( R A , R B ) IN THE G ENERALIZED C ROSS -P OLYTOPE C ASE
Let R A and R B be two generalized cross-polytopes R A
= { x ∈ R n : (cid:107) A ( x − a ) (cid:107) 1 ≤ 1 } and
R B = { x ∈ R n : (cid:107) B ( x − b ) (cid:107) 1 ≤ 1 } , where A and B are positive deﬁnite diagonal matrices with elements { A ii } ni =1 and { B ii } ni =1 , respectively.
The task is to ﬁnd the largest possible generalized cross-polytope R ˜ B centered at b such that R ˜ B ⊆ R B where R A ∩ R ˜ B = ∅ .
As with the ellipsoid case, solving this problem for a generalized cross-polytope is not trivial, so instead we consider a maximum enclosing cross-polytope (i.e., (cid:96) 1 -ball) of R ˜ B = { x ∈ R n : (cid:107) x − b (cid:107) 1 ≤ r } that does not intersect R A and is a subset of R B .
To obtain this (cid:96) 1 -ball, we project the center of R B , b , to the generalized cross-polytope R A in a similar fashion to the ellipsoid case in Appendix D.4.
We formulate the problem as the projection of the vector y =
b − a to the 0 n centered generalized cross-polytope { x ∈ R n : (cid:107) A x (cid:107) 1 ≤ 1 } .
Lemma 4.
Consider the hyperplane H
= { x ∈ R n : w (cid:62) x − k = 0 } and a point y ∈ R n .
The (cid:96)projection of y on the hyperplane is the point x ∗ =
y − ( w (cid:62) y − k ) w / (cid:107) w (cid:107) 22 .
Proof.
We deﬁne the projection problem in a similar fashion to the ellipsoid case: min x 2 (cid:107) x − y (cid:107)s.t.
w (cid:62) x − k = 0 , and obtain the Lagrangian as L ( x, λ ) = 12 (cid:107) x
{+− y (cid:107) 
+}
+ λ ( w (cid:62) x − k ) , from where we get (using the KKT conditions): x ∗ =
y − λ ∗ w and λ ∗ = w (cid:62) y − k / (cid:107) w (cid:107) 22 ; thus obtaining: x ∗ = y − ( w (cid:62) y − k ) w (cid:107) w (cid:107).
While this formulation does not yield the closest point from a hyperplane when measured with the (cid:96)norm, the fact that (cid:107) x − x ∗ (cid:107) 1 ≥ (cid:107) x − x ∗ (cid:107) 2 implies the certiﬁcation set obtained in the (cid:96) 1 norm via this method is a subset of the (cid:96) 2 -ball of the minimum projection point.
Crucially, this (cid:96) 2 projection has the advantage of having a closed-form solution, while an (cid:96) 1 one would require solving the problem using an iterative linear programming solver.
As such, for the sake of computational complexity, we decided to use this projection, despite the sub-optimality of the result from the (cid:96) 1 perspective.
Empirically, we have found this does not affect our results.
Since the set of vertices of the generalized cross-polytope { x ∈ R n : (cid:107) A x (cid:107) 1 ≤ 1 } is given by { e i / A ii , − e i / A ii } ni =1 , and considering the distance between the projections and the original y , the hyperplane that minimizes it is deﬁned by the set of vertices { sign ( y i )
e i / A ii } ni =1 .
By writing it as a system of n equations, we obtain the hyperplane deﬁned by w =
[ − sign ( y 1 )
A 11 , ..., − sign ( y n ) A nn ] and k = 1 .
Finally, after computing x ∗ as per Lemma 4, we can deﬁne the maximum radius of the (cid:96) 1 -ball centered at b that does not intersect R A as:
[Equation]
for an arbitrarily small (cid:15) .
Finally, and similar to the ellipsoids case, we obtain R ˜ B as the maximum generalized cross-polytope contained within R B that has a radius smaller than r ∗ , that is:
[Equation]
Similar to before, to guarantee that the (cid:96) 1 ball R ˜ B is still a subset to R B , we take the minimum between r ∗ and min i B ii to be the radius of R ˜ B .
As with the ellipsoid case, this approach does not work for the case in which b ∈ R A , since the assumption of the closest plane to y would not hold.
Thus, our classiﬁer must abstain in that situation.
E L IMITATIONS OF A N C ER
Given A N C ER uses a memorization procedure similar to the one presented in Alfarra et al.
, it incurs in limitations at the level of memory and runtime complexity.
The main limitations of the general memorization procedure are outlined in Appendix E of Alfarra et al.
The anisotropic case in particular increases on the complexity of the isotropic framework by the increased runtime of
Intersect and LargestOutSubset .
A practical comparison of the runtime of both cases can be found in Section 7.4.
While the linear dependency in terms of runtime might appear a daunting prospect for the deployment of such a system in a real world scenario, there are a few factors that could mitigate the cost.
The ﬁrst is that in practice the models deployed get regularly updated in deployment, and the memory should be reset in those situations.
A second factor is the fact that there are possible solutions which might lead to sublinear runtime at the level of the post-certiﬁcation stage, such as the application of k -d trees to reduce the space of comparisons and speed-up the process.
A further limitation of the memorization procedure has to do with the impact of the order in which inputs are certiﬁed on the overall statistics obtained.
As is clear from Algorithm 1, certifying xwith x 1 in memory can be different from certifying x 1 with x 2 in memory if they intersect due to the largest subset procedure.
In practice, given the low number of intersections observed with the original certiﬁed regions, this effect was negligible in the results presented in Section 7.
In the future, we plan to explore how an attacker could exploit this information to attack memory-based certiﬁcation.
F E XPERIMENTAL S ETUP
The experiments reported in the paper used the CIFAR-10 Krizhevsky (2009) 3 and ImageNet Deng et al.
(2009) 4 datasets, and trained ResNet18, WideResNet40 and ResNet50 networks He et al.
Experiments used the typical data split for these datasets found in the PyTorch implementation Paszke et al.
The procedures to obtain the baseline networks used in the experiments are detailed in
Appendix F.1 and F.2 for ellipsoids and generalized cross-polytopes, respectively.
Source code to reproduce the A N C ER optimization and certiﬁcation results of this paper is available as supplementary material.
Isotropic DD Optimization.
We used the available code of Alfarra et al.
(2020) 5 to obtain the isotropic data dependent smoothing parameters.
To train our models from scratch, we used an adapted version of the code provided in the same repository.
Certiﬁcation.
Following Cohen et al.
Salman et al. (2019a); Zhai et al.
; Yang et al.
(2020); Alfarra et al.
(2020), all results were certiﬁed with N 0 = 100 Monte Carlo samples for selection and N = 100 , 000 estimation samples, with failure a probability of α = 0 .
F. E LLIPSOID CERTIFICATION BASELINE NETWORKS
In terms of ellipsoid certiﬁcation, the baselines we considered were C OHEN Cohen et al.
S MOOTH A DV Salman et al.
(2019a)
7 and MACER Zhai et al.
In the CIFAR-10 experiments, we used a ResNet18 architecture, instead of the ResNet110 used in Cohen et al.
Salman et al. (2019a); Zhai et al.
due to constraints at the level of computation power.
As such, we had to train each of the networks from scratch following the procedures available in the source code of each of the baselines.
We did so under our own framework, and the training scripts are available in the supplementary material.
For the ImageNet experiments we used the ResNet50 networks provided by each of the baselines in their respective open source repositories.
We trained the ResNet18 networks for 120 epochs, with a batch size of 256 and stochastic gradient descent with a learning rate of 10 − 2 , and momentum of 0.9.
F. G ENERALIZED C ROSS -P OLYTOPE CERTIFICATION BASELINE NETWORKS
For the certiﬁcation of generalized cross-polytopes we considered RS4A Yang et al.
As described in RS4A Yang et al.
(2020), we take λ = σ / √ 3 and report results as a function of σ for ease of comparison.
As with the baseline, we ran experiments on CIFAR-10 on a WideResNet40 architecture, and
ImageNet on a ResNet50 Yang et al.
However, due to limited computational power, we were not able to run experiments on the wide range of distributional parameters the original work considers, i.e. σ = { 0 . 15 , 0 . 25 , 0 . 5 , 0 .
75 , 3 . 0 , 3 . 25 , 3 . 5 } on CIFAR-10 and σ = { 0 .
75 , 3 . 0 , 3 . 25 , 3 . 5 } on
ImageNet.
Instead, and matching the requirements from the ellipsoid section, we choose a subset of σ = { 0 . 25 , 0 .
and performed our analysis at that level.
While the trained models are available in the source code of RS4A, we ran into several issues when we attempted to use them, the most problematic of which being the fact that the clean accuracy of such models was very low in both the WideResNet40 and ResNet50 ones.
To avoid these issues we trained the models from scratch, but using the stability training loss as presented in the source code of RS4A. All of these models achieved clean accuracy of over 70%.
Following the procedures described in the original work, we trained the WideResNet40 models with the stability loss used in Yang et al.
(2020) for 120 epochs, with a batch size of 128 and stochastic gradient descent with a learning rate of 10 − 2 , and momentum of 0.9, along with a step learning rate scheduler with a γ of 0.1.
For the ResNet50 networks on ImageNet, we trained them from scratch with stability loss for 90 epochs with a learning rate of 0.1 that drops by a factor of 0.1 after eachepochs and a batch size of 256.
G S UPERSET ARGUMENT
The results we present in Section 7 support the argument that A N C ER achieves, in general, a certiﬁcate that is a superset of the Fixed σ and Isotropic DD ones.
To conﬁrm this at an individual test set sample level, we compare the (cid:96) 2 , (cid:96) 1 , (cid:96) Σ2 and (cid:96) Λ1 certiﬁcation results across the different methods, and obtain the percentage of the test set in which A N C ER performs at least as well as all other methods in each certiﬁcates of the samples.
Results of this analysis are presented in Tables 4 and 5.
For most networks and datasets, we observe that A N C ER achieves a larger (cid:96) p certiﬁcate than the baselines in a signiﬁcant portion of the dataset, showcasing the fact that it obtains a superset of the isotropic region per sample.
This is further conﬁrmed by the comparison with the anisotropic certiﬁcates, in which, for all trained networks except MACER in CIFAR-10, A N C ER ’s certiﬁcate is superior in over 90% of the test set samples.
[Table] [Table]
[Table] [Table]
[Table]
H E XPERIMENTAL R ESULTS PER σ H. C ERTIFYING E LLIPSOIDS - (cid:96) 2 AND (cid:96) Σ2 CERTIFICATION RESULTS PER σ
In this section we report certiﬁed accuracy at various (cid:96) 2 radii and (cid:96) Σ2 proxy radii, following the metrics deﬁned in Section 7, for each training method (C OHEN Cohen et al.
, S MOOTH A DV Salman et al.
(2019a)
and
MACER Zhai et al.
, dataset (CIFAR-10 and ImageNet) and σ ( σ ∈ { 0 .
Figures 7 and 8 shows certiﬁed accuracy at different (cid:96) 2 radii for CIFARand ImageNet, respectively, whereas Figures 9 and 10 plot certiﬁed accuracy and different (cid:96) Σ2 proxy radii for CIFAR-10 and ImageNet, respectively.
[Figure]
[Figure]
[Figure]
In this section we report certiﬁed accuracy at various (cid:96) 1 radii and (cid:96) Λ1 proxy radii, following the metrics deﬁned in Section 7, for RS4 A , dataset (CIFAR-10 and ImageNet) and σ ( σ ∈ { 0 . 25 , 0 . 5 , 1 . 0 } ).
[Figure]
[Figure]
[Figure]
[Figure]
Figures 11 and 12 shows certiﬁed accuracy at different (cid:96) 1 radii for CIFAR-10 and ImageNet, respectively
[Figure] whereas Figures 13 and 14 plot certiﬁed accuracy and different (cid:96) Λ1 proxy radii for CIFARand ImageNet, respectively [Figure] [Figure]
[Figure]
I V ISUAL C OMPARISON OF P ARAMETERS IN E LLIPSOID C ERTIFICATES
Anisotropic certiﬁcation allows for a better characterization of the decision boundaries of the base classiﬁer f .
For example, the directions aligned with the major axes of the ellipsoids (cid:107) δ (cid:107) Σ , 2 = r , i.e. locations where Σ is large, are, by deﬁnition, expected to be less sensitive to perturbations compared to the minor axes directions.
To visualize this concept, Figure 15 shows CIFAR-10 images along with their corresponding optimized (cid:96) 2 isotropic parameters obtained by Isotropic DD, and (cid:96) Σ anisotropic parameters obtained by A N C ER .
First, we note the richness of information provided by the anisotropic parameters when compared to the (cid:96) 2 worst-case, isotropic one.
Interestingly, pixel locations where the intensity of Σ is large (higher intensity in Figure 15) are generally the ones corresponding least with the underlying true class and overlapping more with background pixels.
A particular insight one can get from A N C ER certiﬁcation is that the decision boundaries are not distributed isotropically around each input.
To quantify this in higher dimensions, we plot in Figurea histogram of the ratio between the maximum and minimum elements of our optimized smoothing parameters for the experiments on SmoothAdv (with an initial σ = 1 . 0 ) on CIFAR-10.
We note that this ratio can be as high as 5 for some of the input points, meaning the decision boundaries in that case could be 5 times closer to a given input for some directions than others.
J N ON DATA - DEPENDENT A NISOTROPIC C ERTIFICATION
As mentioned brieﬂy in Section 6, it is our intuition that anisotropic certiﬁcation requires a datadependent approach, as different points will have fairly different decision boundaries and the certiﬁed regions will extend in different directions (as exempliﬁed in Figure 1). To validate this claim, we perform certiﬁcation of SmoothAdv Salman et al. (2019a) with an initial σ = 1 on CIFAR-10 using a Σ which is the average of all the optimized Σ x .
The results of the certiﬁed accuracy, ACR and AC ˜
R are presented in Table 6, along with the same results for the methods reported in the main paper.
As can be observed, moving away from the data-dependent certiﬁcation in the anisotropic scenario leads to a signiﬁcant performance drop in terms of robustness.
[Figure]
K T HEORETICAL AND E MPIRICAL C OMPARISON WITH M OHAPATRA ET AL . (2020)
In regards to the theoretical results, unfortunately the certiﬁed regions of Mohapatra et al.
(2020) do not exhibit a closed form solution similarly to ours. Thus, a direct theoretical volume bound comparison is not possible. As for the empirical comparison, A N C ER ’s performance on both (cid:96) 2 and (cid:96) 1 certiﬁcates far out-does that of Mohapatra et al. For example, with (cid:96) 2 certiﬁcates at a radius of 0 . 5 , Cohen certiﬁed with A N C ER achieves 77% certiﬁed accuracy (see Table 1) while Mohapatra et al.
(2020) achieves under 60% certiﬁed accuracy.
Note that Mohapatra et al.
(2020) has only a marginal improvement over Cohen et al. As for the (cid:96) 1 certiﬁcates, Mohapatra et al. (2020) uses the Gaussian distribution of Cohen et al, resulting in worse performance than existing state-of-art in (cid:96) 1 Yang et al. (2020) that uses a uniform distribution.
Our approach improves further upon the performance of Yang et al.
For example, as per Table 2, RS4A with ANCER certiﬁcation achieves 84% certiﬁed accuracy at an (cid:96) 1 radius of 0 .
5 , Yang et al.
(2020) achieves 75% certiﬁed accuracy while Mohapatra et al.
(2020) achieves below 60% .
However, we believe that the combination of both approaches,
ANCER and Mohapatra et al.
(2020) can further boost the performance as also hinted on in the abstract of Mohapatra et al.
(2020) on the use of data-dependent smoothing. [Figure] [Table] [Table]