Original: T IGHT S ECOND -O RDER C ERTIFICATES FOR R ANDOMIZED S MOOTHING
Revised: T IGHT S ECOND -O RDER C ERTIFICATES FOR R ANDOMIZED S MOOTHING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Randomized smoothing is a popular way of providing robustness guarantees against adversarial attacks: randomly-smoothed functions have a universal
Revised: Randomized smoothing is a popular way of providing robustness guarantees against adversarial attacks: randomly-smoothed functions have a universal

Original: Lipschitz-like bound, allowing for robustness certiﬁcates to be easily computed.
Revised: Lipschitz-like bound, allowing for robustness certiﬁcates to be easily computed.

Original: In this work, we show that there also exists a universal curvature-like bound for
Revised: In this work, we show that there also exists a universal curvature-like bound for

Original: Gaussian random smoothing: given the exact value and gradient of a smoothed function, we compute a lower bound on the distance of a point to its closest adversarial example, called the S
Revised: Gaussian random smoothing: given the exact value and gradient of a smoothed function, we compute a lower bound on the distance of a point to its closest adversarial example, called the S

Original: econd-
Revised: econd-

Original: o rder S moothing ( SoS ) robustness certiﬁcate.
Revised: o rder S moothing ( SoS ) robustness certiﬁcate.

Original: In addition to proving the correctness of this novel certiﬁcate, we show that SoS certiﬁcates are realizable and therefore tight.
Revised: In addition to proving the correctness of this novel certiﬁcate, we show that SoS certiﬁcates are realizable and therefore tight.

Original: Interestingly, we show that the maximum achievable beneﬁts, in terms of certiﬁed robustness, from using the additional information of the gradient norm are relatively small: because our bounds are tight, this is a fundamental negative result.
Revised: Interestingly, we show that the maximum achievable beneﬁts, in terms of certiﬁed robustness, from using the additional information of the gradient norm are relatively small: because our bounds are tight, this is a fundamental negative result.

Original: The gain of SoS certiﬁcates further diminishes if we consider the estimation error of the gradient norms, for which we have developed an estimator.
Revised: The gain of SoS certiﬁcates further diminishes if we consider the estimation error of the gradient norms, for which we have developed an estimator.

Original: We therefore additionally develop a variant of Gaussian smoothing, called Gaussian dipole smoothing , which provides similar bounds to randomized smoothing with gradient information, but with much-improved sample efﬁciency.
Revised: We therefore additionally develop a variant of Gaussian smoothing, called Gaussian dipole smoothing , which provides similar bounds to randomized smoothing with gradient information, but with much-improved sample efﬁciency.

Original: This allows us to achieve (marginally) improved robustness certiﬁcates on high-dimensional datasets such as CIFAR-10 and ImageNet.
Revised: This allows us to achieve (marginally) improved robustness certiﬁcates on high-dimensional datasets such as CIFAR-10 and ImageNet.

Revised: Code is available at https://github.com/alevine0/smoothing_second_ order .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A topic of much recent interest in machine learning has been the design of deep classiﬁers with provable robustness guarantees.
Revised: A topic of much recent interest in machine learning has been the design of deep classiﬁers with provable robustness guarantees.

Original: In particular, for an m -class classiﬁer h : R d → [ m ] , the Lcertiﬁcation problem for an input x is to ﬁnd a radius ρ such that, for all δ with (cid:107) δ (cid:107) 2 < ρ , h ( x )
Revised: In particular, for an m -class classiﬁer h : R d → [ m ] , the Lcertiﬁcation problem for an input x is to ﬁnd a radius ρ such that, for all δ with (cid:107) δ (cid:107) 2 < ρ , h ( x )

Original: = h ( x + δ ) .
Revised: = h ( x + δ ) .

Original: This robustness certiﬁcate serves as a lower bound on the magnitude of any adversarial perturbation of the input that can change the classiﬁcation: therefore, the certiﬁcate is a security guarantee against adversarial attacks.
Revised: This robustness certiﬁcate serves as a lower bound on the magnitude of any adversarial perturbation of the input that can change the classiﬁcation: therefore, the certiﬁcate is a security guarantee against adversarial attacks.

Original: There are many approaches to the certiﬁcation problem, including exact methods, which compute the precise norm to the decision boundary (Tjeng et al., 2019; Carlini et al., 2017; Huang et al., 2017) as well as methods for which the certiﬁcate ρ is merely a lower bound on the distance to the decision boundary (Wong & Kolter, 2018; Gowal et al., 2018; Raghunathan et al., 2018).
Revised: There are many approaches to the certiﬁcation problem, including exact methods, which compute the precise norm to the decision boundary (Tjeng et al., 2019; Carlini et al., 2017; Huang et al., 2017) as well as methods for which the certiﬁcate ρ is merely a lower bound on the distance to the decision boundary (Wong & Kolter, 2018; Gowal et al., 2018; Raghunathan et al., 2018).

Original: One approach that belongs to the latter category is Lipschitz function approximation .
Revised: One approach that belongs to the latter category is Lipschitz function approximation .

Original: Recall that a function f : R d → R is L -Lipschitz if, for all x , x (cid:48) , | f ( x ) − f ( x (cid:48) )
Revised: Recall that a function f : R d → R is L -Lipschitz if, for all x , x (cid:48) , | f ( x ) − f ( x (cid:48) )

Original: | ≤ L (cid:107) x − x (cid:48) (cid:107) 2 .
Revised: | ≤ L (cid:107) x − x (cid:48) (cid:107) 2 .

Original: If a classiﬁer is known to be a Lipschitz function, this immediately implies a robustness certiﬁcate.
Revised: If a classiﬁer is known to be a Lipschitz function, this immediately implies a robustness certiﬁcate.

Original: In particular, consider a binary classiﬁcation for simplicity, where we use an L -Lipschitz function f as a classiﬁer, using the sign of f ( x ) as the classiﬁcation.
Revised: In particular, consider a binary classiﬁcation for simplicity, where we use an L -Lipschitz function f as a classiﬁer, using the sign of f ( x ) as the classiﬁcation.

Original: Then for any input x , we are assured that the classiﬁcation (i.e, the sign) will remain constant for all x (cid:48) within a radius | f ( x ) | /L of x .
Revised: Then for any input x , we are assured that the classiﬁcation (i.e, the sign) will remain constant for all x (cid:48) within a radius | f ( x ) | /L of x .

Original: Numerous methods for training Lipschitz neural networks with small, known Lipschitz constants have been proposed.
Revised: Numerous methods for training Lipschitz neural networks with small, known Lipschitz constants have been proposed.

Original: (Fazlyab et al., 2019; Zhang et al., 2019; Anil et al., 2019; Li et al., 2019b)
Revised: (Fazlyab et al., 2019; Zhang et al., 2019; Anil et al., 2019; Li et al., 2019b)

Original: It is desirable that the network be as expressive as possible, while still maintaining the desired Lipschitz property.
Revised: It is desirable that the network be as expressive as possible, while still maintaining the desired Lipschitz property.

Original: Anil et al.
Revised: Anil et al.

Original: (2019) in particular demonstrates that their proposed method can universally approximate Lipschitz functions, given sufﬁcient network complexity.
Revised: (2019) in particular demonstrates that their proposed method can universally approximate Lipschitz functions, given sufﬁcient network complexity.

Original: However, in practice, for the robust certiﬁcation problem on large-scale input, randomized smoothing (Cohen et al., 2019) is the current state-of-the-art method.
Revised: However, in practice, for the robust certiﬁcation problem on large-scale input, randomized smoothing (Cohen et al., 2019) is the current state-of-the-art method.

Original: The key observation of randomized smoothing (as formalized by  et al., 2019; Levine et al., 2019)) is that, for any x → Φ
Revised: The key observation of randomized smoothing (as formalized by (Salman et al., 2019; Levine et al., 2019)) is that, for any arbitrary base classiﬁer function f : R d → [0 , 1] , the function

Original: [Equation]
Revised: [Equation]

Original: 1 ( p a ) where p a ( x ) :=
Revised: where p a ( x ) :=

Original: [Equation]
Revised: [Equation]

Original: [Equation] ) is (1 /σ ) -Lipschitz, where N (0 , σ 2 I ) is a d -dimensional isometric Gaussian distribution with variance σ 2 and Φ − 1 is the inverse normal CDF function.
Revised: ) is (1 /σ ) -Lipschitz, where N (0 , σ 2 I ) is a d -dimensional isometric Gaussian distribution with variance σ 2 and Φ − 1 is the inverse normal CDF function.

Original: As a result, given the smoothed classiﬁer value p a ( x ) at x , one can calculate the certiﬁed radius ρ ( x ) =
Revised: As a result, given the smoothed classiﬁer value p a ( x ) at x , one can calculate the certiﬁed radius ρ ( x ) =

Original: σ Φ − 1 ( p a ( x )) in which p a ( x ) ≥ 0 .
Revised: σ Φ − 1 ( p a ( x )) in which p a ( x ) ≥ 0 .

Original: 5 (i.e., Φ − 1 ( p a ( x )) ≥ 0 ).
Revised: 5 (i.e., Φ − 1 ( p a ( x )) ≥ 0 ).

Original: This means that we can use p a ( x ) ∈ R d → [0 , 1] as a robust binary classiﬁer (with one class assignment if p a ( x ) ≥ 0 .
Revised: This means that we can use p a ( x ) ∈ R d → [0 , 1] as a robust binary classiﬁer (with one class assignment if p a ( x ) ≥ 0 .

Original: 5 , and the other if p a ( x ) < 0 . 5 ).
Revised: 5 , and the other if p a ( x ) < 0 . 5 ).

Original: Cohen et al.
Revised: Cohen et al.

Original: (2019) shows that this is a tight certiﬁcate result for a classiﬁer smoothed with Gaussian noise: given the value of p a ( x ) , there exists a base classiﬁer function f such that, if p a is the Gaussian-smoothed version of f , then there exists an x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 

Revised: (2019) shows that this is a tight certiﬁcate result for a classiﬁer smoothed with Gaussian noise: given the value of p a ( x ) , there exists a base classiﬁer function f such that, if p a is the Gaussian-smoothed version of f , then there exists an x (cid:48) with (cid:107) x − x (cid:48) (cid:107)

Original: = ρ such that p a ( x (cid:48) )
Revised: = ρ such that p a ( x (cid:48) )

Original: = 0 .
Revised: = 0 .

Original: In other words, the certiﬁcate provided by (Cohen et al., 2019) is the largest possible certiﬁcate for Gaussian smoothing, given only the value of p a ( x ) .
Revised: In other words, the certiﬁcate provided by (Cohen et al., 2019) is the largest possible certiﬁcate for Gaussian smoothing, given only the value of p a ( x ) .

Original: Previous results (Li et al., 2019a; Lecuyer et al., 2019) provided looser bounds for Gaussian smoothing.
Revised: Previous results (Li et al., 2019a; Lecuyer et al., 2019) provided looser bounds for Gaussian smoothing.

Original: Singla & Feizi (2020) have recently shown, for shallow neural networks, that, rather than globally bounding the (ﬁrst-order)
Revised: Singla & Feizi (2020) have recently shown, for shallow neural networks, that, rather than globally bounding the (ﬁrst-order)

Original: Lipschitz constant of the network, it  possible to achieve larger robustness certiﬁcates by instead globally bounding the Lipschitz constant of the gradient of the network.
Revised: Lipschitz constant of the network, it is possible to achieve larger robustness certiﬁcates by instead globally bounding the Lipschitz constant of the gradient of the network.

Original: This second-order, curvature-based method takes advantage of the fact that the gradient at x can be computed easily via back-propagation, so certiﬁcates can make use of both f ( x ) and ∇ x f ( x ) .
Revised: This second-order, curvature-based method takes advantage of the fact that the gradient at x can be computed easily via back-propagation, so certiﬁcates can make use of both f ( x ) and ∇ x f ( x ) .

Original: This leads to a question: can we also use the gradient of a smoothed classiﬁer ∇ x p a ( x ) to improve smoothing-based certiﬁcates?
Revised: This leads to a question: can we also use the gradient of a smoothed classiﬁer ∇ x p a ( x ) to improve smoothing-based certiﬁcates?

Original: In this work, we show that there is a universal curvature-like bound for all randomly-smoothed classiﬁers.
Revised: In this work, we show that there is a universal curvature-like bound for all randomly-smoothed classiﬁers.

Original: Therefore, given p a ( x ) and ∇ x p a ( x ) , we can compute larger certiﬁcates than is possible using the value of p a ( x ) alone.
Revised: Therefore, given p a ( x ) and ∇ x p a ( x ) , we can compute larger certiﬁcates than is possible using the value of p a ( x ) alone.

Original: Moreover, our bound is tight in that, given only the pair ( p a ( x ) , ∇ x p a ( x )) , the certiﬁcate we provide is the largest possible certiﬁcate for Gaussian smoothing.
Revised: Moreover, our bound is tight in that, given only the pair ( p a ( x ) , ∇ x p a ( x )) , the certiﬁcate we provide is the largest possible certiﬁcate for Gaussian smoothing.

Original: We call our certiﬁcates “Second-order Smoothing” (SoS) certiﬁcates.
Revised: We call our certiﬁcates “Second-order Smoothing” (SoS) certiﬁcates.

Original: As shown in Figure 1, the smoothing-based certiﬁcates which we can achieve using second-order smoothing represent relatively modest improvements compared to the ﬁrst-order bounds.
Revised: As shown in Figure 1, the smoothing-based certiﬁcates which we can achieve using second-order smoothing represent relatively modest improvements compared to the ﬁrst-order bounds.

Original: This is a meaningful negative result, given the tightness of our bounds, and is therefore useful in guiding (or limiting) future research into higher-order smoothing certiﬁcates.
Revised: This is a meaningful negative result, given the tightness of our bounds, and is therefore useful in guiding (or limiting) future research into higher-order smoothing certiﬁcates.

Original: Additionally, this result shows that randomized smoothing (or, speciﬁcally, functions in the form of Equation 1) can not be used to uni- versally approximate Lipschitz functions: all randomly smoothed functions will have the additional curvature constraint described in this work.
Revised: Additionally, this result shows that randomized smoothing (or, speciﬁcally, functions in the form of Equation 1) can not be used to universally approximate Lipschitz functions: all randomly smoothed functions will have the additional curvature constraint described in this work.

Original: If the base classiﬁer f is a neural network, computing the expectation in Equation 1 analytically is not tractable.
Revised: If the base classiﬁer f is a neural network, computing the expectation in Equation 1 analytically is not tractable.

Original: Therefore it is standard (Lecuyer et al., 2019;
Revised: Therefore it is standard (Lecuyer et al., 2019;

Original: Cohen et al., 2019;
Revised: Cohen et al., 2019;

Original: Salman et al., 2019) to estimate this expectation using N random samples, and bound the expectation probabilistically.
Revised: Salman et al., 2019) to estimate this expectation using N random samples, and bound the expectation probabilistically.

Original: The certiﬁcate is then as a high-probability, rather than exact, result, using the estimated lower bound of p a ( x ) .
Revised: The certiﬁcate is then as a high-probability, rather than exact, result, using the estimated lower bound of p a ( x ) .

Original: In Section 3.1, we discuss empirical estimation of the gradient norm of a smoothed classiﬁer for second-order certiﬁcation. This also leads to somewhat negative result: we ﬁnd that (for an unrestricted base classiﬁer) the number ofGaussian samples required tomeaningfully estimate the gradientunavoidably scales  with the dimensionality d of the input.We develop an estimation scheme which tightly achieves this scaling.
Revised: In Section 3.1, we discuss empirical estimation of the gradient norm of a smoothed classiﬁer for second-order certiﬁcation, and develop an estimator for this quantity, in which the number of samples required to estimate the gradient scales linearly with the dimensionality d of the input.

Original: In order to overcome this, in Section 4, we develop a modiﬁed form of Gaussian randomized smoothing, Gausian Dipole Smoothing, which allows for a dipole certiﬁcate , related to the second-order certiﬁcate, to be computed.
Revised: In order to overcome this, in Section 4, we develop a modiﬁed form of Gaussian randomized smoothing, Gausian Dipole Smoothing, which allows for a dipole certiﬁcate , related to the secondorder certiﬁcate, to be computed.

Original: Unlike the secondorder certiﬁcate, however, the dipole certiﬁcate has no explicit dependence of dimensionality in its estimation, and therefore can practically scale to real-world high-dimensional datasets.
Revised: Unlike the second-order certiﬁcate, however, the dipole certiﬁcate has no explicit dependence of dimensionality in its estimation, and therefore can practically scale to real-world high-dimensional datasets.

Original:  P RELIMINARIES , A SSUMPTIONS AND N OTATION
Revised:  P RELIMINARIES , A SSUMPTIONS AND N OTATION

Original: We use f ( x ) to represent a generic scalar-valued “base” function to be smoothed.
Revised: We use f ( x ) to represent a generic scalar-valued “base” function to be smoothed.

Original: In general, we assume f ∈ R d → [0 , 1] .
Revised: In general, we assume f ∈ R d → [0 , 1] .

Original: However, for empirical estimation results (Theorem 3), we assume that f is a “hard” base classiﬁer: f ∈ R d → { 0 , 1 } .
Revised: However, for empirical estimation results (Theorem 3), we assume that f is a “hard” base classiﬁer: f ∈ R d → { 0 , 1 } .

Original: This will be made clear in context.
Revised: This will be made clear in context.

Original: The smoothed version of f is notated as p a ∈ R d → [0 , 1] , deﬁned as in equation 1.
Revised: The smoothed version of f is notated as p a ∈ R d → [0 , 1] , deﬁned as in equation 1.

Original: Recall that Φ is the normal CDF function and Φ (cid:48) is the normal PDF function.
Revised: Recall that Φ is the normal CDF function and Φ (cid:48) is the normal PDF function.

Original: In randomized smoothing for multi-class problems, the base classiﬁer is typically a vector-valued function f ∈ R d → { 0 , 1 } m , (cid:80) c
Revised: In randomized smoothing for multi-class problems, the base classiﬁer is typically a vector-valued function f ∈ R d → { 0 , 1 } m , (cid:80) c

Original: f c ( x ) = 1 , where m is the number of classes.
Revised: f c ( x ) = 1 , where m is the number of classes.

Original: The ﬁnal classiﬁcation returned by the smoothed classiﬁer is then given by a := arg max c E (cid:15)
Revised: The ﬁnal classiﬁcation returned by the smoothed classiﬁer is then given by a := arg max c E (cid:15)

Original: f c ( x + (cid:15) ) .
Revised: f c ( x + (cid:15) ) .

Original: However, in most prominent implementations (Cohen et al., 2019; Salman et al., 2019), certiﬁcates are computed using only the smoothed value for the estimated top class a , where a is estimated using a small number
Revised: However, in most prominent implementations (Cohen et al., 2019; Salman et al., 2019), certiﬁcates are computed using only the smoothed value for the estimated top class a , where a is estimated using a small number

Original: N 0 of initial random samples, before the ﬁnal value of p a ( x ) is computed using N samples.
Revised: N 0 of initial random samples, before the ﬁnal value of p a ( x ) is computed using N samples.

Original: The certiﬁcate then determines the radius in which p a ( x (cid:48) ) will remain above 0 .
Revised: The certiﬁcate then determines the radius in which p a ( x (cid:48) ) will remain above 0 .

Original: 5 : this guarantees that a will remain the top class, regardless of the other logits.
Revised: 5 : this guarantees that a will remain the top class, regardless of the other logits.

Original: While some works (Lecuyer et al., 2019;
Revised: While some works (Lecuyer et al., 2019;

Original: Feng et al., 2020) independently estimate each smoothed logit, this incurs additional estimation error as the number of classes increases.
Revised: Feng et al., 2020) independently estimate each smoothed logit, this incurs additional estimation error as the number of classes increases.

Original: In this work, we assume that only estimates for the top-class smoothed logit p a ( x ) and its gradient ∇ x p a ( x ) are available (although we brieﬂy discuss the case with more estimated logits in Section 3.2).
Revised: In this work, we assume that only estimates for the top-class smoothed logit p a ( x ) and its gradient ∇ x p a ( x ) are available (although we brieﬂy discuss the case with more estimated logits in Section 3.2).

Original: When discussing empirical estimation, we use η as the accepted probability of failure of an estimation method.
Revised: When discussing empirical estimation, we use η as the accepted probability of failure of an estimation method.

Original:  S ECOND -O RDER S MOOTHING C ERTIFICATE
Revised:  S ECOND -O RDER S MOOTHING C ERTIFICATE

Original: We now state our main second-order robustness certiﬁcate result:
Revised: We now state our main second-order robustness certiﬁcate result:

Original: Theorem 1.
Revised: Theorem 1.

Original: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ , and for all f : R d → [0 , 1] ,
Revised: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ , and for all f : R d → [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original:  where a (cid:48) is the (unique) solution to
Revised: (2) where a (cid:48) is the (unique) solution to

Original: [Equation]
Revised: [Equation]

Original: Further, for all pairs ( p a ( x ) , (cid:107)∇ x p a ( x ) (cid:107) 2 ) which are possible, there exists a base classiﬁer f and an adversarial point x (cid:48) such that Equation 4 is an equality.
Revised: Further, for all pairs ( p a ( x ) , (cid:107)∇ x p a ( x ) (cid:107) 2 ) which are possible, there exists a base classiﬁer f and an adversarial point x (cid:48) such that Equation 2 is an equality.

Original: This implies that our certiﬁcate is realizable, and therefore tight.
Revised: This implies that our certiﬁcate is realizable, and therefore tight.

Original: Note that the right-hand side of Equation 2 is monotonically decreasing with ρ : we can then compute a robustness certiﬁcate by simply setting p a ( x (cid:48) )
Revised: Note that the right-hand side of Equation 2 is monotonically decreasing with ρ : we can then compute a robustness certiﬁcate by simply setting p a ( x (cid:48) )

Original: = 0 .
Revised: = 0 .

Original: 5 and solving for the certiﬁed radius ρ .
Revised: 5 and solving for the certiﬁed radius ρ .

Original: Also,  a (cid:48) can be computed easily, because the left-hand side of Equation 3 is monotonic in a (cid:48) .
Revised: Also, [Figure] a (cid:48) can be computed easily, because the left-hand side of Equation 3 is monotonic in a (cid:48) .

Original: Evaluated certiﬁcate values are shown in Figure 1-b, and compared with ﬁrst-order certiﬁcates.
Revised: Evaluated certiﬁcate values are shown in Figure 1-b and compared with ﬁrst-order certiﬁcates.

Original: [Figure] x p a ( x ) (cid:107) 2 = 
Revised: both experiments, N = 10 8 , and η = 0 .

Revised: All proofs are presented in Appendix A.

Original: Like in Cohen et al.
Revised: Like in Cohen et al.

Original:  we proceed by constructing the worst-case base classiﬁer f given p a ( x ) and (cid:107)∇ x p a ( x ) (cid:107) 2 .
Revised: (2019), we proceed by constructing the worst-case base classiﬁer f given p a ( x ) and (cid:107)∇ x p a ( x ) (cid:107) 2 .

Original: This is the base classiﬁer f which creates an adversarial point to the smoothed classiﬁer as close as possible to x , given the constraints that p a ( x ) and (cid:107)∇ p a ( x ) (cid:107) 2 are equal to their reported values.
Revised: This is the base classiﬁer f which creates an adversarial point to the smoothed classiﬁer as close as possible to x , given the constraints that p a ( x ) and (cid:107)∇ p a ( x ) (cid:107) 2 are equal to their reported values.

Original: In Cohen et al.
Revised: In Cohen et al.

Original: (2019), given only p a ( x ) , this is simply a linear classiﬁer.
Revised: (2019), given only p a ( x ) , this is simply a linear classiﬁer.

Original: With the gradient norm, the worst case is that x lies in a region with class a which is a slice between two linear decision boundaries, both perpendicular to ∇ p a ( x ) .
Revised: With the gradient norm, the worst case is that x lies in a region with class a which is a slice between two linear decision boundaries, both perpendicular to ∇ p a ( x ) .

Original: See Figure 3.
Revised: See Figure 3.

Original: Note that, by isometry and because ∇ p a ( x ) is the only vector information we have, there is no beneﬁt in certiﬁed radius to having the direction of ∇ p a ( x ) : the norm is sufﬁcient.
Revised: Note that, by isometry and because ∇ p a ( x ) is the only vector information we have, there is no beneﬁt in certiﬁed radius to having the direction of ∇ p a ( x ) : the norm is sufﬁcient.

Original: In the case of a linear classiﬁer the gradient takes its maximum possible value: (cid:107)∇ [Equation] σ − 1 Φ (cid:48) (Φ − 1 ( p a ( x )) .
Revised: In the case of a linear classiﬁer the gradient takes its maximum possible value: (cid:107)∇ x p a ( x ) (cid:107) 2 = σ − 1 Φ (cid:48) (Φ − 1 ( p a ( x )) .

Original: This case is shown in
Revised: This case is shown in

Original: Figure 3-a: if the gradient norm is equal to this value, the second-order certiﬁcate is identical to the ﬁrst-order certiﬁcate (Cohen et al., 2019).
Revised: Figure 3-a: if the gradient norm is equal to this value, the second-order certiﬁcate is identical to the ﬁrst-order certiﬁcate (Cohen et al., 2019).

Original: However, if the gradient norm is smaller, then we cannot be in this worst-case linear-classiﬁer scenario.
Revised: However, if the gradient norm is smaller, then we cannot be in this worst-case linear-classiﬁer scenario.

Original: Instead, the new “worst case” is constructed by introducing a second “wrong class” region opposite to the direction of the adversarial point (Figure 3-b).
Revised: Instead, the new “worst case” is constructed by introducing a second “wrong class” region opposite to the direction of the adversarial point (Figure 3-b).

Original: In the extreme case (Figure 3-c) where the gradient norm is zero, this is accomplished by balancing two adversarial regions in a “sandwich” around x .
Revised: In the extreme case (Figure 3-c) where the gradient norm is zero, this is accomplished by balancing two adversarial regions in a “sandwich” around x .

Original: This “sandwich” conﬁguration reveals the relative weakness of gradient information in improving robustness certiﬁcates: having zero gradient does not require that the adversarial regions be evenly distributed around x .
Revised: This “sandwich” conﬁguration reveals the relative weakness of gradient information in improving robustness certiﬁcates: having zero gradient does not require that the adversarial regions be evenly distributed around x .

Original: Rather, it is sufﬁcient to distribute the adversarial probability mass 1 − p a ( x ) into just two adversarial regions.
Revised: Rather, it is sufﬁcient to distribute the adversarial probability mass 1 − p a ( x ) into just two adversarial regions.

Original: Therefore, the certiﬁed radius, even in this most extreme case, is similar to the Cohen et al.
Revised: Therefore, the certiﬁed radius, even in this most extreme case, is similar to the Cohen et al.

Original: (2019) certiﬁcate in the case with half as much adversarial probability mass (the ﬁrst-order certiﬁcate for p a ( x ) :
Revised: (2019) certiﬁcate in the case with half as much adversarial probability mass (the ﬁrst-order certiﬁcate for p a ( x ) :

Original: = (1 + p a ( x )) / 2 ).
Revised: = (1 + p a ( x )) / 2 ).

Original: This can be seen in Figure 1-b: note that at p a ( x ) = 0 .
Revised: This can be seen in Figure 1-b: note that at p a ( x ) = 0 .

Original: 6 , if the gradient norm is known to be zero, the certiﬁcate is slightly below the certiﬁcate for p a ( x ) = 0 .
Revised: 6 , if the gradient norm is known to be zero, the certiﬁcate is slightly below the certiﬁcate for p a ( x ) = 0 .

Original: 8 with no gradient information.
Revised: 8 with no gradient information.

Original: The second-order cer- tiﬁcate at p a ( x ) = 0 .
Revised: The second-order certiﬁcate when ( p a ( x ) = 0 .

Original: 6 , (cid:107) nabla x p a ( x ) |(cid:107) 2 = 0 is in fact slightly below the ﬁrst-order certiﬁcate for p a ( x ) = 0 .
Revised: 6 , (cid:107)∇ x p a ( x ) |(cid:107) 2 = 0) is in fact slightly below the ﬁrst-order certiﬁcate for p a ( x ) = 0 .

Original: 8 , because the Gaussian noise samples throughout all of space, so the smoothed classiﬁer decision boundary is slightly affected by the adversarial region in the opposite direction of x .
Revised: 8 , because the Gaussian noise samples throughout all of space, so the smoothed classiﬁer decision boundary is slightly affected by the adversarial region in the opposite direction of x .

Original: Because we can explicitly construct “worst-case” classiﬁers which represent the equality case of
Revised: Because we can explicitly construct “worst-case” classiﬁers which represent the equality case of

Original: Equation 2, our certiﬁcates are known to be tight: the reported certiﬁed radii are the largest possible certiﬁcates, if only p a ( x ) and (cid:107)∇ p a ( x ) (cid:107) 2 are known.
Revised: Equation 2, our certiﬁcates are known to be tight: the reported certiﬁed radii are the largest possible certiﬁcates, if only p a ( x ) and (cid:107)∇ p a ( x ) (cid:107) 2 are known.

Original: In Figure 2, we show how our second-order certiﬁcate behaves on a simple, two-dimensional, nonlinearly separable dataset, the classic Swiss Roll.
Revised: In Figure 2, we show how our second-order certiﬁcate behaves on a simple, two-dimensional, nonlinearly separable dataset, the classic Swiss Roll.

Original: The increases are marginal, mostly because the certiﬁcates using standard randomized smoothing are already fairly tight.
Revised: The increases are marginal, mostly because the certiﬁcates using standard randomized smoothing are already fairly tight.

Original: On these data, the certiﬁed radii for the two classes are nearly touching in many places along the decision boundary.
Revised: On these data, the certiﬁed radii for the two classes are nearly touching in many places along the decision boundary.

Original: However, for the blue class, which is surrounded on multiple sides by the red class, there are noticeable increases in the certiﬁed radius.
Revised: However, for the blue class, which is surrounded on multiple sides by the red class, there are noticeable increases in the certiﬁed radius.

Original: This is especially true for points near the center of the blue class, which are at the “top of the hill” of the blue class probability, and therefore have smaller gradient.
Revised: This is especially true for points near the center of the blue class, which are at the “top of the hill” of the blue class probability, and therefore have smaller gradient.

Original: 3. G RADIENT N ORM E STIMATION
Revised: 3. G RADIENT N ORM E STIMATION

Original: In order to use the second-order certiﬁcate in practice, we must ﬁrst bound, with high-probability, the gradient norm (cid:107)∇ x p a ( x ) (cid:107) 2 using samples from the base classiﬁer f .
Revised: In order to use the second-order certiﬁcate in practice, we must ﬁrst bound, with high-probability, the gradient norm (cid:107)∇ x p a ( x ) (cid:107) 2 using samples from the base classiﬁer f .

Original: Because Theorem 1 provides certiﬁcates that are strictly decreasing with (cid:107)∇ x p a ( x ) (cid:107) 2 , it is only necessary to lower bound (cid:107)∇ x p a ( x ) (cid:107) 2 with high probability.
Revised: Because Theorem 1 provides certiﬁcates that are strictly decreasing with (cid:107)∇ x p a ( x ) (cid:107) 2 , it is only necessary to lower bound (cid:107)∇ x p a ( x ) (cid:107) 2 with high probability.

Original: Salman et al.
Revised: Salman et al.

Original: (2019) suggests two ways of approximate the gradient vector ∇ x p a ( x ) itself, both based on the following important observation:
Revised: (2019) suggest two ways of approximate the gradient vector ∇ x p a ( x ) itself, both based on the following important observation:

Original: [Equation]
Revised: [Equation]

Original: These two methods are:
Revised: These two methods are:

Original: At each sampled point, one can measure the gradient of f using back-propagation, and take the mean vector of these estimates.
Revised: At each sampled point, one can measure the gradient of f using back-propagation, and take the mean vector of these estimates.

Original: At each sampled point, one can multiply f ( x + (cid:15) ) by the noise vector (cid:15) , and take the mean vector of these estimates.
Revised: At each sampled point, one can multiply f ( x + (cid:15) ) by the noise vector (cid:15) , and take the mean vector of these estimates.

Original: Note, however, that Salman et al.
Revised: Note, however, that Salman et al.

Original: (2019) does not provide statistical bounds on these estimates: for our certiﬁcate application, we must do so.
Revised: (2019) does not provide statistical bounds on these estimates: for our certiﬁcate application, we must do so.

Original: While we ultimately use an approach based on method 2, we will ﬁrst brieﬂy discuss method 1.
Revised: While we ultimately use an approach based on method 2, we will ﬁrst brieﬂy discuss method 1.

Original: The major obstacle to using method 1 is that it requires that the base classiﬁer f itself to be a Lipschitz function, with a small Lipschitz constant.
Revised: The major obstacle to using method 1 is that it requires that the base classiﬁer f itself to be a Lipschitz function, with a small Lipschitz constant.

Original: This can be understood from Markov’s inequality.
Revised: This can be understood from Markov’s inequality.

Original: For example, consider the value of some component z ( x ) :
Revised: For example, consider the value of some component z ( x ) :

Original: = u · ∇ f ( x ) , where u is an arbitrary vector.
Revised: = u · ∇ f ( x ) , where u is an arbitrary vector.

Original: Suppose N samples are taken, but that z is distributed such that:
Revised: Suppose N samples are taken, but that z is distributed such that:

Original: [Equation]
Revised: [Equation]

Original: This would be the case if f is a function that approximates a step function from 0 to 1, with a small buffer region of very high slope, for example.
Revised: This would be the case if f is a function that approximates a step function from 0 to 1, with a small buffer region of very high slope, for example.

Original: Note that the probability that any of the N samples measures the nonzero gradient component is < 0 . 5 , but the expected value of this component is in fact 1 . 0 .
Revised: Note that the probability that any of the N samples measures the nonzero gradient component is < 0 . 5 , but the expected value of this component is in fact 1 . 0 .

Original: This example shows that, in order to accurately estimate the gradient with high probability, the number of samples used must at least scale linearly with the maximum possible value of the gradient norm for f .
Revised: This example shows that, in order to accurately estimate the gradient with high probability, the number of samples used must at least scale linearly with the maximum possible value of the gradient norm for f .

Original: For un-restricted deep neural networks, Lipschitz constants are NP-hard to estimate, and upper bounds on it are typically very large (Virmaux & Scaman, 2018).
Revised: For un-restricted deep neural networks, Lipschitz constants are NP-hard to compute, and upper bounds on them are typically very large (Virmaux & Scaman, 2018).

Original: Of course, we could use Lipschitz-constrained networks as described in Section 1 for the base classiﬁer, but this would defeat the purpose of using randomized smoothing in the ﬁrst place.
Revised: Of course, we could use Lipschitz-constrained networks as described in Section 1 for the base classiﬁer, but this would defeat the purpose of using randomized smoothing in the ﬁrst place.

Original: Moreover, in standard “hard” randomized smoothing as typically implemented (Cohen et al., 2019; Salman et al., 2019), the range of f is { 0 , 1 } , so f is non-differentiable: therefore, this back-propagation method can not be used at all.
Revised: Moreover, in standard “hard” randomized smoothing as typically implemented (Cohen et al., 2019; Salman et al., 2019), the range of f is { 0 , 1 } , so f is non-differentiable: therefore, this back-propagation method can not be used at all.

Original: We therefore use method 2. This also has a fundamental limitation: consider the case where the gradient is zero, and the base classiﬁer f ( x ) = c everywhere, for some constant c (if considering a “hard” base classiﬁer, equivalently consider the base classiﬁer as very rapidly oscillating between zero and one along some dimension). In this case, the N observed values of (cid:15)f ( x + (cid:15) ) will each simply follow a d -dimensional normal distribution.
Revised: [Figure]

Original: In each dimension, the sample mean vector will then follow a Gaussian distribution with mean zero and variance c 2 σ 2 /N . However, the normsquared of the ﬁnal estimated gradient, as a sum of the squares of each of these Gaussians, will be distributed by a χ 2 distribution with d degrees of freedom: the mean will concentrate at c 2 σ 2 d N . Because the actual norm-squared of the mean of the samples will concentrate at a value proportional to d/N , the best we can hope for is an estimation method where the number of samples we require is proportional to d . In fact, we are able to achieve this.
Revised: We therefore use method 2.

Original: In particular, we reject the naive approach of estimating each component independently, taking a union bound, and the taking the norm: not only would the error in the norm-squared scale with d as the error from each component accumulates, but there would be an additional dependence on d from the union bound: each component would have to be bounded with failure probability η/d , where η is the total failure probability for measuring the gradient norm.
Revised: In particular, we reject the naive approach of estimating each component independently, taking a union bound, and the taking the norm: not only would the error in the normsquared scale with d as the error from each component accumulates, but there would be an additional dependence on d from the union bound: each component would have to be bounded with failure probability η/d , where η is the total failure probability for measuring the gradient norm.

Original: Note that this issue will also be encountered in method 1 above, but in that case, a loose upper bound could at least be achieved without this dependency using Jensen’s inequality (the mean of the norms of the gradient is larger than the norm of the mean).
Revised: Note that this issue will also be encountered in method 1 above, but in that case, a loose upper bound could at least be achieved without this dependency using Jensen’s inequality (the mean of the norms of the gradient is larger than the norm of the mean).

Original: Instead, we estimate the norm-squared of the mean using a single, unbiased estimator.
Revised: Instead, we estimate the norm-squared of the mean using a single, unbiased estimator.

Original: Note that:
Revised: Note that:

Original: [Equation]
Revised: [Equation]

Original: In other words, we can estimate the norm-squared of the mean by taking pairs of smoothing samples, and taking the dot product of the noise vectors times the product of the sampled values.
Revised: In other words [Equation] we can estimate the norm-squared of the mean by taking pairs of smoothing samples, and taking the dot product of the noise vectors times the product of the sampled values.

Original: We show that this is a subexponential random variable (see Appendix), which gives us the asymptotically linear scaling of N with d , as desired:
Revised: We show that this is a subexponential random variable (see Appendix), which gives us an asymptotically linear scaling of N with d :

Original: Theorem 2.
Revised: Theorem 2.

Original: Let V :=
Revised: Let V :=

Original: E (cid:15),(cid:15) (cid:48)
Revised: E (cid:15),(cid:15) (cid:48)

Original: [( (cid:15)f ( x + (cid:15) )) · ( (cid:15) (cid:48)
Revised: [( (cid:15)f ( x + (cid:15) )) · ( (cid:15) (cid:48)

Original: f ( x + (cid:15) (cid:48) ))] , and ˜ V be its empirical estimate.
Revised: f ( x + (cid:15) (cid:48) ))] , and ˜ V be its empirical estimate.

Original: If n pairs of samples ( = N/ 2 ) are used to estimate V , then, with probability at most η , E [ V ] − ˜ V ≥ t , where:
Revised: If n pairs of samples ( = N/ 2 ) are used to estimate V , then, with probability at most η , E [ V ] − ˜ V ≥ t , where:

Original: [Equation]
Revised: [Equation]

Original: Note that in practice, we can use the same samples to estimate (cid:107)∇ x p a ( x ) (cid:107) 2 as are used to estimate p a ( x ) .
Revised: Note that in practice, we can use the same samples to estimate (cid:107)∇ x p a ( x ) (cid:107) 2 as are used to estimate p a ( x ) .

Original: However, this requires reducing the failure probability of each estimate to η (cid:48)
Revised: However, this requires reducing the failure probability of each estimate to η (cid:48)

Original: =
Revised: =

Original: η/ 2 , in order to use a union bound.
Revised: η/ 2 , in order to use a union bound.

Original: This means that, if N is small (or d large), second-order smoothing can in fact give worse certiﬁcates than standard smoothing, because the beneﬁt of a (loose, for N small) estimate of the gradient is less signiﬁcant than the negative effect of reducing the estimate of p a ( x ) .
Revised: This means that, if N is small (or d large), second-order smoothing can in fact give worse certiﬁcates than standard smoothing, because the beneﬁt of a (loose, for N small) estimate of the gradient is less signiﬁcant than the negative effect of reducing the estimate of p a ( x ) .

Original: As shown in Figure 4-a, even for very large N and relatively small dimension, the empirical estimation signiﬁcantly reduces the radii of certiﬁcates which can be calculated.
Revised: As shown in Figure 4-a, even for very large N and relatively small dimension, the empirical estimation signiﬁcantly reduces the radii of certiﬁcates which can be calculated.

Original: See Section 5 for experimental results.
Revised: See Section 5 for experimental results.

Original: 3. U PPER - BOUND AND MULTI - CLASS CERTIFICATES
Revised: 3. U PPER - BOUND AND MULTI - CLASS CERTIFICATES

Original: We can easily convert Theorem 1 into a tight upper bound on p a ( x (cid:48) ) by simply evaluating it for f (cid:48)
Revised: We can easily convert Theorem 1 into a tight upper bound on p a ( x (cid:48) ) by simply evaluating it for f (cid:48)

Original: = 1 − f (and therefore p (cid:48) a = 1 − p a ).
Revised: = 1 − f (and therefore p (cid:48) a = 1 − p a ).

Original: If estimates and gradients are available for multiple classes, it would then be possible to achieve an even larger certiﬁcate, by setting the lower bound of the top logit equal to the upper bounds of each of the other logits.
Revised: If estimates and gradients are available for multiple classes, it would then be possible to achieve an even larger certiﬁcate, by setting the lower bound of the top logit equal to the upper bounds of each of the other logits.

Original: Note, however, that unlike ﬁrst-order smoothing works (Lecuyer et al., 2019; Feng et al., 2020) which use this approach, it is not sufﬁcient to compare against just the “‘runner-up” class, because other logits may have less restrictive upper-bounds due to having larger gradients.
Revised: Note, however, that unlike ﬁrst-order smoothing works (Lecuyer et al., 2019; Feng et al., 2020) which use this approach, it is not sufﬁcient to compare against just the “‘runner-up” class, because other logits may have less restrictive upper-bounds due to having larger gradients.

Original: As discussed above, gradient norm estimation can be computationally expensive, so gradient estimation for many classes may not be feasible.
Revised: As discussed above, gradient norm estimation can be computationally expensive, so gradient estimation for many classes may not be feasible.

Original: Also, note that while this approach would produce larger, correct certiﬁcates, we do not claim that these would be tight certiﬁcates given the value and gradient information for all classes: the “worst case” constructions we describe above for a single logit might not be simultaneously construct-able for multiple logits.[Figure]
Revised: Also, note that while this approach would produce larger, correct certiﬁcates, we do not claim that these would be tight certiﬁcates given the value and gradient information for all classes: the “worst case” constructions we describe above for a single logit might not be simultaneously construct-able for multiple logits.

Original:  D IPOLE S MOOTHING
Revised:  D IPOLE S MOOTHING

Original: For large-scale image datasets, the dependence on d in Theorem 2 can create statistical barriers.
Revised: For large-scale image datasets, the dependence on d in Theorem 2 can create statistical barriers.

Original: However, the general approach of second-order smoothing, especially using the discrete estimation method (method 2) described above, has an interesting interpretation: rather than using simply the mean of f ( x + (cid:15) ) , we are also using the geometrical distribution of the values of f ( x + (cid:15) ) in space to compute a larger certiﬁed bound.
Revised: However, the general approach of second-order smoothing, especially using the discrete estimation method (method 2) described above, has an interesting interpretation: rather than using simply the mean of f ( x + (cid:15) ) , we are also using the geometrical distribution of the values of f ( x + (cid:15) ) in space to compute a larger certiﬁed bound.

Original: In particular, if we can show that points which are adversarial for the base classiﬁer (points with f ( x + (cid:15) ) = 0 ) are dispersed , then this will imply larger certiﬁcates, because it makes it impossible for a perturbation in a single direction to move x towards the adversarial region.
Revised: In particular, if we can show that points which are adversarial for the base classiﬁer (points with f ( x + (cid:15) ) = 0 ) are dispersed , then this will imply larger certiﬁcates, because it makes it impossible for a perturbation in a single direction to move x towards the adversarial region.

Original: Second-order smoothing, above, is merely an example of this.
Revised: Second-order smoothing, above, is merely an example of this.

Original: We therefore introduce Gaussian Dipole smoothing .
Revised: We therefore introduce Gaussian Dipole smoothing .

Original: This is a method which, like second-order smoothing, also harnesses the geometrical distribution of the values of f ( x ) to improve certiﬁcates.
Revised: This is a method which, like second-order smoothing, also harnesses the geometrical distribution of the values of f ( x ) to improve certiﬁcates.

Original: However, unlike second-order smoothing, there is no explicit dependence on d in the empirical dipole smoothing bound.
Revised: However, unlike second-order smoothing, there is no explicit dependence on d in the empirical dipole smoothing bound.

Original: In this method, when we sample f ( x + (cid:15) ) when estimating p a ( x ) , we also sample f ( x − (cid:15) ) .
Revised: In this method, when we sample f ( x + (cid:15) ) when estimating p a ( x ) , we also sample f ( x − (cid:15) ) .

Original: This allows us to compute two quantities:
Revised: This allows us to compute two quantities:

Original: [Equation]
Revised: [Equation]

Original: The certiﬁcate we can calculate is then as follows:
Revised: The certiﬁcate we can calculate is then as follows:

Original: Theorem 3.
Revised: Theorem 3.

Original: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ , and for all f : R d → [0 , 1] ,
Revised: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ , and for all f : R d → [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: We also compute this bound by constructing the worst possible classiﬁer.
Revised: We also compute this bound by constructing the worst possible classiﬁer.

Original: In this case, the trick is that, if two adversarial sampled points are opposite one another
Revised: In this case, the trick is that, if two adversarial sampled points are opposite one another

Original: (i.e., f ( x + (cid:15) ) = f
Revised: (i.e., f ( x + (cid:15) ) = f

Original: ( x − (cid:15) ) = 0 ) then they cannot both contribute to the same adversarial “direction”.
Revised: ( x − (cid:15) ) = 0 ) then they cannot both contribute to the same adversarial “direction”.

Original: In the worst case, the “reﬂected” adversarial points form a plane opposite the base classiﬁer decision boundary (See Figure 4-b).
Revised: In the worst case, the “reﬂected” adversarial points form a plane opposite the base classiﬁer decision boundary (See Figure 4-b).

Original: In the extreme case where C N = 0 , the “worst case” classiﬁer is the same as for second-order smoothing.
Revised: In the extreme case where C N = 0 , the “worst case” classiﬁer is the same as for second-order smoothing.

Original: Experimentally, we simply need to lower-bound both C S and C N from samples.
Revised: Experimentally, we simply need to lower-bound both C S and C N from samples.

Original: This reduces the precision of our estimates, for two reasons: we have half as many independent samples for the same number of evaluations we must perform, and we are bounding two quantities, which requires halving the error probability for each.
Revised: This reduces the precision of our estimates, for two reasons: we have half as many independent samples for the same number of evaluations we must perform, and we are bounding two quantities, which requires halving the error probability for each.

Original: However, unlike second-order smoothing, there is no dependence on d : this allows for practical certiﬁcates of real-world datasets.[Figure]
Revised: However, unlike second-order smoothing, there is no dependence on d : this allows for practical certiﬁcates of real-world datasets.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Experimental results are presented in Figure 5, with further results in Appendix A. Because both dipole and second-order certiﬁcates reduce the precision with which empirical quantities needed for certiﬁcation can be estimated, but both provide strictly larger certiﬁcates at the population level, the key question becomes at what number of samples N does each higher-order method become beneﬁcial.
Revised: Experimental results are presented in Figures 5 and 6, with further results in Appendix B. Because both dipole and second-order certiﬁcates reduce the precision with which empirical quantities needed for certiﬁcation can be estimated, but both provide strictly larger certiﬁcates at the population level, the key question becomes at what number of samples N does each higher-order method become beneﬁcial.

Original: Note that in the ﬁgures, we are comparing the new methods to standard smoothing,  using the same N for standard smoothing as for the new method.
Revised: Note that in the ﬁgures, we are comparing the new methods to standard smoothing, [Figure] [Figure] using the same N for standard smoothing as for the new method.

Original: Due to the poor scaling of second-order certiﬁcates with dimension, we tested second-order smoothing on a low-dimensional dataset, 7 ×  MNIST.
Revised: Due to the poor scaling of secondorder certiﬁcates with dimension, we tested second-order smoothing on a low-dimensional dataset, 7 × 7 MNIST.

Original: However, signiﬁcant increases to certiﬁcates were not seen until N = 10 8 even on this dataset.
Revised: However, signiﬁcant increases to certiﬁcates were not seen until N = 10 7 even on this dataset.

Original: By contrast, dipole smoothing is beneﬁcial for many samples even at N = 10 6 .
Revised: By contrast, dipole smoothing is beneﬁcial for many images even when smaller numbers of smoothing samples are used.

Original: Because it scales to higher-dimensional data, we also tested Gaussian dipole smoothing on CIFAR-10 and ImageNet, where it lead to modest improvements in certiﬁcates.
Revised: Because it scales to higher-dimensional data, we also tested Gaussian dipole smoothing on CIFAR-10 and ImageNet, where it led to modest improvements in certiﬁcates, in particular at N = 10 6 .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we explored the limits of using gradient information to improve randomized smoothing certiﬁcates.
Revised: In this work, we explored the limits of using gradient information to improve randomized smoothing certiﬁcates.

Original: In particular, we introduced second-order smoothing certiﬁcates and showed tight and realizable upper bounds on their maximum achievable beneﬁts.
Revised: In particular, we introduced second-order smoothing certiﬁcates and showed tight and realizable upper bounds on their maximum achievable beneﬁts.

Original: We also proposed Gaussian dipole smoothing, a novel method for robustness certiﬁcation, which can improve smoothing-based robustness certiﬁcates even on large-scale data sets.
Revised: We also proposed Gaussian dipole smoothing, a novel method for robustness certiﬁcation, which can improve smoothing-based robustness certiﬁcates even on large-scale data sets.

Original: This introduces a broader question for future work: what other information about the spacial distribution of classes in randomized smoothing can be efﬁciently used to improve robustness certiﬁcates?
Revised: This introduces a broader question for future work: what other information about the spacial distribution of classes in randomized smoothing can be efﬁciently used to improve robustness certiﬁcates?

Original: A P ROOFS A. S IMPLE PROOF OF T HEOREM 1 FROM C OHEN ET AL . (2019)
Revised: A P ROOFS A. S IMPLE PROOF OF T HEOREM 1 FROM C OHEN ET AL . (2019)

Original: We ﬁrst provide a novel, simple, and intuitive proof for ﬁrst-order randomized smoothing: this will allow us to develop methods and notations for later proofs
Revised: We ﬁrst provide a novel, simple, and intuitive proof for ﬁrst-order randomized smoothing: this will allow us to develop methods and notations for later proofs.

Original: Theorem.
Revised: Theorem.

Original: Let (cid:15) ∼ N (0 , σ 2 I ) .
Revised: Let (cid:15) ∼ N (0 , σ 2 I ) .

Original: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ , and for all f : R d → [0 , 1] :
Revised: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ , and for all f : R d → [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: Where Φ is the normal cdf function and Φ − 1 is its inverse.
Revised: Where Φ is the normal cdf function and Φ − 1 is its inverse.

Original: Proof.
Revised: Proof.

Original: Let R =
Revised: Let R =

Original: (cid:107) x − x (cid:48)
Revised: (cid:107) x − x (cid:48)

Original: (cid:107) 2 .
Revised: (cid:107) 2 .

Original: Choose our basis so that x = 0 and x (cid:48) =
Revised: Choose our basis so that x = 0 and x (cid:48) =

Original: [ R, 0 , 0 , ..., 0] T (Note that by isometry, we still have (cid:15) ∼ N (0 , σ 2 I ) ).
Revised: [ R, 0 , 0 , ..., 0] T (Note that by isometry, we still have (cid:15) ∼ N (0 , σ 2 I ) ).

Original: Then deﬁne g : R → [0 , 1] :
Revised: Then deﬁne g : R → [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: Note that:
Revised: that:

Original: [Equation]
Revised: [Equation]

Original: Now, in one dimension, (cid:15) 1 ∼ N (0 , σ ) , and so has a pdf of z → σ − 1 Φ (cid:48) (cid:0) zσ (cid:1) , where Φ (cid:48) is the normal pdf function.
Revised: Now, in one dimension, (cid:15) 1 ∼ N (0 , σ ) , and so has a pdf of z → σ − 1 Φ (cid:48) (cid:0) zσ (cid:1) , where Φ (cid:48) is the normal pdf function.

Original: By the deﬁnition of expected value:
Revised: By the deﬁnition of expected value:

Original: [Equation]
Revised: [Equation]

Original: We perform a change of integration variables, using y =
Revised: We perform a change of integration variables, using y =

Original: Φ (cid:0) (cid:15) 1 σ (cid:1) (and noting that dyd(cid:15) 1 = σ − 1 Φ 
Revised: Φ (cid:0) (cid:15) 1 σ (cid:1) (and noting that dyd(cid:15) 1 = σ − 1 Φ (cid:48) (cid:0) (cid:15) 1 σ (cid:1) ):

Original: [Equation]
Revised: [Equation]

Original: (cid:0)
Revised: Note that:

Original: [Equation]
Revised: [Equation]

Original: 1 σ (cid:1) ):
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: Note that:
Revised: to simplify notation

Original: [Equation]
Revised: [Equation]

Original: Also, to simplify notation, deﬁne g Φ : [0 , 1] → [0 , 1] as g Φ ( y ) :
Revised: deﬁne g Φ : [0 , 1] → [Equation] 1] as g Φ ( y ) :

Original: = g ( σ Φ − 1 ( y )) .
Revised: = g ( σ Φ − 1 ( y )) .

Original: Then we have (combining Equations 14 and 15):
Revised: Then we have (combining Equations 14 and 15):

Original: [Equation]
Revised: [Equation]

Original: Fix the expectation at x , E (cid:15) 

Revised: Fix the expectation at x , E (cid:15)

Original: [ g ( (cid:15) 1 )] , at a constant C , let us consider the function g Φ which minimizes the expectation at x (cid:48) :
Revised: [ g ( (cid:15) 1 )] , at a constant C , let us consider the function g Φ which minimizes the expectation at x (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: However, note that e Φ − 1 ( y ) − R 2 2 σ 2 increases monotonically with y .
Revised: However, note that e Φ − 1 ( y ) − R 2 2 σ 2 increases monotonically with y .

Original: Then the minimum is achieved at:
Revised: Then the minimum is achieved at:

Original: [Equation]
Revised: [Equation]

Original: In terms of the function g ( z ) , this is:
Revised: In terms of the function g ( z ) [Equation] this is:

Original: [Equation]
Revised: [Equation]

Original: Then we can evaluate the minimum, using the form of the integral given in Equation 13:
Revised: Then we can evaluate the minimum [Equation] using the form of the integral given in Equation 13:

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of C and Equation [Equation] this is:
Revised: By the deﬁnition of C and Equation 12, this is:

Original: [Equation]
Revised: [Equation]

Original: which was to be proven
Revised: which was to be proven.

Original: [Equation]

Original: A. S ECOND O RDER S MOOTHING
Revised: A. S ECOND O RDER S MOOTHING

Original: Theorem 1.
Revised: Theorem 1.

Original: Let (cid:15) ∼ N (0 , σ 2 I ) .
Revised: Let (cid:15) ∼ N (0 , σ 2 I ) .

Original: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ [Equation] and for all f : R d → [0 , 1] ,
Revised: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ , and for all f : R d → [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Where Φ is the normal cdf function, Φ − 1 is its inverse, and a (cid:48) is the (unique) solution to
Revised: Where Φ is the normal cdf function, Φ − 1 is its inverse, and a (cid:48) is the (unique) solution to

Original: [Equation]
Revised: [Equation]

Original:  Further, for all pairs ( E (cid:15) [ f ( x + (cid:15) )]) , (cid:107)∇ x E (cid:15) [ f ( x + (cid:15) )] (cid:107) 2 ) which are possible, there exists a base classiﬁer f and an adversarial point x (cid:48) such that Equation 4 is an equality.
Revised: [Equation] Further, for all pairs ( E (cid:15) [ f ( x + (cid:15) )]) , (cid:107)∇ x E (cid:15) [ f ( x + (cid:15) )] (cid:107) 2 ) which are possible, there exists a base classiﬁer f and an adversarial point x (cid:48) such that Equation 4 is an equality.

Original: As show by Salman et al.
Revised: As show by Salman et al.

Original: (2019), we have, for all x (cid:48)(cid:48) ∈ R
Revised: (2019), we have, for all x (cid:48)(cid:48) ∈ R

Original: d :
Revised: d :

Original: [Equation]
Revised: [Equation]

Original: Under the choice of basis of the above proof (in particular, x = 0 ), when evaluated at x this becomes:
Revised: Under the choice of basis of the above proof (in particular, x = 0 ), when evaluated at x this becomes:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Let u :=
Revised: Let u :=

Original: [1 , 0 , 0 , ..., 0] T , and deﬁne g as in the above proof.
Revised: [1 , 0 , 0 , ..., 0] T , and deﬁne g as in the above proof.

Original: Note that:
Revised: Note that:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of expectation, and again using the change of integration variables y :=
Revised: By the deﬁnition of expectation, and again using the change of integration variables y :

Original: Φ (cid:0) (cid:15) 1 σ (cid:1) ,

Original: E

Original: [Equation]

Original: [Equation]

Original: g

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]

Original: Deﬁne:
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then, by Equations 16 and
Revised: Then, by Equations 16 and

Original: [Equation]
Revised: [Equation]

Original: and deﬁning g Φ [Equation] as above,
Revised: and deﬁning g Φ , as above,

Original: [Equation]
Revised: [Equation]

Original: Note that our constraints are linear in the space of functions; we can then introduce Lagrange multipliers:
Revised: Note that our constraints are linear in the space of functions; we can then introduce Lagrange multipliers:

Original: [Equation]
Revised: [Equation]

Original: This is simply the inner product between g Φ and a function: the inner product is minimized by setting g Φ = 1 where the expression is negative [Equation] and g Φ = 0 where the expression is positive:
Revised: This is simply the inner product between g Φ and a function: the inner product is minimized by setting g Φ = 1 where the expression is negative, and g Φ = 0 where the expression is positive:

Original: [Equation]
Revised: [Equation]

Original: Sign changes occur at:
Revised: Sign changes occur at:

Original: [Equation]
Revised: [Equation]

Original: Where W is the product-log (Lambert W) function.
Revised: Where W is the product-log (Lambert W) function.

Original: This returns zero, one, or two possible values, depending on the argument (zero values if the argument < − e − 1 , two values on [ − e − 1 , 0) , and one value on non-negative arguments).
Revised: This returns zero, one, or two possible values, depending on the argument (zero values if the argument < − e − 1 , two values on [ − e − 1 , 0) [Equation] and one value on non-negative arguments).

Original: Therefore there are at most two sign changes.
Revised: Therefore there are at most two sign changes.

Original: Also, note that as y → 1 , Φ − 1 ( y ) → ∞ , taking the limit in Equation 32, we know thatg ∗ Φ (1)
Revised: Also, note that as y → 1 , Φ − 1 ( y ) → ∞ , taking the limit in Equation 32, we know that

Original: = 0 .
Revised: [Equation]

Original: Therefore, taking into account the constraint (cid:82) 1 0 g Φ ( y ) dy =
Revised: Therefore, taking into account the constraint (cid:82) 1 0 g Φ ( y ) dy =

Original: C , we know that g ∗ Φ is either:
Revised: C , we know that g ∗ Φ is either:

Original: • 0 everywhere (and C = 0 ), if no sign changes • 1 at y < C , 0 otherwise, if one sign change • 1 on the interval [ a, a + C ] , 0 otherwise, for some a ∈ [0 , 1 − C ] , if two sign changes
Revised: • 0 everywhere (and C = 0 ), if no sign changes • 1 at y < C , 0 otherwise, if one sign change • 1 on the interval [ a, a + C ] , 0 otherwise, for some a ∈ [0 , 1 − C ] , if two sign changes

Original: In fact, the ﬁnal case includes the ﬁrst two, so all that we need to do now is ﬁnd a to satisfy the C (cid:48) constraint.
Revised: In fact, the ﬁnal case includes the ﬁrst two, so all that we need to do now is ﬁnd a to satisfy the C (cid:48) constraint.

Original: This constraint (Equation 27) becomes:
Revised: This constraint (Equation 27) becomes:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Because Φ − 1 ( y ) is monotone increasing, the LHS of Equation 34 is a monotone increasing function of a .
Revised: Because Φ − 1 ( y ) is monotone increasing, the LHS of Equation 34 is a monotone increasing function of a .

Original: Using the indeﬁnite integral of Φ − 1 :
Revised: Using the indeﬁnite integral of Φ − 1 :

Original: [Equation]
Revised: [Equation]

Original: Where erf − 1 is the inverse error function.
Revised: Where erf − 1 is the inverse error function.

Original: Then the constraint becomes:
Revised: Then the constraint becomes:

Original: [Equation]
Revised: [Equation]

Original: We can now evaluate the value of the smoothed function at x (cid:48) , again using the form of the integral given in Equation 13:
Revised: We can now evaluate the value of the smoothed function at x (cid:48) , again using the form of the integral given in Equation 13:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: If we consider the form of this integral in Equation 
Revised: If we consider the form of this integral in Equation 29, which reduces to simply

Original: [Equation]
Revised: [Equation]

Original: which reduces to simply:

Original: [Equation]
Revised: [Equation]

Original: 37) we see that is is a monotonically increasing function in a .
Revised: we see that is is a monotonically increasing function in a .

Original: Furthermore [Equation] we have that the LHS of
Revised: Furthermore, we have that the LHS of

Original: Equation 35 is monotonic in a .
Revised: Equation 35 is monotonic in a .

Original: Therefore, if we deﬁne a (cid:48) as the solution to:
Revised: Therefore, if we deﬁne a (cid:48) as the solution to:

Original: [Equation]
Revised: [Equation]

Original: Then by Equation [Equation] we know a (cid:48) ≤ a .
Revised: Then by Equation 26, we know a (cid:48) ≤ a .

Original: Then because the RHS of Equation36 is also monotonic in a ,
Revised: Then because the RHS of Equation36 is also monotonic in a ,

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnitions of g and C [Equation] and because the RHS is monotonically decreasing in R (See Equation 37), we can conclude the theorem as stated.
Revised: By the deﬁnitions of g and C , and because the RHS is monotonically decreasing in R (See Equation 37), we can conclude the theorem as stated.

Original: Further, we can conclude that an equality case is possible by noting that it is achieved by the function g ∗ ( z ) as described above: the minimal f ∗ ( z ) can then be constructed as f ∗ ( z, · , cdot, ... ) :
Revised: Further, we can conclude that an equality case is possible by noting that it is achieved by the function g ∗ ( z ) as described above: the minimal f ∗ ( z ) can then be constructed as f ∗ ( z, · , · , ... ) :

Original: = g ∗ ( z ) .
Revised: = g ∗ ( z ) .

Original: Note that we also need Equation 26 to be tight: this is achieved where the adversarial direction x (cid:48) − x is parallel to the gradient of the smoothed function.
Revised: Note that we also need Equation 26 to be tight: this is achieved where the adversarial direction x (cid:48) − x is parallel to the gradient of the smoothed function.

Original: A. P RACTICAL C ERTIFICATION A LGORITHM 
Revised: A. P RACTICAL C ERTIFICATION A LGORITHM Deﬁne

Original: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: Let V :=
Revised: Let V :=

Original: E (cid:15),(cid:15) (cid:48)
Revised: E (cid:15),(cid:15) (cid:48)

Original: [( (cid:15)f ( x + (cid:15) )) · ( (cid:15) (cid:48)
Revised: [( (cid:15)f ( x + (cid:15) )) · ( (cid:15) (cid:48)

Original: f ( x + (cid:15) (cid:48) ))] , and ˜ V be its empirical estimate.
Revised: f ( x + (cid:15) (cid:48) ))] , and ˜ V be its empirical estimate.

Original: If n pairs of samples ( = N/ 2 ) are used to estimate V , then, with probability at most η , E [ V ] − ˜ V ≥ t , where:
Revised: If n pairs of samples ( = N/ 2 ) are used to estimate V , then, with probability at most η , E [ V ] − ˜ V ≥ t , where:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Wainwright (2019) gives the following condition for any centered (mean-zero) sub-exponential random variable X :
Revised: Wainwright (2019) gives the following condition for any centered (mean-zero) sub-exponential random variable X :

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A centered R.V. X is (a,b)-subexponential if:
Revised: A centered R.V. X is (a,b)-subexponential if:

Original: [Equation]
Revised: [Equation]

Original: First, we establish bounds for (cid:15) · (cid:15) (cid:48) .
Revised: First, we establish bounds for (cid:15) · (cid:15) (cid:48) .

Original: (This can be considered a simpliﬁed case of Gaussian chaos of the second order, see Vershynin (2018)).
Revised: (This can be considered a simpliﬁed case of Gaussian chaos of the second order, see Vershynin (2018)).

Original: For each i ∈ [ d ] , (cid:15) i and (cid:15) (cid:48)
Revised: For each i ∈ [ d ] , (cid:15) i and (cid:15) (cid:48)

Original: i are independent Gaussian random variables.
Revised: i are independent Gaussian random variables.

Original: Recall the moment-generating function for a Gaussian:
Revised: Recall the moment-generating function for a Gaussian:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Then for the product (cid:15) i (cid:15) (cid:48)
Revised: Then for the product (cid:15) i (cid:15) (cid:48)

Original: i we have that:
Revised: i we have that:

Original: [Equation]
Revised: [Equation]

Original: Note that this has a similar form to the moment generating function of the Chi-squared distribution for k = 1 :
Revised: Note that this has a similar form to the moment generating function of the Chi-squared distribution for k = 1 :

Original: [Equation]
Revised: [Equation]

Original: Then:
Revised: Then:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Where the ﬁnal inequality can be shown by observing that, if λ 2 σ 4 ≤ 1 / 2 :
Revised: Where the ﬁnal inequality can be shown by observing that, if λ 2 σ 4 ≤ 1 / 2 :

Original: [Equation]
Revised: [Equation]

Original: 48) and taking square roots.
Revised: and taking square roots.

Original: Because (cid:15) i (cid:15) (cid:48)
Revised: Because (cid:15) i (cid:15) (cid:48)

Original: i is centered, this implies that (cid:15) i
Revised: i is centered, this implies that (cid:15) i

Original: (cid:15) (cid:48)
Revised: (cid:15) (cid:48)

Original: i is ( √ 2 σ 2 , √ 2 σ 2 ) - subexponential.
Revised: i is ( √ 2 σ 2 , √ 2 σ 2 ) - subexponential.

Original: Now, (cid:15) · (cid:15) (cid:48) is simply the sum of d such identical, independent, centered subexponential variables: by (Wainwright (2019) Equation 2.18), we conclude that (cid:15) · (cid:15) (cid:48) is ( √ 2 σ 2 √ d, √ 2 σ 2 ) - subexponential.
Revised: Now, (cid:15) · (cid:15) (cid:48) is simply the sum of d such identical, independent, centered subexponential variables: by (Wainwright (2019) Equation 2.18), we conclude that (cid:15) · (cid:15) (cid:48) is ( √ 2 σ 2 √ d, √ 2 σ 2 ) - subexponential.

Original: This implies:
Revised: This implies:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Recall that the quantity which we are measuring is ( (cid:15)f ( (cid:15) )) · ( (cid:15) (cid:48) f
Revised: Recall that the quantity which we are measuring is ( (cid:15)f ( (cid:15) )) · ( (cid:15) (cid:48) f

Original: ( (cid:15) (cid:48) )) .
Revised: ( (cid:15) (cid:48) )) .

Original: For notation convenience, let v ( (cid:15), (cid:15) (cid:48) ) :
Revised: For notation convenience, let v ( (cid:15), (cid:15) (cid:48) ) :

Original: R d × R d → { 0 , 1 } be deﬁned as f ( (cid:15) )
Revised: R d × R d → { 0 , 1 } be deﬁned as f ( (cid:15) )

Original: f ( (cid:15) (cid:48) ) , so that the quantity of interest is
Revised: f ( (cid:15) [Equation] ) , so that the quantity of interest is

Original: [Equation]
Revised: [Equation]

Original: We further deﬁne a centered version of this quantity
Revised: We further deﬁne a centered version of this quantity

Original: V (cid:48)

Original: [Equation]
Revised: [Equation]

Original: We now introduce an important lemma:
Revised: We now introduce an important lemma:

Original: Lemma 1.
Revised: Lemma 1.

Original: V (cid:48) is (2 √ 2 σ 2 √ d [Equation] 2 √ 2 σ 2 ) -subexponential.
Revised: V (cid:48) is (2 √ 2 σ 2 √ d, 2 √ 2 σ 2 ) -subexponential.

Original: Proof.
Revised: Proof.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then:
Revised: Then:

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: Deﬁne:
Revised: Deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Trivially, we have:
Revised: Trivially, we have:

Original: [Equation]
Revised: [Equation]

Original: However, note that also:
Revised: However, note that also:

Original: [Equation]
Revised: [Equation]

Original: Then by Jensen’s inequality
Revised: Then by Jensen’s inequality

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similarly:
Revised: Similarly:

Original: [Equation]
Revised: [Equation]

Original: Again, by Jensen’s inequality:
Revised: Again, by Jensen’s inequality:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Because p ≤ [Equation] we then have:
Revised: Because p ≤ 1 , we then have:

Original: [Equation]
Revised: [Equation]

Original: [Equation] In other words [Equation] we have shown that both A and B are both ( √ 2 σ 2 √ d, √ 2 σ 2 ) -subexponential.
Revised: In other words, we have shown that both A and B are both ( √ 2 σ 2 √ d, √ 2 σ 2 ) -subexponential.

Original: Then, by Cauchy-Schwartz:
Revised: Then, by Cauchy-Schwartz:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] In other words, V (cid:48) is (2 √ 2 σ 2 √ d, 2 √ 2 σ 2 ) -subexponential.
Revised: In other words, V (cid:48) is (2 √ 2 σ 2 √ d, 2 √ 2 σ 2 ) -subexponential.

Original: Finally, using the form of the one-sided Bernstein tail bound for subexponential random variables given in Wainwright (2019), we have, given n measurements and an empirical mean estimate of V as ˜ V :
Revised: Finally, using the form of the one-sided Bernstein tail bound for subexponential random variables given in Wainwright (2019), we have, given n measurements and an empirical mean estimate of V as ˜ V :

Original: [Equation]
Revised: [Equation]

Original: Then, given a failure rate η , we can compute the minimum deviation t such that the failure probability is less than η :
Revised: Then, given a failure rate η , we can compute the minimum deviation t such that the failure probability is less than η :

Original: [Equation]
Revised: [Equation]

Original: A. D IPOLE S MOOTHING
Revised: A. D IPOLE S MOOTHING

Original: Theorem 3.
Revised: Theorem 3.

Original: Let (cid:15) ∼ N (0 , σ 2 I ) .
Revised: Let (cid:15) ∼ N (0 , σ 2 I ) .

Original: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ , and for all f : R d → [0 , 1] , deﬁne:
Revised: For all x , x (cid:48) with (cid:107) x − x (cid:48) (cid:107) 2 < ρ , and for all f : R d → [0 , 1] , deﬁne:

Original: [Equation]
Revised: [Equation]

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Where Φ is the normal cdf function and Φ − 1 is its inverse.
Revised: Where Φ is the normal cdf function and Φ − 1 is its inverse.

Original: Proof.
Revised: Proof.

Original: As in the proof of Theorem A.1, let R
Revised: As in the proof of Theorem A.1, let R

Original: =
Revised: =

Original: (cid:107) x − x (cid:48)
Revised: (cid:107) x − x (cid:48)

Original: (cid:107) 2 , and choose our basis so that x = 0 and x (cid:48) =
Revised: (cid:107) 2 , and choose our basis so that x = 0 and x (cid:48) =

Original: [ R, 0 , 0 , ..., 0] T .
Revised: [ R, 0 , 0 , ..., 0] T .

Original: First, for f : R d → [0 , 1] , we deﬁne a decomposition into symmetric and non-symmetric components, f S , f N : R d → [0 , 1] : 
Revised: First, for f : R d → [0 , 1] , we deﬁne a decomposition into symmetric and non-symmetric components, f S , f N : R d → [0 , 1] : [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that f ( (cid:15) ) =
Revised: Note that f ( (cid:15) ) =

Original: f S ( (cid:15) )
Revised: f S ( (cid:15) )

Original: + f N ( (cid:15) ) and also that f S ( (cid:15) )
Revised: + f N ( (cid:15) ) and also that f S ( (cid:15) )

Original: =
Revised: =

Original: f S ( − (cid:15) ) .
Revised: f S ( − (cid:15) ) .

Original: Deﬁne g S ( z ) , g N ( z ) : R → [0 , 1] by analogy to Equation 12.
Revised: Deﬁne g S ( z ) , g N ( z ) : R → [0 , 1] by analogy to Equation 12.

Original: By linearity of expectation, note that g ( z ) =
Revised: By linearity of expectation, note that g ( z ) =

Original: g S ( z ) + g N ( z ) .
Revised: g S ( z ) + g N ( z ) .

Original: Also note that:
Revised: Also note that:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Similarly, deﬁne g S Φ and g N Φ .
Revised: Similarly, deﬁne g S Φ and g N Φ .

Original: We still have:
Revised: We still have:

Original: [Equation]
Revised: [Equation]

Original: Also (using Φ − 1 ( y ) =
Revised: Also (using Φ − 1 ( y ) =

Original: − Φ − 1 (1 − y ) ):
Revised: − Φ − 1 (1 − y ) ):

Original: [Equation]
Revised: [Equation]

Original: Note that all of the mechanics of the proof of Theorem 1 can be applied to f, f S and f N .
Revised: Note that all of the mechanics of the proof of Theorem 1 can be applied to f, f S and f N .

Original: Following  13, we have:
Revised: Following Equation 13, we have:

Original: [Equation]
Revised: [Equation]

Original: We may then write the minimization in Equation 17, ﬁxing C N and C S as constants separately 1 :
Revised: We may then write the minimization in Equation 17, ﬁxing C N and C S as constants separately 2 :

Original: [Equation]
Revised: [Equation]

Original: The second minimum can be computed as in the proof of Theorem 1, it is simply Φ (cid:0) Φ − 1 ( C N ) − Rσ (cid:1) .For the ﬁrst minimum
Revised: The second minimum can be computed as in the proof of Theorem 1, it is simply Φ (cid:0) Φ − 1 ( C N ) − Rσ (cid:1) .

Original: [Equation] we consider the additional constraint [Equation] that g S Φ ( y ) =
Revised: For the ﬁrst minimum, we consider the additional constraint, that g S Φ ( y ) =

Original: g S Φ (1 − y ) .
Revised: g S Φ (1 − y ) .

Original: Then we can rewrite the integral as:
Revised: Then we can rewrite the integral as:

Original: [Equation]
Revised: [Equation]

Original: So the minimization becomes:
Revised: So the minimization becomes:

Original: [Equation]
Revised: [Equation]

Original: Note that cosh(Φ − 1 ( y )) is a monotonically decreasing function of y on the range [0 , 12 ] .
Revised: Note that cosh(Φ − 1 ( y )) is a monotonically decreasing function of y on the range [0 , 12 ] .

Original: Then the minimum is achieved by the function:
Revised: Then the minimum is achieved by the function:

Original: [Equation]
Revised: [Equation]

Original: Where the value in the domain [
Revised: Where the value in the domain [

Original: [Equation]
Revised: [Equation]

Original: 1] can be computed using g S ∗ Φ (1 − y )
Revised: 1] can be computed using g S ∗ Φ (1 − y )

Original: =
Revised: =

Original: g S ∗ Φ ( y )
Revised: g S ∗ Φ ( y )

Original: In terms of the function g S ( z ) [Equation] this is:
Revised: In terms of the function g S ( z ) , this is:

Original: [Equation]
Revised: [Equation]

Original: We can now evaluate the integral, again using the form of the integral given in Equation 13:
Revised: We can now evaluate the integral, again using the form of the integral given in Equation 13:

Original: [Equation]
Revised: [Equation]

Original: So, combining the g S and g N terms, we have:
Revised: So, combining the g S and g N terms [Equation] we have:

Original: [Equation]
Revised: [Equation]

Original: From Equation 12, and noting that the last two terms together are monotonically decreasing 2 with
Revised: From Equation [Equation] and noting that the last two terms together are monotonically decreasing 3 with

Original: R , we complete the proof.
Revised: R [Equation] we complete the proof

Revised: [Equation]

Original: B A DDITIONAL E XPERIMENTS
Revised: B A DDITIONAL E XPERIMENTS

Original: Here, we present experiments at a wider range of parameters.
Revised: Here, we present experiments at a wider range of parameters.

Original: For all ﬁgures, images misclassiﬁed or not certiﬁed for both the baseline and the tested method are not counted: the total test set size isfor MNIST, 500 for CIFAR and ImageNet with N = 10 5 , and 100 for CIFAR and ImageNet with
Revised: For all ﬁgures, images misclassiﬁed or not certiﬁed for both the baseline and the tested method are not counted: the total test set size isfor MNIST, 500 for CIFAR and ImageNet with N = 10 5 , and 100 for CIFAR and ImageNet with

Original: N = 10 6 .
Revised: N = 10 6 .

Original: For all experiments, N 0 = 100 .
Revised: For all experiments, N 0 = 100 .

Original: Also, note that we test independently for the baseline and higher-order methods (i.e., we use different smoothing samples).
Revised: Also, note that we test independently for the baseline and higher-order methods (i.e., we use different smoothing samples).

Original: This is necessary to compare fairly to dipole smoothing, where the sampling method if different; however, it does lead to some noise, especially at N = 10 5 .
Revised: This is necessary to compare fairly to dipole smoothing, where the sampling method is different; however, it does lead to some noise, especially at N = 10 5 .

Original: B. N UMBER OF SMOOTHING SAMPLES N .

Original: In Figure 6, we present results at a wider range of number of samples.

Original: We see that dipole smoothing is effective even at N = 10 5 , while Second order Smoothing is only effective at N = 10 7 .

Original: B. N OISE LEVEL σ .
Revised: B. N OISE LEVEL σ .

Original: We see (Figures 7 and 8) that at a smaller level of noise ( σ = 0 .
Revised: We see (Figures 7 and 8) that at a smaller level of noise ( σ = 0 .

Original: 12 ), the effect of higher-order smoothing is diminished.
Revised: 12 ), the effect of higher-order smoothing is diminished.

Original: This can be understood in terms of the curves in Figure 1-b: lower noise leads to more inputs with higher p a , which reduces the beneﬁt of the higher-order certiﬁcate.
Revised: This can be understood in terms of the curves in Figure 1-b: lower noise leads to more inputs with higher p a , which reduces the beneﬁt of the higher-order certiﬁcate.

Original: Conversely, higher noise increases the effects of the higher-order certiﬁcates, although it also leads to decreased total accuracy.
Revised: Conversely, higher noise increases the effects of the higher-order certiﬁcates, although it also leads to decreased total accuracy.

Original: The dipole certiﬁcate underperforms at N = 10 5 , σ = 0 .
Revised: The dipole certiﬁcate underperforms at N = 10 5 , σ = 0 .

Original: 5 : this is likely due to the increase in estimation error, which becomes signiﬁcant near p a = 0 .
Revised: 5 : this is likely due to the increase in estimation error, which becomes signiﬁcant near p a = 0 .

Original: B. D IMENSIONALITY d
Revised: B. D IMENSIONALITY d

Original: To test second-order smoothing on a lower-dimensional dataset, we performed PCA on the 7 ×
Revised: To test second-order smoothing on a lower-dimensional dataset, we performed PCA on the 7 ×

Original: MNIST images, and classiﬁed using the top 10 principal components.
Revised: MNIST images, and classiﬁed using the top 10 principal components.

Original: ( d = 10 ).
Revised: ( d = 10 ).

Original: Results are shown in Figures 9, 10, and 11.
Revised: Results are shown in Figures 9, 10, and 11.

Original: We see that, at N = 10 6 , second-order smoothing has a marginal positive impact at this smaller scale.
Revised: We see that, at N = 10 6 , second-order smoothing has a marginal positive impact at this smaller scale.

Original: B. D IPOLE S MOOTHING ON CIFAR- 
Revised: B. D IPOLE S MOOTHING ON CIFAR-

Original: In Figure 13, we see experiments on CIFAR-10 using dipole smoothing, for a range of σ ∈ { 0 .
Revised: In Figure 12, we see experiments on CIFAR-10 using dipole smoothing, for a range of σ ∈ { 0 .

Original: 50 , 1 . 00 } and N ∈ { 10 5 , 10 6 } .
Revised: 50 , 1 . 00 } and N ∈ { 10 5 , 10 6 } .

Original: Note that dipole smoothing appears to be beneﬁcial even at N = 10 5 on CIFAR-10, at all noise levels ≥ 0.25.
Revised: Note that dipole smoothing appears to be beneﬁcial even at N = 10 5 on CIFAR-10, at all noise levels ≥ 0.25.

Revised: these terms given in Equation

Original: B. D IPOLE S MOOTHING ON I MAGE N ET
Revised: B. D IPOLE S MOOTHING ON I MAGE N ET

Original: In Figure 13, we see experiments on ImageNet using dipole smoothing, for a range of σ ∈ { 0 . 25 , 0 .
Revised: In Figure 13, we see experiments on ImageNet using dipole smoothing, for a range of σ ∈ { 0 . 25 , 0 .

Original: 50 , 1 . 00 } and N ∈ { 10 5 , 10 6 } .
Revised: 50 , 1 . 00 } and N ∈ { 10 5 , 10 6 } .

Original: There is an anomalous result for σ = 0 .
Revised: There is an anomalous result for σ = 0 .

Original: 50 , N = 10 5 , in that this is the only case where dipole smoothing appears to perform worse than standard smoothing.
Revised: 50 , N = 10 5 , in that this is the only case where dipole smoothing appears to perform worse than standard smoothing.

Original: However, this turns out to be a computational artifact.
Revised: However, this turns out to be a computational artifact.

Original: At both σ = 0 .
Revised: At both σ = 0 .

Original: 50 and σ = 0 .
Revised: 50 and σ = 0 .

Original: 25 , there are a large number of images where every smoothing sample is correctly classiﬁed, so p a is as close to 1 as the measurement bounds allow.
Revised: 25 , there are a large number of images where every smoothing sample is correctly classiﬁed, so p a is as close to 1 as the measurement bounds allow.

Original: Note that if p a truly equals 1, the certiﬁed radius is inﬁnite, so in this domain, the reported certiﬁcate is entirely a function of the estimation error.
Revised: Note that if p a truly equals 1, the certiﬁed radius is inﬁnite, so in this domain, the reported certiﬁcate is entirely a function of the estimation error.

Original: Because dipole smoothing reduces measurement precision, these samples have somewhat smaller certiﬁed radii under dipole smoothing, especially at small N .
Revised: Because dipole smoothing reduces measurement precision, these samples have somewhat smaller certiﬁed radii under dipole smoothing, especially at small N .

Original: However, this gap should be exactly proportional to σ .
Revised: However, this gap should be exactly proportional to σ .

Original: The cause of the anomaly is the fact that our code (adapted from (Cohen et al., 2019)) records each radius to three signiﬁcant ﬁgures.
Revised: The cause of the anomaly is the fact that our code (adapted from (Cohen et al., 2019)) records each radius to three signiﬁcant ﬁgures.

Original: At σ = 0 .
Revised: At σ = 0 .

Original: 5 , for an image where all noise samples are correctly classiﬁed, the ratio of the dipole smoothing radius to the standard smoothing radius is reported as 1 . 89 / 1 .
Revised: 5 , for an image where all noise samples are correctly classiﬁed, the ratio of the dipole smoothing radius to the standard smoothing radius is reported as 1 . 89 / 1 .

Original: 91 = 98 .
Revised: 91 = 98 .

Original: 95% , while for σ = 0 .
Revised: 95% , while for σ = 0 .

Original: 25 it is reported as 0 .
Revised: 25 it is reported as 0 .

Original: 953 = 99 .
Revised: 953 = 99 .

Original: This explains the large number of samples with reported > 1% decrease in certiﬁcates for σ = 0 .
Revised: This explains the large number of samples with reported > 1% decrease in certiﬁcates for σ = 0 .

Original: 50 , N = 10 5 .
Revised: 50 , N = 10 5 .

Revised: C A BSOLUTE C ERTIFICATES FOR MAIN - TEXT EXPERIMENTS

Revised: In Figures 14 and 15, we show the absolute, rather than relative, values of the certiﬁcates reported in the main text, compared to the baseline ﬁrst-order randomized smoothing.

Revised: We see that the beneﬁt of the proposed techniques is greatest for images with small absolute certiﬁcates, and that, on CIFAR10 and ImageNet, there is some disadvantage to dipole smoothing on the largest possible certiﬁcates, where all smoothing samples are classiﬁed correctly.

Revised: This is because, for these images, the certiﬁcate depends entirely on estimation error.

Revised: [Figure]

Revised: [Figure]