Original: F ILLING THE S OAP B UBBLES : E FFICIENT B LACK -B OX A DVERSARIAL C ERTIFICATION WITH N ON -G AUSSIAN S MOOTHING
Revised: F ILLING THE S OAP B UBBLES : E FFICIENT B LACK -B OX A DVERSARIAL C ERTIFICATION WITH N ON -G AUSSIAN S MOOTHING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Randomized classiﬁers have been shown to provide a promising approach for achieving certiﬁed robustness against adversarial attacks in deep learning.
Revised: Randomized classiﬁers have been shown to provide a promising approach for achieving certiﬁed robustness against adversarial attacks in deep learning.

Original: However, most existing methods only leverage Gaussian smoothing noise and only work for (cid:96) 2 perturbation.
Revised: However, most existing methods only leverage Gaussian smoothing noise and only work for (cid:96) 2 perturbation.

Original: We propose a general framework of adversarial certiﬁcation with non-Gaussian noise and for more general types of attacks, from a uniﬁed functional optimization perspective.
Revised: We propose a general framework of adversarial certiﬁcation with non-Gaussian noise and for more general types of attacks, from a uniﬁed variational optimization perspective.

Original: Our new framework allows us to identify a key trade-off between accuracy and robustness via designing smoothing distributions, helping to design two new families of non-Gaussian smoothing distributions that work more efﬁciently for (cid:96) 2 and (cid:96) ∞ attacks, respectively.
Revised: Our new framework allows us to identify a key trade-off between accuracy and robustness via designing smoothing distributions, helping to design two new families of non-Gaussian smoothing distributions that work more efﬁciently for (cid:96) 2 and (cid:96) ∞ attacks, respectively.

Original: Our proposed methods achieve better results than previous works and provide a new perspective on randomized smoothing certiﬁcation.
Revised: Our proposed methods achieve better results than previous works and provide a new perspective on randomized smoothing certiﬁcation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have achieved state-of-the-art performance on many tasks such as image classiﬁcation (He et al., 2016; Lu et al., 2018) and language modeling (Devlin et al., 2019).
Revised: Deep neural networks have achieved state-of-the-art performance on many tasks such as image classiﬁcation (He et al., 2016; Lu et al., 2018) and language modeling (Devlin et al., 2019).

Original: Nonetheless, modern deep learning models have been shown to be highly sensitive to small and adversarially crafted perturbations on the inputs (Goodfellow et al., 2015), which means a human-imperceptible changes on inputs could cause the model to make dramatically different predictions.
Revised: Nonetheless, modern deep learning models have been shown to be highly sensitive to small and adversarially crafted perturbations on the inputs (Goodfellow et al., 2015), which means a human-imperceptible changes on inputs could cause the model to make dramatically different predictions.

Original: Although many robust training algorithms have been developed to overcome adversarial attacking, most heuristically developed methods can be shown to be broken by more powerful adversaries eventually, (e.g., Athalye et al., 2018;
Revised: Although many robust training algorithms have been developed to overcome adversarial attacking, most heuristically developed methods can be shown to be broken by more powerful adversaries eventually, (e.g., Athalye et al., 2018;

Original: Madry et al., 2018; Zhang et al., 2019; Wang et al., 2019).
Revised: Madry et al., 2018; Zhang et al., 2019; Wang et al., 2019).

Original: This casts an urgent demand for developing robust classiﬁers with provable worst case guarantees.
Revised: This casts an urgent demand for developing robust classiﬁers with provable worst case guarantees.

Original: One promising approach for certiﬁably robustness is the recent randomized smoothing method (e.g.,
Revised: One promising approach for certiﬁably robustness is the recent randomized smoothing method (e.g.,

Original: Cohen et al., 2019; Salman et al., 2019; Lee et al., 2019; Li et al., 2019; Lecuyer et al., 2018), which constructs smoothed classiﬁers with certiﬁable robustness by introducing noise on the inputs.
Revised: Cohen et al., 2019; Salman et al., 2019; Lee et al., 2019; Li et al., 2019; Lecuyer et al., 2018), which constructs smoothed classiﬁers with certiﬁable robustness by introducing noise on the inputs.

Original: Compared with the other more traditional veriﬁcation approaches (e.g. Wong & Kolter, 2017; Jordan et al., 2019; Dvijotham et al., 2018) that exploits special structures of the neural networks (such as the properties of ReLU), the randomized smoothing methods work more ﬂexibly on general blackbox classiﬁers and is shown to be more scalable and provide tighter bounds on challenging datasets such as ImageNet (Deng et al., 2009).
Revised: Compared with the other more traditional veriﬁcation approaches (e.g. Wong & Kolter, 2017; Jordan et al., 2019; Dvijotham et al., 2018) that exploits special structures of the neural networks (such as the properties of ReLU), the randomized smoothing methods work more ﬂexibly on general blackbox classiﬁers and is shown to be more scalable and provide tighter bounds on challenging datasets such as ImageNet (Deng et al., 2009).

Original: However, the existing randomized smoothing methods can only work against (cid:96) 2 attack, in which the perturbations are allowed within an (cid:96) 2 ball of certain radius.
Revised: However, the existing randomized smoothing methods can only work against (cid:96) 2 attack, in which the perturbations are allowed within an (cid:96) 2 ball of certain radius.

Original: A stronger type of attack, such as the (cid:96) ∞ attacks, is much more challenging to defense and verify due to the larger set of perturbations, but is more relevant in practice.
Revised: A stronger type of attack, such as the (cid:96) ∞ attacks, is much more challenging to defense and verify due to the larger set of perturbations, but is more relevant in practice.

Original: In addition, all the existing randomized smoothing methods use Gaussian noise for smoothing.
Revised: In addition, all the existing randomized smoothing methods use Gaussian noise for smoothing.

Original: Although appearing to be a natural choice, one of our key observations is that Gaussian distributions is in fact a rather sub-optimal choice in high dimensional spaces, even for (cid:96) 2 attack.
Revised: Although appearing to be a natural choice, one of our key observations is that Gaussian distributions is in fact a rather sub-optimal choice in high dimensional spaces, even for (cid:96) 2 attack.

Original: This is due to a counter-intuitive phenomenon in high dimensional spaces (Vershynin, 2018) that almost all of the probability mass of standard Gaussian distribution concentrates around the sphere of radius one (and hence “soap bubble” in the title), instead of the center point (which corresponds to the original input).
Revised: This is due to a counter-intuitive phenomenon in high dimensional spaces (Vershynin, 2018) that almost all of the probability mass of standard Gaussian distribution concentrates around the sphere of radius one (and hence “soap bubble” in the title), instead of the center point (which corresponds to the original input).

Original: As a result, the variance of the Gaussian noise needs to be sufﬁciently small to yield good approximation to the original classiﬁers (by squeezing the “soap bubble” towards the center point), which, however, makes it difﬁcult to verify due to the small noise.
Revised: As a result, the variance of the Gaussian noise needs to be sufﬁciently small to yield good approximation to the original classiﬁers (by squeezing the “soap bubble” towards the center point), which, however, makes it difﬁcult to verify due to the small noise.

Original: Further, for the more challenging (cid:96) ∞ attack, Gaussian smoothing provably degenerates in high dimensions.
Revised: Further, for the more challenging (cid:96) ∞ attack, Gaussian smoothing provably degenerates in high dimensions.

Original: Our contribution We propose a general framework of adversarial certiﬁcation using nonGaussian smoothing noises, based on a new perspective from functional optimization.
Revised: Our contribution We propose a general framework of adversarial certiﬁcation using nonGaussian smoothing noises, based on a new perspective from variational optimization.

Original: Our framework re-derives the method of Cohen et al.
Revised: Our framework re-derives the method of Cohen et al.

Original: as a special case, and is applicable to more general families of non-Gaussian smoothing distributions and more types of attacks beyond (cid:96) 2 norm.
Revised: as a special case, and is applicable to more general families of non-Gaussian smoothing distributions and more types of attacks beyond (cid:96) 2 norm.

Original: Importantly, our new framework reveals a fundamental trade-off between accuracy and robustness for guiding better choices of smoothing distributions.
Revised: Importantly, our new framework reveals a fundamental trade-off between accuracy and robustness for guiding better choices of smoothing distributions.

Original: Leveraging our insight, we develop two new families of distributions for better certiﬁcation results on (cid:96) 2 and (cid:96) ∞ attacks, respectively.
Revised: Leveraging our insight, we develop two new families of distributions for better certiﬁcation results on (cid:96) 2 and (cid:96) ∞ attacks, respectively.

Original: Efﬁcient computational approaches are developed to enable our method in practice.
Revised: Efﬁcient computational approaches are developed to enable our method in practice.

Original: Empirical results show that our new framework and smoothing distributions signiﬁcantly outperform the existing approaches for both (cid:96) 2 and (cid:96) ∞ attacking, on challenging datasets such as CIFAR-10 and ImageNet.
Revised: Empirical results show that our new framework and smoothing distributions signiﬁcantly outperform the existing approaches for both (cid:96) 2 and (cid:96) ∞ attacking, on challenging datasets such as CIFAR-10 and ImageNet.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Empirical Defenses Since Szegedy et al.
Revised: Empirical Defenses Since Szegedy et al.

Original: and Goodfellow et al. (2015), many previous works have focused on utilizing small perturbation δ under certain constraint, e.g. in a (cid:96) p norm ball, to attack a neural network.
Revised: and Goodfellow et al. (2015), many previous works have focused on utilizing small perturbation δ under certain constraint, e.g. in a (cid:96) p norm ball, to attack a neural network.

Original: Adversarial training (Madry et al., 2018) and its variants (Kannan et al., 2018; Zhang & Wang, 2019; Zhai et al., 2019) are the most successful defense methods to date, in which the network is forced to solve a mini-max game between the defender and attacker with adversarial examples as data augmentation.
Revised: Adversarial training (Madry et al., 2018) and its variants (Kannan et al., 2018; Zhang & Wang, 2019; Zhai et al., 2019) are the most successful defense methods to date, in which the network is forced to solve a mini-max game between the defender and attacker with adversarial examples as data augmentation.

Original: However, these empirical defense methods are still easy to be broken and cannot provide provable defense.
Revised: However, these empirical defense methods are still easy to be broken and cannot provide provable defense.

Original: Certiﬁed Defenses
Revised: Certiﬁed Defenses

Original: Unlike the empirical defense methods, once a classiﬁer can guarantee a constant classiﬁcation within a local region, it is called a robust certiﬁcate.
Revised: Unlike the empirical defense methods, once a classiﬁer can guarantee a constant classiﬁcation within a local region, it is called a robust certiﬁcate.

Original: Exact certiﬁcation methods provide the minimal perturbation condition which leads to a different classiﬁcation result.
Revised: Exact certiﬁcation methods provide the minimal perturbation condition which leads to a different classiﬁcation result.

Original: This line of work focus on deep neural networks with ReLU-like activation that makes the classiﬁer a piece-wise linear function.
Revised: This line of work focus on deep neural networks with ReLU-like activation that makes the classiﬁer a piece-wise linear function.

Original: This enables researchers to introduce satisﬁability modulo theories (Carlini et al., 2017; Ehlers, 2017) or mix integer linear programming (Cheng et al., 2017; Dutta et al., 2018).
Revised: This enables researchers to introduce satisﬁability modulo theories (Carlini et al., 2017; Ehlers, 2017) or mix integer linear programming (Cheng et al., 2017; Dutta et al., 2018).

Original: Sufﬁcient certiﬁcation methods take a conservative way and try to bound the Lipschitz constant or other information of the network (Jordan et al., 2019; Wong & Kolter, 2017; Raghunathan et al., 2018; Zhang et al., 2018).
Revised: Sufﬁcient certiﬁcation methods take a conservative way and try to bound the Lipschitz constant or other information of the network (Jordan et al., 2019; Wong & Kolter, 2017; Raghunathan et al., 2018; Zhang et al., 2018).

Original: However, these certiﬁcation strategies share a drawback that they are not feasible on large-scale scenarios, e.g. large enough practical networks, large enough datasets.
Revised: However, these certiﬁcation strategies share a drawback that they are not feasible on large-scale scenarios, e.g. large enough practical networks, large enough datasets.

Original: Randomized Smoothing To mitigate this limitation of previous certiﬁable defenses, improving network robustness via randomness has been recently discussed (Xie et al., 2018;
Revised: Randomized Smoothing To mitigate this limitation of previous certiﬁable defenses, improving network robustness via randomness has been recently discussed (Xie et al., 2018;

Original: Liu et al., 2018).
Revised: Liu et al., 2018).

Original: In certiﬁcation community, Lecuyer et al.
Revised: In certiﬁcation community, Lecuyer et al.

Original: ﬁrst introduced randomization with technique in differential privacy.
Revised: ﬁrst introduced randomization with technique in differential privacy.

Original: Li et al.
Revised: Li et al.

Original: (2019) improved their work with a bound given by R ´ enyi divergence.
Revised: (2019) improved their work with a bound given by R ´ enyi divergence.

Original: In succession, Cohen et al.
Revised: In succession, Cohen et al.

Original: (2019) ﬁrstly provided a tight bound for arbitrary Gaussian smoothed classiﬁers based on previous theorems found by Li & Kuelbs (1998).
Revised: (2019) ﬁrstly provided a tight bound for arbitrary Gaussian smoothed classiﬁers based on previous theorems found by Li & Kuelbs (1998).

Original: Salman et al.
Revised: Salman et al.

Original: combined the empirical and certiﬁcation robustness, by applying adversarial training on randomized smoothed classiﬁers to achieve a higher certiﬁed accuracy.
Revised: combined the empirical and certiﬁcation robustness, by applying adversarial training on randomized smoothed classiﬁers to achieve a higher certiﬁed accuracy.

Original: Lee et al.
Revised: Lee et al.

Original: (2019) focused on (cid:96) 0 norm perturbation setting, and proposed a discrete smoothing distribution to beat the Gaussian distribution baseline.
Revised: (2019) focused on (cid:96) 0 norm perturbation setting, and proposed a discrete smoothing distribution to beat the Gaussian distribution baseline.

Original: Similar to (Lee et al., 2019), we also focus on ﬁnding a suitable distribution to trade-off accuracy and robustness for different types of adversarial attacks, such as (cid:96) 2 and (cid:96) ∞ .
Revised: Similar to (Lee et al., 2019), we also focus on ﬁnding a suitable distribution to trade-off accuracy and robustness for different types of adversarial attacks, such as (cid:96) 2 and (cid:96) ∞ .

Original:  B LACK - BOX C ERTIFICATION WITH F UNCTIONAL O PTIMIZATION
Revised:  B LACK - BOX C ERTIFICATION WITH VARIATIONAL O PTIMIZATION

Original: We start with introducing the background of adversarial certiﬁcation problem and and randomized smoothing method.
Revised: We start with introducing the background of adversarial certiﬁcation problem and and randomized smoothing method.

Original: We then introduce in Section 3.1 our general framework of adversarial certiﬁcation using non-Gaussian smoothing noises, from a new functional optimization perspective.
Revised: We then introduce in Section 3.1 our general framework of adversarial certiﬁcation using non-Gaussian smoothing noises, from a new variational optimization perspective.

Original: Our framework includes the method of Cohen et al.
Revised: Our framework includes the method of Cohen et al.

Original: as a special case, and reveals a critical trade-off between accuracy and robustness that provides important guidance for better choices of smoothing distributions in Section 4.
Revised: as a special case, and reveals a critical trade-off between accuracy and robustness that provides important guidance for better choices of smoothing distributions in Section 4.

Original: Adversarial Certiﬁcation We consider binary classiﬁcation of predicting binary labels y ∈ { 0 , 1 } from feature vectors x ∈ R d for simplicity.
Revised: Adversarial Certiﬁcation We consider binary classiﬁcation of predicting binary labels y ∈ { 0 , 1 } from feature vectors x ∈ R d for simplicity.

Original: The extension to multi-class cases is straightforward, and is discussed in Appendix D. Assume f (cid:93) :
Revised: The extension to multi-class cases is straightforward, and is discussed in Appendix D. Assume f (cid:93) :

Original: R d → [0 , 1] is a pre-trained binary classiﬁer ( (cid:93) means the classiﬁer is given ), which maps from the feature space R d to either the class probability in interval [0 , 1] or the binary labels in { 0 , 1 } .
Revised: R d → [0 , 1] is a pre-trained binary classiﬁer ( (cid:93) means the classiﬁer is given ), which maps from the feature space R d to either the class probability in interval [0 , 1] or the binary labels in { 0 , 1 } .

Original: In the robustness certiﬁcation problem, a testing data point x 0 ∈ R d is given, and one is asked to verify if the classiﬁer outputs the same prediction when we perturb the input x 0 arbitrarily in B , a given neighborhood of x 0 .
Revised: In the robustness certiﬁcation problem, a testing data point x 0 ∈ R d is given, and one is asked to verify if the classiﬁer outputs the same prediction when we perturb the input x 0 arbitrarily in B , a given neighborhood of x 0 .

Original: Speciﬁcally, let B be a set of possible perturbation vectors,
Revised: Speciﬁcally, let B be a set of possible perturbation vectors,

Original: e.g., B = { δ ∈ R
Revised: e.g., B = { δ ∈ R

Original: d : (cid:107) δ (cid:107) p ≤ r } for (cid:96) p norm with a radius r .
Revised: d : (cid:107) δ (cid:107) p ≤ r } for (cid:96) p norm with a radius r .

Original: If the classiﬁer predicts y = 1 on x 0 , i.e. f (cid:93) ( x 0 ) > 1 / 2 , we want to verify if f (cid:93) ( x 0 + δ ) > 1 /holds for any δ ∈ B .
Revised: If the classiﬁer predicts y = 1 on x 0 , i.e. f (cid:93) ( x 0 ) > 1 / 2 , we want to verify if f (cid:93) ( x 0 + δ ) > 1 /holds for any δ ∈ B .

Original: In this paper, we consider two types of attacks, including the (cid:96) 2 attack
Revised: In this paper, we consider two types of attacks, including the (cid:96) 2 attack

Original: B (cid:96) 2 ,r def = { δ : (cid:107) δ (cid:107) 2 ≤ r } , and the (cid:96) ∞ attack B (cid:96) ∞ ,r def = { δ : (cid:107) δ (cid:107) ∞
Revised: B (cid:96) 2 ,r def = { δ : (cid:107) δ (cid:107) 2 ≤ r } , and the (cid:96) ∞ attack B (cid:96) ∞ ,r def = { δ : (cid:107) δ (cid:107) ∞

Original: ≤ r } .
Revised: ≤ r } .

Original: More general (cid:96) p attack can also be handled by our framework but is left as future works.
Revised: More general (cid:96) p attack can also be handled by our framework but is left as future works.

Original: Black-box Certiﬁcation with Randomness Directly verifying f (cid:93) heavily relies on the smoothness of f (cid:93) , which has been explored in a series of recent works (Lecuyer et al., 2018;
Revised: Black-box Certiﬁcation with Randomness Directly verifying f (cid:93) heavily relies on the smoothness of f (cid:93) , which has been explored in a series of recent works (Lecuyer et al., 2018;

Original: Wong &
Revised: Wong &

Original: Kolter, 2017).
Revised: Kolter, 2017).

Original: These methods typically depend on the special structure property (e.g., the use of
Revised: These methods typically depend on the special structure property (e.g., the use of

Original: ReLU units) of f (cid:93) , and thus can not serve as general purpose algorithms.
Revised: ReLU units) of f (cid:93) , and thus can not serve as general purpose algorithms.

Original: We are instead interested in black-box veriﬁcation methods that could work for arbitrary classiﬁers.
Revised: We are instead interested in black-box veriﬁcation methods that could work for arbitrary classiﬁers.

Original: One approach to enable this, as explored in recent works (Cohen et al., 2019; Lee et al., 2019), is to replace f (cid:93) with a smoothed classiﬁer by convovling with Gaussian noise, and verify the smoothed classiﬁer.
Revised: One approach to enable this, as explored in recent works (Cohen et al., 2019; Lee et al., 2019), is to replace f (cid:93) with a smoothed classiﬁer by convovling with Gaussian noise, and verify the smoothed classiﬁer.

Original: Speciﬁcally, assume π 0 is a smoothing distribution with zero mean and bounded variance, e.g., π 0 =
Revised: Speciﬁcally, assume π 0 is a smoothing distribution with zero mean and bounded variance, e.g., π 0 =

Original: N ( 0 , σ 2 ) .
Revised: N ( 0 , σ 2 ) .

Original: The randomized smoothed classiﬁer is deﬁned by
Revised: The randomized smoothed classiﬁer is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: which returns the averaged probability of x 0 + z under the perturbation of z ∼ π 0 .
Revised: which returns the averaged probability of x 0 + z under the perturbation of z ∼ π 0 .

Original: Assume we replace the original classiﬁer with f (cid:93)π 0 , then the goal becomes verifying f (cid:93)π 0 using its inherent smoothness.
Revised: Assume we replace the original classiﬁer with f (cid:93)π 0 , then the goal becomes verifying f (cid:93)π 0 using its inherent smoothness.

Original: Speciﬁcally, if f (cid:93)π 0 ( x 0 ) > 1 / 2 , we want to verify that f (cid:93)π 0 ( x 0 + δ ) > 1 / 2 for every δ ∈ B , that is,
Revised: Speciﬁcally, if f (cid:93)π 0 ( x 0 ) > 1 / 2 , we want to verify that f (cid:93)π 0 ( x 0 + δ ) > 1 / 2 for every δ ∈ B , that is,

Original: [Equation]
Revised: [Equation]

Original: In this case, it is sufﬁcient to obtain a guaranteed lower bound of min δ ∈B f
Revised: In this case, it is sufﬁcient to obtain a guaranteed lower bound of min δ ∈B f

Original: (cid:93)π 0 ( x 0 + δ ) and check if it is larger than 1 / 2 .
Revised: (cid:93)π 0 ( x 0 + δ ) and check if it is larger than 1 / 2 .

Original: When π 0 is Gaussian N ( 0 , σ 2 ) and for (cid:96) 2 attack, this problem was studied in
Revised: When π 0 is Gaussian N ( 0 , σ 2 ) and for (cid:96) 2 attack, this problem was studied in

Original: Cohen et al.
Revised: Cohen et al.

Original: (2019), which shows that a lower bound of
Revised: (2019), which shows that a lower bound of

Original: [Equation]
Revised: [Equation]

Original: where Φ( · ) is the cumulative density function (CDF) of standard Gaussian distribution, and Φ − 1 ( · ) represents its inverse function.
Revised: where Φ( · ) is the cumulative density function (CDF) of standard Gaussian distribution, and Φ − 1 ( · ) represents its inverse function.

Original: The proof of this result in Cohen et al.
Revised: The proof of this result in Cohen et al.

Original: (2019) uses Neyman-Pearson lemma (Li & Kuelbs, 1998), while in the following section another derivation using functional optimization is provided.
Revised: (2019) uses NeymanPearson lemma (Li & Kuelbs, 1998), while in the following section another derivation using variational optimization is provided.

Original: Note that the bound in Equation (2) is tractable since it only requires to evaluate the smoothed classiﬁer f (cid:93)π 0 ( x 0 ) at the original image x 0 , instead of solving the difﬁcult adversarial optimization over perturbation z in Equation (1).
Revised: Note that the bound in Equation (2) is tractable since it only requires to evaluate the smoothed classiﬁer f (cid:93)π 0 ( x 0 ) at the original image x 0 , instead of solving the difﬁcult adversarial optimization over perturbation z in Equation (1).

Original: In practice, f (cid:93)π 0 ( x 0 ) is approximated by Monte Carlo approximation with a non-asymptotic conﬁdence bound.
Revised: In practice, f (cid:93)π 0 ( x 0 ) is approximated by Monte Carlo approximation with a non-asymptotic conﬁdence bound.

Original: 3.  C ERTIFICATIONVIA F UNCTIONAL O PTIMIZATION
Revised: 3. V ARIATIONAL A DVERSARIAL C ERTIFICATION

Original: We propose a general framework for obtaining a guaranteed lower bound for Equation (1) based on functional optimization.
Revised: We propose a variational adversarial certiﬁcation (AVC) framework, which yields a guaranteed lower bound for Equation (1) based on variational optimization.

Original: The main idea is simple: assume F is a function class which is known to include f (cid:93) , then the following optimization immediately yields a guaranteed lower bound,
Revised: The main idea is simple: assume F is a function class which is known to include f (cid:93) , then the following optimization immediately yields a guaranteed lower bound,

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne f π 0 ( x 0 )
Revised: where we deﬁne f π 0 ( x 0 )

Original: =
Revised: =

Original: E z ∼ π 

Revised: E z ∼ π

Original: [ f ( x 0 + z )] for any f , and search for the minimum value of f π 0 ( x 0 + δ ) for all classiﬁers in F and satisﬁes f π 0 ( x 0 ) =
Revised: [ f ( x 0 + z )] for any f , and search for the minimum value of f π 0 ( x 0 + δ ) for all classiﬁers in F and satisﬁes f π 0 ( x 0 ) =

Original: f (cid:93) π 0 ( x 0 ) .
Revised: f (cid:93)π 0 ( x 0 ) .

Original: This obviously yields a lower bound once f (cid:93) ∈ F .
Revised: This obviously yields a lower bound once f (cid:93) ∈ F .

Original: If F includes only f (cid:93) , then the bound is exact, but is computationally prohibitive due to the difﬁculty of optimizing δ .
Revised: If F includes only f (cid:93) , then the bound is exact, but is computationally prohibitive due to the difﬁculty of optimizing δ .

Original: The idea is then to choose F properly to incorporate rich information of f (cid:93) , while allowing us to calculate the lower bound in Equation (3) computationally tractably.
Revised: The idea is then to choose F properly to incorporate rich information of f (cid:93) , while allowing us to calculate the lower bound in Equation (3) computationally tractably.

Original: In this paper, we consider the set of all functions bounded in [0 , 1] , that is,
Revised: In this paper, we consider the set of all functions bounded in [0 , 1] , that is,

Original: [Equation]
Revised: [Equation]

Original: which guarantees to include f (cid:93) by deﬁnition.
Revised: which guarantees to include f (cid:93) by deﬁnition.

Original: There are other F that also yields computationally tractable bounds, including the L p space F = { f : (cid:107) f (cid:107) L p ≤ v } , which we leave for future work.
Revised: There are other F that also yields computationally tractable bounds, including the L p space F = { f : (cid:107) f (cid:107) L p ≤ v } , which we leave for future work.

Original: Denote by V π 0 ( F , B ) the lower bound in Equation (3).
Revised: Denote by V π 0 ( F , B ) the lower bound in Equation (3).

Original: We can rewrite it into the following minimax form using the Lagrangian function,
Revised: We can rewrite it into the following minimax form using the Lagrangian function,

Original: [Equation]
Revised: [Equation]

Original: where λ is the Lagrangian multiplier.
Revised: where λ is the Lagrangian multiplier.

Original: Exchanging the min and max yields the following dual form.
Revised: Exchanging the min and max yields the following dual form.

Original: Theorem 1.
Revised: Theorem 1.

Original: I)
Revised: I)

Original: ( Dual Form )
Revised: ( Dual Form )

Original: Denote by π δ the distribution of z
Revised: Denote by π δ the distribution of z

Original: + δ when z ∼ π 0 .
Revised: + δ when z ∼ π 0 .

Original: Assume F and
Revised: Assume F and

Original: B are compact set.
Revised: B are compact set.

Original: We have the following dual form of V π 0 ( F , B ) via strong duality:
Revised: We have the following dual form of V π 0 ( F , B ) via strong duality:

Original: [Equation]
Revised: [Equation]

Original: 5) where we deﬁne
Revised: 5) where we deﬁne

Original: [Equation]
Revised: [Equation]

Original: which measures the difference of λπ 0 and π δ by seeking the maximum discrepancy of the expectation for f ∈ F .
Revised: which measures the difference of λπ 0 and π δ by seeking the maximum discrepancy of the expectation for f ∈ F .

Original: As we show later, the bound in (5) is computationally tractable with proper ( F , B , π 0 ) .
Revised: As we show later, the bound in (5) is computationally tractable with proper ( F , B , π 0 ) .

Original: II) When F = F [0 , 1] :=
Revised: II) When F = F [0 , 1] :=

Original: { f : f ( x ) ∈ [0 , 1] , x ∈ R d } , we have in particular
Revised: { f : f ( x ) ∈ [0 , 1] , x ∈ R d } , we have in particular

Original: [Equation]
Revised: [Equation]

Original: where [Equation]
Revised: where ( t ) + = max(0 , t ) .

Original: In addition, we have 0 ≤ D F [0 , 1] ( λπ 0 (cid:107) π δ ) ≤ λ for any π 0 , π δ and λ > 0 .
Revised: In addition, we have 0 ≤ D F [0 , 1] ( λπ 0 (cid:107) π δ ) ≤ λ for any π 0 , π δ and λ > 0 .

Original: Note that D
Revised: Note that D

Original: F [0 , 1] ( λπ 0 (cid:107) π δ ) coincides with the total variation distance between π 0 and π δ when λ
Revised: F [0 , 1] ( λπ 0 (cid:107) π δ ) coincides with the total variation distance between π 0 and π δ when λ

Original: = 1 .
Revised: = 1 .

Revised: III)

Revised: Suppose

Revised: F

Revised: =

Revised: F

Original: Proof.
Revised: 0 , 1] and suppose that for any λ ≥ 0 , min δ ∈B min f ∈F [0 , 1] L ( f, δ , λ = min f ∈F [

Revised: ] L ( f, δ ∗ , λ ) , for some δ ∗ ∈ B

Revised: , we have

Revised: [Equation]

Revised: Remark We will show later that the proposed methods and the cases we study satisfy the condition in part III of the theorem and thus all the lower bounds of the proposed method are tight.

Revised: Proof.

Original: First, observe that the constraint in Equation (3) can be equivalently replaced by an inequality constraint f π 0 ( x 0 ) ≥ f
Revised: First, observe that the constraint in Equation (3) can be equivalently replaced by an inequality constraint f π 0 ( x 0 ) ≥ f

Original: (cid:93)π 0 ( x 0 ) .
Revised: (cid:93)π 0 ( x 0 ) .

Original: Therefore, the Lagrangian multiplier can be restricted to be λ ≥ 0 .
Revised: Therefore, the Lagrangian multiplier can be restricted to be λ ≥ 0 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: The proof of the strong duality is in Appendix A.1.
Revised: The proof of the strong duality is in Appendix A.1.

Original: II) follows a straightforward calculation.
Revised: II) follows a straightforward calculation

Original:  Although the lower bound in Equation (5) still involves an optimization on δ and λ , both of them are much easier than the original adversarial optimization in Equation (1).
Revised: [Equation] Although the lower bound in Equation (5) still involves an optimization on δ and λ , both of them are much easier than the original adversarial optimization in Equation (1).

Original: With proper choices of F , B and π 0 , the optimization of δ can be shown to provide simple closed form solutions by exploiting the symmetry of B , and the optimization of λ is a very simple one-dimensional searching problem.
Revised: With proper choices of F , B and π 0 , the optimization of δ can be shown to provide simple closed form solutions by exploiting the symmetry of B , and the optimization of λ is a very simple one-dimensional searching problem.

Original: As a corollary of Theorem 1, we can exactly recover the bound derived by Cohen et al.
Revised: As a corollary of Theorem 1, we can exactly recover the bound derived by Cohen et al.

Original: Corollary 1.
Revised: Corollary 1.

Original: With isotropic Gaussian noise π 0 = N ( 0 , σ 2 I d × d ) , (cid:96) 2 attack B = { δ : (cid:107) δ (cid:107) 2 ≤ r } and F = F [0 , 1] , the lower bound in Equation (5) equals the bound in Equation (2) by Cohen et al.
Revised: With isotropic Gaussian noise π 0 = N ( 0 , σ 2 I d × d ) , (cid:96) 2 attack B = { δ : (cid:107) δ (cid:107) 2 ≤ r } and F = F [0 , 1] , the lower bound in Equation (5) equals the bound in Equation (2) by Cohen et al.

Original: (2019), that is,
Revised: (2019), that is,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B See Appendix A.2 for more details.
Revised: See Appendix A.2 for more details.

Original: A key step of the proof is to show that max (cid:107) δ (cid:107) 2 ≤ r D F [0 , 1] ( λπ 0 (cid:107) π δ ) is achieved when δ is on the boundary of the (cid:96) 2 ball B = {(cid:107) δ (cid:107) 2 ≤ r } , which can be, for example, δ =
Revised: A key step of the proof is to show that max (cid:107) δ (cid:107) 2 ≤ r D F [ [Equation] 1] ( λπ 0 (cid:107) π δ ) is achieved when δ is on the boundary of the (cid:96) 2 ball B = {(cid:107) δ (cid:107) 2 ≤ r } , which can be, for example, δ =

Original: [ r, 0 , . . . , 0] (cid:62) , due to symmetry of B (see ﬁgure on the right).
Revised: [ r, 0 , . . . , 0] (cid:62) , due to symmetry of B (see ﬁgure on the right).

Original: Trade-off between Accuracy and Robustness
Revised: Trade-off between Accuracy and Robustness

Original: The lower bound in
Revised: The lower bound in

Original: Equation (5) reﬂects an intuitive trade-off between the robustness and accuracy,
Revised: Equation (5) reﬂects an intuitive trade-off between the robustness and accuracy,

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term reﬂects the accuracy of the smoothed classiﬁer (assuming the true label is y = 1 ), while the second term max δ ∈B D F ( λπ 0 (cid:107) π δ ) measures the robustness of the smoothed classiﬁer, via the maximum difference between the original smoothing distribution π 0 and perturbed distribution π δ for δ ∈ B .
Revised: where the ﬁrst term reﬂects the accuracy of the smoothed classiﬁer (assuming the true label is y = 1 ), while the second term max δ ∈B D F ( λπ 0 (cid:107) π δ ) measures the robustness of the smoothed classiﬁer, via the maximum difference between the original smoothing distribution π 0 and perturbed distribution π δ for δ ∈ B .

Original: The scalar λ can be viewed as looking for a best balance between these two terms to achieve the largest lower bound.
Revised: The scalar λ can be viewed as looking for a best balance between these two terms to achieve the largest lower bound.

Original: More critically, different choices of smoothing distributions also yields a fundamental trade-off between accuracy and robustness in Equation (7).
Revised: More critically, different choices of smoothing distributions also yields a fundamental trade-off between accuracy and robustness in Equation (7).

Original: If π 0 has large variance or high tail probability, the distance D F ( λπ 0 (cid:107) π δ ) will tend to be large and the model is robust.
Revised: If π 0 has large variance or high tail probability, the distance D F ( λπ 0 (cid:107) π δ ) will tend to be large and the model is robust.

Original: However, if the variance of π 0 is too large, we may obtain a low value of f (cid:93)π 0 ( x 0 ) and hence less accurate model.
Revised: However, if the variance of π 0 is too large, we may obtain a low value of f (cid:93)π 0 ( x 0 ) and hence less accurate model.

Original: The optimal choice of the smoothing distribution should optimally balance the accuracy and robustness, by distribute its mass properly to yield small max δ ∈B D F ( λπ 0 (cid:107) π δ ) and large f (cid:93)π 0 ( x 0 ) simultaneously.
Revised: The optimal choice of the smoothing distribution should optimally balance the accuracy and robustness, by distribute its mass properly to yield small max δ ∈B D F ( λπ 0 (cid:107) π δ ) and large f (cid:93)π 0 ( x 0 ) simultaneously.

Original:  F ILLING THE S OAP B UBBLES : N EW F AMILIES OF N ON -G AUSSIAN S MOOTHING D ISTRIBUTIONS
Revised:  F ILLING THE S OAP B UBBLES : N EW F AMILIES OF N ON -G AUSSIAN S MOOTHING D ISTRIBUTIONS

Original: In this section, we identify a key problem of using Gaussian smoothing noise in high dimensional space, due to the “thin shell” phenomenon that the probability mass of Gaussian distributions concentrates on a sphere far away from the center points in high dimensional spaces.
Revised: In this section, we identify a key problem of using Gaussian smoothing noise in high dimensional space, due to the “thin shell” phenomenon that the probability mass of Gaussian distributions concentrates on a sphere far away from the center points in high dimensional spaces.

Original: Motivated by this observation, we propose in Section 4.1 a new family of non-Gaussian smoothing distributions that alleviate this problem for (cid:96) 2 attack, and also in Section 4.2 another mixed norm smoothing distribution designed speciﬁcally for (cid:96) ∞ attack.
Revised: Motivated by this observation, we propose in Section 4.1 a new family of non-Gaussian smoothing distributions that alleviate this problem for (cid:96) 2 attack, and also in Section 4.2 another mixed norm smoothing distribution designed speciﬁcally for (cid:96) ∞ attack.

Original: 4. (cid:96) 2 R EGION C ERTIFICATION
Revised: 4. (cid:96) 2 R EGION C ERTIFICATION

Original: Although isotropic Gaussian distributions appears to be a natural choice of smoothing distributions, they are in fact sub-optimal for trading-off accuracy and robustness in Equation (7), especially in high dimensions.
Revised: Although isotropic Gaussian distributions appears to be a natural choice of smoothing distributions, they are in fact sub-optimal for trading-off accuracy and robustness in Equation (7), especially in high dimensions.

Original: The key problem is that, in high dimensional spaces, the probability mass of
Revised: The key problem is that, in high dimensional spaces, the probability mass of

Original: Gaussian distributions concentrates on a thin shell away from the center, and hence looks like “soap bubbles”, instead of “solid balls” as what it appears in low dimension spaces.
Revised: Gaussian distributions concentrates on a thin shell away from the center, and hence looks like “soap bubbles”, instead of “solid balls” as what it appears in low dimension spaces.

Original: Lemma 1 (Vershynin (2018), Section 3.1) .
Revised: Lemma 1 (Vershynin (2018), Section 3.1) .

Original: Let z ∼ N ( 0 , I d × d ) be a d -dimensional standard
Revised: Let z ∼ N ( 0 , I d × d ) be a d -dimensional standard

Original: Gaussian random variable.
Revised: Gaussian random variable.

Original: Then there exists a constant c , such that for δ ∈ (0 , 1) ,
Revised: Then there exists a constant c , such that for δ ∈ (0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: This suggests that with high probability (e.g, 1 − δ = 0 . 99 ), z takes values very close to the sphere of radius √ d , within a constant distance from that sphere!
Revised: This suggests that with high probability (e.g, 1 − δ = 0 . 99 ), z takes values very close to the sphere of radius √ d , within a constant distance from that sphere!

Original: See Vershynin (2018) for more discussion.
Revised: See Vershynin (2018) for more discussion.

Original: This phenomenon makes it problematic to use standard Gaussian distribution for adversarial certiﬁcation, because one would expect that the smoothing distribution should concentrate around the center (the original image) in order to make the smoothed classiﬁer close to the original classiﬁer (and hence accurate).
Revised: This phenomenon makes it problematic to use standard Gaussian distribution for adversarial certiﬁcation, because one would expect that the smoothing distribution should concentrate around the center (the original image) in order to make the smoothed classiﬁer close to the original classiﬁer (and hence accurate).

Original: To illustrate the problem, consider a simple example when the true classiﬁer is f (cid:93) ( x )
Revised: To illustrate the problem, consider a simple example when the true classiﬁer is f (cid:93) ( x )

Original: =
Revised: =

Original: I ( (cid:107) x − x 0 (cid:107) 2 ≤ (cid:15) √ d ) for a constant (cid:15) < 1 , where I is the indicator function.
Revised: I ( (cid:107) x − x 0 (cid:107) 2 ≤ (cid:15) √ d ) for a constant (cid:15) < 1 , where I is the indicator function.

Original: Then when the dimension d is large, we would have f (cid:93) ( x 0 )
Revised: Then when the dimension d is large, we would have f (cid:93) ( x 0 )

Original: = 1 while f (cid:93)π 0 ( x 0 )
Revised: = 1 while f (cid:93)π 0 ( x 0 )

Original: ≈ 

Revised: ≈

Original: when π 0 = N ( 0 , I d × d ) .
Revised: when π 0 = N ( 0 , I d × d ) .

Original: It is of course possible to decrease the variance of π 0 to improve the accuracy of the smoothed classiﬁer f
Revised: It is of course possible to decrease the variance of π 0 to improve the accuracy of the smoothed classiﬁer f

Original: (cid:93)π 0 .
Revised: (cid:93)π 0 .

Original: However, this would signiﬁcantly improve the distance term in Equation (7) and does not yield an optimal trade-off on accuracy and robustness.
Revised: However, this would signiﬁcantly improve the distance term in Equation (7) and does not yield an optimal trade-off on accuracy and robustness.

Original: In this work, we introduce a new family of non-Gaussian distributions to address this curse of dimensionality.
Revised: In this work, we introduce a new family of non-Gaussian distributions to address this curse of dimensionality.

Original: To motivate our method, it is useful to examine the density function of the distributions of the radius of spherical distributions in general.
Revised: To motivate our method, it is useful to examine the density function of the distributions of the radius of spherical distributions in general.

Original: Lemma 2.
Revised: Lemma 2.

Original: Assume z is a spherically symmetric random variable on R d with a probability density function (PDF) of form π 0 ( z ) ∝ φ ( (cid:107) z (cid:107) 2 ) , where φ : [0 , ∞ )
Revised: Assume z is a spherically symmetric random variable on R d with a probability density function (PDF) of form π 0 ( z ) ∝ φ ( (cid:107) z (cid:107) 2 ) , where φ : [0 , ∞ )

Original: → [0 , ∞ ) is a univariate function, then the PDF of the norm of z is p (cid:107) z (cid:107) 2 ( r ) ∝ r d − 1 φ ( r ) .
Revised: → [0 , ∞ ) is a univariate function, then the PDF of the norm of z is p (cid:107) z (cid:107) 2 ( r ) ∝ r d − 1 φ ( r ) .

Original: The term r d − 1 arises due to the integration on the sphere of radius r in R d .
Revised: The term r d − 1 arises due to the integration on the sphere of radius r in R d .

Original: In particular, when z ∼ π 0 = N (0 , σ 2 I d × d ) , we have φ ( r ) =
Revised: In particular, when z ∼ π 0 = N (0 , σ 2 I d × d ) , we have φ ( r ) =

Original: exp( − r 2 / (2 σ 2 )) and hence p (cid:107) z (cid:107) 2 ( r ) ∝ r d − 1 exp( − r 2 / (2 σ 2 )) , which is a scaled Chi distribution, also known as Nakagami distribution.
Revised: exp( − r 2 / (2 σ 2 )) and hence p (cid:107) z (cid:107) 2 ( r ) ∝ r d − 1 exp( − r 2 / (2 σ 2 )) , which is a scaled Chi distribution, also known as Nakagami distribution.

Original: Examining this P.D.F., we can see that the concentration of the norm is caused by the r d − 1 term, which makes the density to be highly peaked when d is large.
Revised: Examining this P.D.F., we can see that the concentration of the norm is caused by the r d − 1 term, which makes the density to be highly peaked when d is large.

Original: To alleviate the concentration phenomenon, we need to have a way to cancel out the effect of r d − 1 .
Revised: To alleviate the concentration phenomenon, we need to have a way to cancel out the effect of r d − 1 .

Original: This motivates the following family of smoothing distributions:
Revised: This motivates the following family of smoothing distributions:

Original: [Equation]
Revised: [Equation]

Original: , and hence
Revised: and hence

Original: [Equation]
Revised: [Equation]

Original: where we introduce a (cid:107) z (cid:107) − k 2 term in π 0 , with k a positive parameter, to make the radius distribution less concentrated when k is large.
Revised: where we introduce a (cid:107) z (cid:107) − k 2 term in π 0 , with k a positive parameter, to make the radius distribution less concentrated when k is large.

Original: [Figure]
Revised: [Figure]

Original: The radius distribution in Equation (8) is controlled by two parameters ( σ, k ) , where σ controls the scale of the distribution (and is hence the scale parameter ), while k controls the shape of the distribution (and hence the shape parameter ).
Revised: The radius distribution in Equation (8) is controlled by two parameters ( σ, k ) , where σ controls the scale of the distribution (and is hence the scale parameter ), while k controls the shape of the distribution (and hence the shape parameter ).

Original: The key idea is that adjusting k allows us to trade-off the accuracy and robustness much more optimally.
Revised: The key idea is that adjusting k allows us to trade-off the accuracy and robustness much more optimally.

Original: As shown in Figure 1, adjusting σ enables us to move the mean close to zero (hence yielding higher accuracy), but at cost of decreasing the variance quadratically (hence more less robust).
Revised: As shown in Figure 1, adjusting σ enables us to move the mean close to zero (hence yielding higher accuracy), but at cost of decreasing the variance quadratically (hence more less robust).

Original: In contrast, adjusting k allows us to decrease the mean without signiﬁcantly impacting the variance, and hence yield a much better trade-off on the accuracy and robustness.
Revised: In contrast, adjusting k allows us to decrease the mean without signiﬁcantly impacting the variance, and hence yield a much better trade-off on the accuracy and robustness.

Original: Computational Method With the more general nonGaussian smoothing distribution, we no longer have the closed form solution of the bound like Equation (6).
Revised: Computational Method With the more general nonGaussian smoothing distribution, we no longer have the closed form solution of the bound like Equation (6).

Original: However, efﬁcient computational methods can be still developed for calculating the bound in Equation (5) with π 0 in Equation (8).
Revised: However, efﬁcient computational methods can be still developed for calculating the bound in Equation (5) with π 0 in Equation (8).

Original: The key is that the maximum of the distance term D
Revised: The key is that the maximum of the distance term D

Original: F [0 , 1] ( λπ 0 ||
Revised: F [0 , 1] ( λπ 0 ||

Original: π δ ) over δ ∈ B is always achieved on the boundary of B as we show in the sequel, while the optimization on λ ≥ 0 is one-dimensional and can be solved numerically efﬁciently.
Revised: π δ ) over δ ∈ B is always achieved on the boundary of B as we show in the sequel, while the optimization on λ ≥ 0 is one-dimensional and can be solved numerically efﬁciently.

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider the (cid:96) 2 attack with B = { δ : (cid:107) δ (cid:107) 2 ≤ r } and smoothing distribution π 0 ( z ) ∝ (cid:107) z (cid:107) − k 2 exp (cid:16) − (cid:107) z (cid:107) 

Revised: Consider the (cid:96) 2 attack with B = { δ : (cid:107) δ (cid:107) 2 ≤ r } and smoothing distribution π 0 ( z ) ∝ (cid:107) z (cid:107) − k 2 exp (cid:16) − (cid:107) z (cid:107)

Original: σ 2 (cid:17) with k ≥ 0 and σ > 0 .
Revised: σ 2 (cid:17) with k ≥ 0 and σ > 0 .

Original: Deﬁne δ ∗ =
Revised: Deﬁne δ ∗ =

Original: [ r, 0 , ..., 0] (cid:62) , we have
Revised: [ r, 0 , ..., 0] (cid:62) , we have

Original: [Equation]
Revised: [Equation]

Original: With Theorem 2, we can compute Equation (5) with δ = δ ∗ .
Revised: With Theorem 2, we can compute Equation (5) with δ = δ ∗ .

Original: We then calculate D F [0 , 1] ( λπ 0 (cid:107) π δ ∗ ) using Monte Carlo approximation.
Revised: We then calculate D F [0 , 1] ( λπ 0 (cid:107) π δ ∗ ) using Monte Carlo approximation.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: which can be approximated with Monte Carlo method with Hoeffding concentration bound.
Revised: which can be approximated with Monte Carlo method with Hoeffding concentration bound.

Original: Let { z i } ni =1 be an i.i.d.
Revised: Let { z i } ni =1 be an i.i.d.

Original: sample from π 0 , then we can approximate D F [0 , 1] ( λπ 0 (cid:107) π δ ∗ ) with ˆ D := n − 1 (cid:80) ni =1 ( λ − π δ ∗ ( z i ) /π 

Revised: sample from π 0 , then we can approximate D F [0 , 1] ( λπ 0 (cid:107) π δ ∗ ) with ˆ D := n − 1 (cid:80) ni =1 ( λ − π δ ∗ ( z i ) /π

Original: ( z i ))
Revised: ( z i ))

Original: + .
Revised: + .

Original: Because 0 ≤ ( λ − π δ ∗ ( z i ) /π 

Revised: Because 0 ≤ ( λ − π δ ∗ ( z i ) /π

Original: ( z i ))
Revised: ( z i ))

Original: + ≤ λ , we have
Revised: + ≤ λ , we have

Original: D F [0 , 1] ( λπ 0 (cid:107) π δ ∗ ) ∈
Revised: D F [0 , 1] ( λπ 0 (cid:107) π δ ∗ ) ∈

Original: [ ˆ D − λ (cid:112) log(2 /δ ) /
Revised: [ ˆ D − λ (cid:112) log(2 /δ ) /

Original: (2 n ) , ˆ D + λ (cid:112) log(2 /δ ) /
Revised: (2 n ) , ˆ D + λ (cid:112) log(2 /δ ) /

Original: (2 n )] with probability 1 − δ for δ ∈ (0 , 1) .
Revised: (2 n )] with probability 1 − δ for δ ∈ (0 , 1) .

Original: Drawing sufﬁciently large number of samples allows us to achieve approximation with arbitrary accuracy.
Revised: Drawing sufﬁciently large number of samples allows us to achieve approximation with arbitrary accuracy.

Original: 4. (cid:96) ∞ R EGION C ERTIFICATION
Revised: 4. (cid:96) ∞ R EGION C ERTIFICATION

Original: Going beyond the (cid:96) 2 attack, we consider the (cid:96) ∞ attack, whose attacking region is B (cid:96) ∞ ,r = { δ : (cid:107) δ (cid:107) ∞
Revised: Going beyond the (cid:96) 2 attack, we consider the (cid:96) ∞ attack, whose attacking region is B (cid:96) ∞ ,r = { δ : (cid:107) δ (cid:107) ∞

Original: ≤
Revised: ≤

Original: r } .
Revised: r } .

Original: This is a far more difﬁcult problem, because the (cid:96) ∞ ball is substantially larger than the (cid:96) 2 ball of the same radius in high dimensional space.
Revised: This is a far more difﬁcult problem, because the (cid:96) ∞ ball is substantially larger than the (cid:96) 2 ball of the same radius in high dimensional space.

Original: This makes the Gaussian smoothing distribution, as well as our (cid:96) 2 -based smoothing distribution in Equation (8), unsuitable for (cid:96) ∞ attack.
Revised: This makes the Gaussian smoothing distribution, as well as our (cid:96) 2 -based smoothing distribution in Equation (8), unsuitable for (cid:96) ∞ attack.

Original: In fact, as shown in the following negative result, if we use Equation (8) as the smoothing distribution for (cid:96) ∞ attack, the bound we obtain is effectively the bound we would get for verifying a (cid:96) 2 ball with radius √ dr , which is too large to give meaningful results when the dimension is high.
Revised: In fact, as shown in the following negative result, if we use Equation (8) as the smoothing distribution for (cid:96) ∞ attack, the bound we obtain is effectively the bound we would get for verifying a (cid:96) 2 ball with radius √ dr , which is too large to give meaningful results when the dimension is high.

Original: Theorem 3.
Revised: Theorem 3.

Original: With the smoothing distribution π 0 in Equation (8) for k ≥ 0 , σ > 0 , and F = F [0 , 1] shown in Equation (4), the bound we get for certifying the (cid:96) ∞ attack on B (cid:96) ∞ ,r = { δ : (cid:107) δ (cid:107) ∞
Revised: With the smoothing distribution π 0 in Equation (8) for k ≥ 0 , σ > 0 , and F = F [0 , 1] shown in Equation (4), the bound we get for certifying the (cid:96) ∞ attack on B (cid:96) ∞ ,r = { δ : (cid:107) δ (cid:107) ∞

Original: ≤ r } is equivalent to that for certifying the (cid:96) 2 attack on B (cid:96) 2 , √ dr = { δ : (cid:107) δ (cid:107) 2 ≤ √ dr } , that is,
Revised: ≤ r } is equivalent to that for certifying the (cid:96) 2 attack on B (cid:96) 2 , √ dr = { δ : (cid:107) δ (cid:107) 2 ≤ √ dr } , that is,

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: The key reason of this negative result is that the furthest points to the origin (vertexes) in B (cid:96) ∞ ,r have an (cid:96) 2 radius of √ dr , illustrated as the “pointy” points in Figure 2.Thus, the maximum distance max [Equation]
Revised: The key reason of this negative result is that the furthest points to the origin (vertexes) in B (cid:96) ∞ ,r have an (cid:96) 2 radius of √ dr , illustrated as the “pointy” points in Figure 2.

Original:  D [Equation] ( λπ 0 (cid:107) π δ ) is achieved at one of these pointy points, making it equivalent to optimizing in the (cid:96) 2 ball with radius √ dr.
Revised: Thus, the maximum distance max δ ∈B (cid:96) ∞ ,r D F ( λπ 0 (cid:107) π δ ) is achieved at one of these pointy points, making it equivalent to optimizing in the (cid:96) 2 ball with radius √ dr.

Original: In order to address this problem, we propose the following new mixed norm family of smoothing distribution that uses a mix of (cid:96) 2 and (cid:96) ∞ norms:
Revised: In order to address this problem, we propose the following new mixed norm family of smoothing distribution that uses a mix of (cid:96) 2 and (cid:96) ∞ norms:

Original: [Equation]
Revised: [Equation]

Original: in which we replace the (cid:107) z (cid:107) − k 2 term in Equation (8) with (cid:107) z (cid:107) − k ∞ .
Revised: in which we replace the (cid:107) z (cid:107) − k 2 term in Equation (8) with (cid:107) z (cid:107) − k ∞ .

Original: The motivation is that this allows us to allocate more probability mass along the “pointy” directions with larger (cid:96) ∞ norm, and hence decrease the maximum distance term [Equation]
Revised: The motivation is that this allows us to allocate more probability mass along the “pointy” directions with larger (cid:96) ∞ norm, and hence decrease the maximum distance term max δ ∈B (cid:96) ∞ ,r D F ( λπ 0 (cid:107) π δ ) .

Original: See Figure 2 for an illustration.
Revised: See Figure 2 for an illustration.

Original: In practice, we ﬁnd that this mixed norm smoothing distribution in Equation (9) work much more efﬁciently than the (cid:96) 2 norm-based family in Equation (8).
Revised: In practice, we ﬁnd that this mixed norm smoothing distribution in Equation (9) work much more efﬁciently than the (cid:96) 2 norm-based family in Equation (8).

Original: Given the difference of (cid:96) 2 and (cid:96) ∞ norms, it is also natural to consider the following pure (cid:96) ∞ norm distributions, which uses (cid:96) ∞ norm in both of the terms of the distribution,
Revised: Given the difference of (cid:96) 2 and (cid:96) ∞ norms, it is also natural to consider the following pure (cid:96) ∞ norm distributions, which uses (cid:96) ∞ norm in both of the terms of the distribution,

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, this seemingly natural choice does not work efﬁciently for (cid:96) ∞ attacks (even worse than the (cid:96) 2 family Equation (8)).
Revised: Unfortunately, this seemingly natural choice does not work efﬁciently for (cid:96) ∞ attacks (even worse than the (cid:96) 2 family Equation (8)).

Original: This is because the volume of the (cid:96) ∞ ball is in some sense “too large” (e.g., compared with the volume of (cid:96) 2 ball).
Revised: This is because the volume of the (cid:96) ∞ ball is in some sense “too large” (e.g., compared with the volume of (cid:96) 2 ball).

Original: As a result, in order to make the probability mass of Equation (10) in a reasonable scale, one has to choose a very small value of σ , which makes maximum distance term too large to be practically useful. 
Revised: As a result, in order to make the probability mass of Equation (10) in a reasonable scale, one has to choose a very small value of σ , which makes maximum distance term too large to be practically useful. A cc u r ac y 0. 1.0 Pure ` 2 (Eq.8) Mixed ` 1 and ` 2 (Eq.9)

Revised: Pure ` 1 (Eq.10)

Revised: [Figure]

Original: Theorem 4.
Revised: Theorem 4.

Original: Consider the adversarial attacks on the (cid:96) ∞ ball B (cid:96) ∞ ,r = { δ : (cid:107) δ (cid:107) ∞ ≤ r } .
Revised: Consider the adversarial attacks on the (cid:96) ∞ ball B (cid:96) ∞ ,r = { δ : (cid:107) δ (cid:107) ∞ ≤ r } .

Original: Suppose we use the smoothing distribution π 0 in Equation (10) and choose the parameters ( k, σ ) such that
Revised: Suppose we use the smoothing distribution π 0 in Equation (10) and choose the parameters ( k, σ ) such that

Original: 1) (cid:107) z (cid:107) ∞ is stochastic bounded when z 
Revised: 1) (cid:107) z (cid:107) ∞ is stochastic bounded when z ∼ π 0 , in that for any (cid:15) >

Original: ∼ π 0 ,

Original: in that for any
Revised: there exists

Original: (cid:15) >

Original: 0 , there exists a ﬁnite M >0 such
Revised: a ﬁnite M > such

Original: that P π 0 ( | z | > M ) ≤ (cid:15) ; 2) the mode of (cid:107) z (cid:107) ∞ under π 0 equals Cr , where
Revised: that P π 0 ( | z | > M ) ≤ (cid:15) ; 2) the mode of (cid:107) z (cid:107) ∞ under π 0 equals Cr , where

Original: C
Revised: C

Original: is some
Revised: is some

Original: ﬁxed positive constant, then for any (cid:15) ∈ (0 , 1) and sufﬁciently large dimension d , there exists a constant t > 1 , such that , we have 
Revised: ﬁxed positive constant, then for any (cid:15) ∈ (0 , 1) and sufﬁciently large dimension d , there exists a constant t > 1 , such that , we have [Figure]

Revised: [Equation]

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: This shows that, in very high dimensions, the maximum distance term is arbitrarily close to λ which is the maximum possible value of D F [0 , 1] ( λπ 0 (cid:107) π δ ) (see Theorem 1).
Revised: This shows that, in very high dimensions, the maximum distance term is arbitrarily close to λ which is the maximum possible value of D F [0 , 1] ( λπ 0 (cid:107) π δ ) (see Theorem 1).

Original: In particular, this implies that in high dimensional scenario, once f (cid:93)π 0 ( x 0 ) ≤ (1 − (cid:15) ) for some small (cid:15) , we have
Revised: In particular, this implies that in high dimensional scenario, once f (cid:93)π 0 ( x 0 ) ≤ (1 − (cid:15) ) for some small (cid:15) , we have

Original: V π 0 ( F [0 , 1] , B (cid:96) ∞ ,r ) =
Revised: V π 0 ( F [0 , 1] , B (cid:96) ∞ ,r ) =

Original: O ( t − d ) and thus fail to certify.
Revised: O ( t − d ) and thus fail to certify.

Original: A cc u r ac y 0.

Original: 1.0 Pure ` 2 (Eq.8)

Original: Mixed ` 1 and ` 2 (Eq.9)

Original: Pure ` 1 (Eq.10)

Original: [Figure]

Original: Remark
Revised: Remark

Original: The condition 1) and 2) in Theorem 4 are used to ensure that the magnitude of the random perturbations generated by π 0 is within a reasonable range such that the value of f (cid:93)π 0 ( x 0 ) is not too small, in order to have a high accuracy in the trade-off in Equation (7).
Revised: The condition 1) and 2) in Theorem 4 are used to ensure that the magnitude of the random perturbations generated by π 0 is within a reasonable range such that the value of f (cid:93)π 0 ( x 0 ) is not too small, in order to have a high accuracy in the trade-off in Equation (7).

Original: Note that the natural images are often contained in cube [0 , 1] d .
Revised: Note that the natural images are often contained in cube [0 , 1] d .

Original: If (cid:107) z (cid:107) ∞ is too large to exceed the region of natural images, the accuracy will be obviously rather poor.
Revised: If (cid:107) z (cid:107) ∞ is too large to exceed the region of natural images, the accuracy will be obviously rather poor.

Original: Note that if we use variants of Gaussian distribution, we only need ||
Revised: Note that if we use variants of Gaussian distribution, we only need ||

Original: z || 2 / √ d to be not too large.
Revised: z || 2 / √ d to be not too large.

Original: Theorem 4 says that once (cid:107) z (cid:107) ∞ is in a reasonably small scale, the maximum distance term must be unreasonably large in high dimensions, yielding a vacuous lower bound.
Revised: Theorem 4 says that once (cid:107) z (cid:107) ∞ is in a reasonably small scale, the maximum distance term must be unreasonably large in high dimensions, yielding a vacuous lower bound.

Original: Empirical Justiﬁcation We construct a simple toy example to verify the advantages of the mixed norm family Equation (9) overall the (cid:96) 2 family in Equation (8) and the (cid:96) ∞ family in Equation (10).
Revised: Empirical Justiﬁcation We construct a simple toy example to verify the advantages of the mixed norm family Equation (9) overall the (cid:96) 2 family in Equation (8) and the (cid:96) ∞ family in Equation (10).

Original: We assume that the true classiﬁer is f (cid:93) ( x )
Revised: We assume that the true classiﬁer is f (cid:93) ( x )

Original: =
Revised: =

Original: I ( (cid:107) x (cid:107) 2 ≤ r ) in r = 0 .
Revised: I ( (cid:107) x (cid:107) 2 ≤ r ) in r = 0 .

Original: 65 , d = 5 case and plot in Figure 4 the Pareto frontier of the accuracy and robustness terms in Equation (7) for the three families of smoothing distributions, as we search for the best combinations of parameters ( k, σ ) .
Revised: 65 , d = 5 case and plot in Figure 4 the Pareto frontier of the accuracy and robustness terms in Equation (7) for the three families of smoothing distributions, as we search for the best combinations of parameters ( k, σ ) .

Original: We can see that the mixed norm smoothing distribution clearly obtain the best trade-off on accuracy and robustness, and hence guarantees a tighter lower bound for certiﬁcation.
Revised: We can see that the mixed norm smoothing distribution clearly obtain the best trade-off on accuracy and robustness, and hence guarantees a tighter lower bound for certiﬁcation.

Original: Computational Method In order to compute the lower bound when using the mixed norm family Equation (9), we need to establish the closed form solution of the maximum distance term max δ ∈B D F [0 , 1] ( λπ 0 || π δ ) similar to Theorem 2.
Revised: Computational Method In order to compute the lower bound when using the mixed norm family Equation (9), we need to establish the closed form solution of the maximum distance term max δ ∈B D F [0 , 1] ( λπ 0 || π δ ) similar to Theorem 2.

Original: The following result shows that the optimal δ is achieved at one vertex (the pointy points) of the (cid:96) ∞ ball.
Revised: The following result shows that the optimal δ is achieved at one vertex (the pointy points) of the (cid:96) ∞ ball.

Original: Theorem 5.
Revised: Theorem 5.

Original: Consider the (cid:96) ∞ attack with B (cid:96) ∞ ,r = { δ : (cid:107) δ (cid:107) ∞ ≤ r } and the mixed norm smoothing distribution in Equation (9) with k ≥ 0 and σ > 0 .
Revised: Consider the (cid:96) ∞ attack with B (cid:96) ∞ ,r = { δ : (cid:107) δ (cid:107) ∞ ≤ r } and the mixed norm smoothing distribution in Equation (9) with k ≥ 0 and σ > 0 .

Original: Deﬁne δ ∗ =
Revised: Deﬁne δ ∗ =

Original: [ r, r, ..., r ] (cid:62) .
Revised: [ r, r, ..., r ] (cid:62) .

Original: We have for any  λ > 0 ,
Revised: We have for any [Table] [Table] λ > 0 ,

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: The proof of Theorem 2 and 5 is non-trivial, thus we defer the details to Appendix A.3.
Revised: The proof of Theorem 2 and 5 is non-trivial, thus we defer the details to Appendix A.3.

Original: With the optimal δ ∗ found above, we can calculate the bound with similar Monte Carlo approximation outlined in Section 4.1.
Revised: With the optimal δ ∗ found above, we can calculate the bound with similar Monte Carlo approximation outlined in Section 4.1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our new bound and smoothing distributions for both (cid:96) 2 and (cid:96) ∞ attacks.
Revised: We evaluate our new bound and smoothing distributions for both (cid:96) 2 and (cid:96) ∞ attacks.

Original: We compare with the randomized smoothing method of Cohen et al.
Revised: We compare with the randomized smoothing method of Cohen et al.

Original: with Gaussian smoothing distribution.
Revised: with Gaussian smoothing distribution.

Original: For fair comparisons, we use the same model architecture and pre-trained models provided by
Revised: For fair comparisons, we use the same model architecture and pre-trained models provided by

Original: Cohen et al.
Revised: Cohen et al.

Original: and
Revised: and

Original: Salman et al.
Revised: Salman et al.

Original: (2019), which are ResNet110 on CIFAR10 and ResNet50 on
Revised: (2019), which are ResNet110 on CIFAR10 and ResNet50 on

Original: ImageNet.
Revised: ImageNet.

Original: [Table]

Original: [Table]

Original: Settings and Hyperparameters
Revised: Settings and Hyperparameters

Original: The details of our method are shown in Algorithm 2 in Appendix.
Revised: The details of our method are shown in Algorithm 2 in Appendix.

Original: Since our method requires Monte Carlo approximation, we draw 0 .
Revised: Since our method requires Monte Carlo approximation, we draw 0 .

Original: 1 M samples from π 

Revised: 1 M samples from π

Original: and construct α = 99.9% conﬁdence lower bounds of that in Equation (7).
Revised: and construct α = 99.9% conﬁdence lower bounds of that in Equation (7).

Original: The optimization on λ is solved using grid search.
Revised: The optimization on λ is solved using grid search.

Original: For (cid:96) 2 attacks, we set k = 500 for CIFAR10 and k = 50000 for ImageNet in our non-Gaussian smoothing distribution Equation (8).
Revised: For (cid:96) 2 attacks, we set k = 500 for CIFAR10 and k = 50000 for ImageNet in our non-Gaussian smoothing distribution Equation (8).

Original: If the used model was trained with a Gaussian perturbation noise of N (0 , σ 20 ) , then the σ parameter of our smoothing distribution is set to be (cid:112) ( d − 1) /
Revised: If the used model was trained with a Gaussian perturbation noise of N (0 , σ 20 ) , then the σ parameter of our smoothing distribution is set to be (cid:112) ( d − 1) /

Original: ( d − 1 − k ) σ 0 , such that the expectation of the norm (cid:107) z (cid:107) 2 under our non-Gaussian distribution Equation (8) matches with the norm of N (0 , σ 20 ) .
Revised: ( d − 1 − k ) σ 0 , such that the expectation of the norm (cid:107) z (cid:107) 2 under our non-Gaussian distribution Equation (8) matches with the norm of N (0 , σ 20 ) .

Original: For (cid:96) ∞ situation, we set k = 250 and σ also equals to (cid:112) ( d − 1) /
Revised: For (cid:96) ∞ situation, we set k = 250 and σ also equals to (cid:112) ( d − 1) /

Original: ( d − 1 − k ) σ 0 for the mixed norm smoothing distribution Equation (9).
Revised: ( d − 1 − k ) σ 0 for the mixed norm smoothing distribution Equation (9).

Original: In both cases, the baseline algorithm uses a Gaussian smoothing distribution N (0 , σ 20 ) .
Revised: In both cases, the baseline algorithm uses a Gaussian smoothing distribution N (0 , σ 20 ) .

Original: More ablation study about k is deferred to Appendix C.
Revised: More ablation study about k is deferred to Appendix C.

Original: Evaluation Metrics
Revised: Evaluation Metrics

Original: The methods are evaluated using the certiﬁed accuracy deﬁned in Cohen et al.
Revised: The methods are evaluated using the certiﬁed accuracy deﬁned in Cohen et al.

Original: Given an input image x and a perturbation region B , the smoothed classiﬁer is called certiﬁed correct if its prediction is correct and has a guaranteed lower bound larger than 1 / 2 for δ ∈ B .
Revised: Given an input image x and a perturbation region B , the smoothed classiﬁer is called certiﬁed correct if its prediction is correct and has a guaranteed lower bound larger than 1 / 2 for δ ∈ B .

Original: The certiﬁed accuracy is the percentage of images that are certiﬁed correct.
Revised: The certiﬁed accuracy is the percentage of images that are certiﬁed correct.

Original: Following
Revised: Following

Original: Salman et al.
Revised: Salman et al.

Original: (2019), we calculate the certiﬁed accuracy of all the classiﬁers in Cohen et al.
Revised: (2019), we calculate the certiﬁed accuracy of all the classiﬁers in Cohen et al.

Original: (2019) or Salman et al.
Revised: (2019) or Salman et al.

Original: for various radius, and report the best results over all of classiﬁers.
Revised: for various radius, and report the best results over all of classiﬁers

Original:  5. (cid:96) 2 C ERTIFICATION
Revised: . 5. (cid:96) 2 C ERTIFICATION

Original: We test our method on CIFAR10 and ImageNet for (cid:96) 2 certiﬁcation.
Revised: We test our method on CIFAR10 and ImageNet for (cid:96) 2 certiﬁcation.

Original: For fair comparison, we use the same pre-trained models as Cohen et al.
Revised: For fair comparison, we use the same pre-trained models as Cohen et al.

Original: (2019), which is trained with Gaussian noise on both CIFAR10 and ImageNet dataset.
Revised: (2019), which is trained with Gaussian noise on both CIFAR10 and ImageNet dataset.

Original: The readers are referred to Appendix C for detailed ablation studies.
Revised: The readers are referred to Appendix C for detailed ablation studies.

Original: Table 1 and Table 2 report the certiﬁed accuracy of our method with the non-Gaussian smoothing distribution in Equation (8) and the baseline on CIFAR10 and ImageNet, respectively.
Revised: Table 1 and Table 2 report the certiﬁed accuracy of our method with the non-Gaussian smoothing distribution in Equation (8) and the baseline on CIFAR10 and ImageNet, respectively.

Original: We ﬁnd that our method consistently outperforms the baseline.
Revised: We ﬁnd that our method consistently outperforms the baseline.

Revised: [Figure]

Revised: CIFAR

Revised: [Figure]

Revised: Salman et al.

Original: 5. (cid:96) ∞ C ERTIFICATION
Revised: [Figure] OURS (%) [Table] [Figure]

Revised: [Table]

Revised: Gaussian

Original:  We test our lower bound based on the mixed norm family in Equation (9) for verifying (cid:96) ∞ attacking on CIFAR10, using the models trained by Salman et al.
Revised: Ours [Figure] 5. (cid:96) ∞ C ERTIFICATION We test our lower bound based on the mixed norm family in Equation (9) for verifying (cid:96) ∞ attacking on CIFAR10, using the models trained by Salman et al.

Original: The certiﬁed accuracy of our method and the baseline using Gaussian smoothing distribution are shown in Table 3.
Revised: The certiﬁed accuracy of our method and the baseline using Gaussian smoothing distribution are shown in Table 3.

Original: We can see that our method consistently outperforms the Gaussian distribution baseline by a large margin, which empirically shows our distribution is a more suitable distribution for (cid:96) ∞ perturbation.
Revised: We can see that our method consistently outperforms the Gaussian distribution baseline by a large margin, which empirically shows our distribution is a more suitable distribution for (cid:96) ∞ perturbation.

Original: To further conﬁrm the advantage of our method, we plot in Figure 5 the certiﬁed accuracy of our method and Gaussian baseline using models trained with Gaussian perturbation of different variances σ 0 , under different (cid:96) ∞ radius.
Revised: To further conﬁrm the advantage of our method, we plot in Figure 5 the certiﬁed accuracy of our method and Gaussian baseline using models trained with Gaussian perturbation of different variances σ 0 , under different (cid:96) ∞ radius.

Original: We again ﬁnd that our approach outperforms the baseline consistently, especially when the (cid:96) ∞ radius is large.
Revised: We again ﬁnd that our approach outperforms the baseline consistently, especially when the (cid:96) ∞ radius is large.

Original: We also experimented our method and baseline on ImageNet, but did not obtain non-trivial results.
Revised: We also experimented our method and baseline on ImageNet, but did not obtain non-trivial results.

Original: This is because (cid:96) ∞ veriﬁcation is extremely hard with very large dimensions.
Revised: This is because (cid:96) ∞ veriﬁcation is extremely hard with very large dimensions.

Original: Future work will investigate how to obtain non-trivial bounds for (cid:96) ∞ attacking at ImageNet scales with smoothing classiﬁers.[Table]
Revised: Future work will investigate how to obtain non-trivial bounds for (cid:96) ∞ attacking at ImageNet scales with smoothing classiﬁers.

Original: [-RADIUS (CIFAR-10) 2 /4 /
-]
Revised: [-RADIUS (CIFAR-10) 2 /4 /
-]

Original: Salman et al.

Original: OURS (%)

Original: [Table]

Original: [Figure]

Original: og [

Original: [Figure]

Original: cc

Original: [Figure]

Original: ac y

Original: [Figure]

Original: [Table]

Original: Gaussian

Original: [Table]

Original: [Figure]

Original: with

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We propose a general functional optimization based framework of adversarial certiﬁcation with non-Gaussian smoothing distributions.
Revised: We propose a general variational optimization based framework of adversarial certiﬁcation with non-Gaussian smoothing distributions.

Original: Based on the insights from our new framework and high dimensional geometry, we propose two new families of non-Gaussian smoothing distributions, which signiﬁcantly outperform the Gaussian-based smoothing for (cid:96) 2 and (cid:96) ∞ attacking, respectively.
Revised: Based on the insights from our new framework and high dimensional geometry, we propose two new families of non-Gaussian smoothing distributions, which signiﬁcantly outperform the Gaussian-based smoothing for (cid:96) 2 and (cid:96) ∞ attacking, respectively.

Original: Our work provides basis for a variety of future directions, including improved methods for (cid:96) p attacks, and tighter bounds based on adding additional constraints to our optimization framework.
Revised: Our work provides basis for a variety of future directions, including improved methods for (cid:96) p attacks, and tighter bounds based on adding additional constraints to our optimization framework.

Original: A P ROOFS A. P ROOF FOR THE STRONG DUALITY IN THEOREM
Revised: A P ROOFS A. P ROOF FOR THE STRONG DUALITY IN THEOREM

Original: Notice that by Lagrange multiplier method, our primal problem can be rewritten as follows:
Revised: We ﬁrst introduce the following lemma, which is a straight forward generalization of the strong Lagrange duality to functional optimization case. Lemma 3. Given some δ ∗ , we have

Revised: [Equation]

Revised: The proof of Lemma 3 is standard.

Revised: However, for completeness, we include it here.

Revised: Proof.

Revised: Without loss of generality, we assume f (cid:93)π 0 ( x 0 ) ∈ (0 , 1) , otherwise the feasible set is trivial.

Revised: Let α ∗ be the value of the optimal solution of the primal problem.

Revised: We deﬁne f (cid:93)π 0 ( x 0 )

Original: {+− E π 
+}
Revised: {+− E π 
+}

Revised: [ f ( x 0 + z )]

Revised: = h

Revised: [ f ] and g [ f ] =

Revised: E π δ ∗ [ f ( x 0 + z )] .

Revised: We deﬁne the following two sets:

Revised: [Equation]

Revised: Notice that both sets A and B are convex.

Revised: This is obvious for B .

Revised: For any ( v 1 , t 1 ) ∈ A and ( v 2 , t 2 ) ∈

Revised: A , we deﬁne f 1 ∈ F [0 , 1] such that h [ f 1 ] = v 1 , g [ f 1 ] ≤ t 1 (and similarly we deﬁne f 2 ).

Revised: Notice that for any γ ∈ [0 , 1] , we have

Revised: [Equation]

Revised: which implies that γ ( v 1 , t 1 ) + (1 − γ )( v 2 , t 2 )

Revised: ∈ A and thus A is convex.

Revised: Also notice that by deﬁnition, A ∩ B = ∅ .

Revised: Using separating hyperplane theorem, there exists a point ( q 1 , q 2 ) (cid:54) =

Revised: (0 , 0) and a value α such that for any ( v, t ) ∈ A , q 1 v + q 2 t ≥ α and for any (0 , s ) ∈ B , q 2 s ≤ α .

Revised: Notice that we must have q 2 ≥ 0 , otherwise, for sufﬁcient s , we will have q 2 s > α .

Revised: We thus have, for any f ∈ F [0 , 1] , we have

Revised: [Equation]

Revised: If q 2 > 0 , we have

Original: [Equation] 
Revised: [Equation] which gives the strong duality. If q 2 = 0 , we have for any f ∈ F [0 , 1] , q 1 h [ f ] ≥ 0 and by the separating hyperplane theorem, q 1 (cid:54) = 0 .

Original: and the dual problem is 
Revised: However, this case is impossible: If q 1 > 0 , choosing f ≡ 1 gives q 1 h [ f ] = q 1 (cid:0) f (cid:93)π 0 ( x 0 ) − 1 (cid:1) < 0 ; If q 1 < 0 , by choosing f ≡ 0 , we have q 1 h [ f ] =

Original: [Equation] Our strategy is to verify that the solution of the dual problem is also a solution of the primal problem. Notice that for both primal and dual problem, it is equivalent to change the maximum over λ from max λ ∈ R to max λ ≥ 0. Suppose that ( δ ∗ , f ∗ , λ ∗ ) is the solution of the dual problem.
Revised: q 1 (cid:0) f (cid:93)π 0 ( x 0 ) − 0 (cid:1) < 0 .

Original: Our proof is by verifying ( δ ∗ , f ∗ , λ ∗ ) = (cid:0) ¯ δ , ¯ f, ¯ λ (cid:1) , where (cid:0) ¯ δ , ¯ f, ¯ λ (cid:1) is some optimal solution of primal problem. Claim¯ f satisﬁes the constraint condition, i.e, E π 0 ¯ f = f (cid:93)π 0 . Proof of Claim 1: Otherwise, for example E π 0 ¯ f < p 0 , by letting λ → ∞ , the minimization problem has optimal value + ∞ .
Revised: Both cases give contradiction.

Original: However, by letting ¯ f be the classiﬁer f (cid:93) we use, we achieve a better solution, which makes contradiction. Claim 2 (cid:0) ¯ δ , ¯ f, λ ∗ (cid:1) is the optimal solution for the  primal problem. Proof of Claim 2: This is obvious as 
Revised: Based on Lemma 3, we have the proof of the strong duality as follows. Notice that by Lagrange multiplier method, our primal problem can be rewritten as follows:

Original: [Equation]
Revised: [Equation]

Original: Claim 3 Deﬁne a new constraint optimization problem Q :
Revised: and the dual problem is

Original: [Equation]
Revised: [Equation]

Original: And both ( δ ∗ , f ∗ ) and (cid:0) ¯ δ , ¯ f (cid:1) is the optimal solution for the problem Q .

Original: Proof of Claim 3:

Original: We ﬁrst prove that (cid:0) ¯ δ , ¯ f (cid:1) is the optimal solution for the problem Q .

Original: If not, then there exists ( (cid:48) , f (cid:48) )

Original: such that

Original: [Equation]
Revised: [Equation]

Original: Then ( δ (cid:48) , f (cid:48) , λ ∗ ) becomes a better solution for the primal problem, which makes contradiction. And by deﬁnition of ( δ ∗ , f ∗ , λ ∗ ) , ( δ ∗ , f ∗ ) is the minimizer of problem Q . Claim 4 Deﬁne
Revised: for have

Original: [Equation]
Revised: [Equation]

Original: By claim 1 and 3, we have L [¯ δ , ¯ f, ¯ λ ] = L [¯ δ , ¯ f, λ ∗ ] = L [ δ ∗ , f ∗ , λ ∗ ] , which implies that ( δ ∗ , f ∗ , λ ∗ ) is the optimal solution of primal solution.
Revised: where the second equality (*) is by Lemma 3.

Original: A. P ROOF FOR COROLLARY
Revised: A. P ROOF FOR COROLLARY

Original: Proof.
Revised: Proof.

Original: Given our conﬁdence lower bound
Revised: Given our conﬁdence lower bound

Original: [Equation]
Revised: [Equation]

Original: deﬁne C λ = { z : λπ 

Revised: deﬁne C λ = { z : λπ

Original: ( z ) ≥ π δ ( z ) } =
Revised: ( z ) ≥ π δ ( z ) } =

Original: { z : δ (cid:62) z ≤ (cid:107) δ (cid:107) 2+ σ 2 ln λ } and Φ( · ) to be the cdf of standard gaussian distribution, then 
Revised: { z : δ (cid:62) z ≤ (cid:107) δ (cid:107) 2+ σ 2 ln λ } and Φ( · ) to be the cdf of standard gaussian distribution, then [Equation] =

Revised: [Equation]

Revised: P (cid:0) N ( z ; 0 , σ 2 I ) ∈

Original: [Equation]
Revised: [Equation]

Revised: λ (cid:1) − P (cid:0) N ( z

Revised: [Equation]

Revised: δ

Revised: [Equation]

Revised: σ 2 I ) ∈ C λ (cid:1) =

Revised: λ · Φ (cid:18) (cid:107) δ (cid:107)2 σ + σ ln λ (cid:107) δ (cid:107)(cid:19) − Φ (cid:18) −(cid:107) δ (cid:107)2 σ + σ ln λ (cid:107) δ (cid:107)(cid:19) .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: F ( δ , λ ) : = λp 0 − (cid:90) ( λπ 0 ( z ) − π δ ( z )) + dz = λp 0 − λ · Φ (cid:18) (cid:107) δ (cid:107)2 σ + σ ln λ (cid:107) δ (cid:107)(cid:19) +Φ (cid:18) −(cid:107) δ (cid:107)2 σ + σ ln λ (cid:107) δ (cid:107)(cid:19) .

Original: For ∀ δ , F is a concave function w.r.t.
Revised: For ∀ δ , F is a concave function w.r.t.

Original: λ , as F is actually a summation of many concave piece wise linear function.
Revised: λ , as F is actually a summation of many concave piece wise linear function.

Original: See Boyd & Vandenberghe (2004) for more discussions of properties of concave functions.
Revised: See Boyd & Vandenberghe (2004) for more discussions of properties of concave functions.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: (cid:17) , simple calculation can show ∂F ( δ ,λ ) ∂λ | λ =ˆ λ δ = 0 , which means
Revised: (cid:17) , simple calculation can show ∂F [Equation] | λ = ˆ λ δ = 0 , which means

Original: [Equation]
Revised: [Equation]

Original: This tells us
Revised: This tells us

Original: [Equation]
Revised: [Equation]

Original: , i.e. the certiﬁcation radius is σ · Φ − 1 ( p 0 ) .
Revised: , i.e. the certiﬁcation radius is σ · Φ − 1 ( p 0 ) .

Original: This is exactly the core theoretical contribution of Cohen et al.
Revised: This is exactly the core theoretical contribution of Cohen et al.

Original: This bound has a straight forward expansion for multi-class classiﬁcation situations, we refer interesting readers to Appendix D.
Revised: This bound has a straight forward expansion for multi-class classiﬁcation situations, we refer interesting readers to Appendix D.

Original: A. P ROOF F OR T HEOREM 2 AND
Revised: A. P ROOF F OR T HEOREM 2 AND

Original: In this subsection, we give the proof of theorem 2 and 5.
Revised: In this subsection, we give the proof of theorem 2 and 5.

Original: Here we consider a more general smooth distribution π 0 ( z ) ∝
Revised: Here we consider a more general smooth distribution π [Equation] , for some k

Original: [Equation]
Revised: [Equation]

Original: σ 2 (cid:17) , for some k 1 , k 2 ≥ 0 and σ > 0 .
Revised: k [Equation] ≥ 0 and σ > 0 .

Original: We ﬁrst gives the following key theorem shows that D F [0 , 1] ( λπ 0 (cid:107) π δ ) increases as | δ i | becomes larger for every dimension i .
Revised: We ﬁrst gives the following key theorem shows that D F [0 , 1] ( λπ 0 (cid:107) π δ ) increases as | δ i | becomes larger for every dimension i .

Original: Theorem 6.
Revised: Theorem 6.

Original: Suppose π 0 ( z ) ∝ (cid:107) z (cid:107) − k 1 ∞ (cid:107) z (cid:107) − k2exp (cid:16) − (cid:107) z (cid:107) 
 
Revised: Suppose π 0 ( z ) ∝ (cid:107) z (cid:107) − k 1 ∞ (cid:107) z (cid:107) − k 
 [Equation]

Original: σ 2 (cid:17) , for some k 1 , k 2 ≥ 0 and σ > 0 , for any λ ≥ 

Revised: 2 (cid:17) , for some k 1 , k 2 ≥ 0 and σ > 0 , for any λ ≥

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: for any i ∈ { 1 , 2 , ..., d } .
Revised: for any i ∈ { 1 , 2 , ..., d } .

Original: Theorem 2 and 5 directly follows the above theorem.
Revised: Theorem 2 and 5 directly follows the above theorem.

Original: Notice that in Theorem 2, as our distribution is spherical symmetry, it is equivalent to set B = (cid:8) δ : δ =
Revised: Notice that in Theorem 2, as our distribution is spherical symmetry, it is equivalent to set B = (cid:8) δ : δ =

Original: [ a, 0 , ..., 0] (cid:62) , a ≤ r (cid:9) by rotating the axis.
Revised: [ a, 0 , ..., 0] (cid:62) , a ≤ r (cid:9) by rotating the axis.

Original: Proof.
Revised: Proof.

Original: Given λ , k 1 and k 2 , we deﬁne φ 1 ( s ) =
Revised: Given λ , k 1 and k 2 , we deﬁne φ 1 ( s ) =

Original: s − k 1 , φ 2 ( s ) =
Revised: s − k 1 , φ 2 ( s ) =

Original: s − k 2 e − s 2 σ 2 .
Revised: s − k 2 e − s 2 σ 2 .

Original: Notice that φ 1 and φ 2 are monotone decreasing for non-negative s .
Revised: Notice that φ 1 and φ 2 are monotone decreasing for non-negative s .

Original: By the symmetry, without loss of generality, we assume δ =
Revised: By the symmetry, without loss of generality, we assume δ =

Original: [ δ 1 , ..., δ d ] (cid:62) for δ i ≥ 0 , i ∈
Revised: [ δ 1 , ..., δ d ] (cid:62) for δ i ≥ 0 , i ∈

Original: [ d ] .
Revised: [ d ] .

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: And also
Revised: And also

Original: [Equation]
Revised: [Equation]

Original: We thus have 
Revised: We thus have ∂

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne
Revised: where we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Notice that as φ (cid:48) 1 ≤ 0 and φ (cid:48) 2 ≤ 

Revised: Notice that as φ (cid:48) 1 ≤ 0 and φ (cid:48) 2 ≤

Original: and we have
Revised: and we have

Original: [Equation]
Revised: [Equation]

Original: Our target is to prove that ∂∂δ 1 (cid:82) ( λπ 0 ( x 0 ) − π δ ( x 0 ))
Revised: Our target is to prove that ∂∂δ 1 (cid:82) ( λπ 0 ( x 0 ) − π δ ( x 0 ))

Original: + d x 0 ≥ 0 .
Revised: + d x 0 ≥ 0 .

Original: Now deﬁne the set
Revised: Now deﬁne the set

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Here the set H 2 is deﬁned as a image of a bijection
Revised: Here the set H 2 is deﬁned as a image of a bijection

Original: [Equation]
Revised: [Equation]

Original: that is constrained on the set H 1 .
Revised: that is constrained on the set H 1 .

Original: Notice that under our deﬁnition,
Revised: Notice that under our deﬁnition,

Original: [Equation]
Revised: [Equation]

Original: Now we prove that 
Revised: Now we prove that (cid:90)

Original: [Equation]
Revised: [Equation]

Original: Property of the projection Before we prove the (1) and (2), we give the following property of the deﬁned projection function.
Revised: Property of the projection Before we prove the (1) and (2), we give the following property of the deﬁned projection function.

Original: For any ˜ x 0 = proj( x 0 ) , x 0 ∈ H 1 , we have
Revised: For any ˜ x 0 = proj( x 0 ) , x 0 ∈ H 1 , we have

Original: [Equation]
Revised: [Equation]

Original: This is because 
Revised: This is because ˜ x i = x

Original: [Equation]
Revised: i , i ∈

Original:  and by the fact that x 1 ≥ δ 1 ≥ 0 , we have | ˜ x 1 | ≤ | x 1 | and | ˜ x 1 − δ 1 | ≤ | x 1 − δ 1 | .
Revised: [ d ] − { 1 } ˜ x 1 = 2 δ 1 − x 1 , and by the fact that x 1 ≥ δ 1 ≥ 0 , we have | ˜ x 1 | ≤ | x 1 | and | ˜ x 1 − δ 1 | ≤ | x 1 − δ 1 | .

Original: Proof of Equality (2)
Revised: Proof of Equality (2)

Original: By the fact that proj is bijective constrained on the set H 1 and the property of proj , we have
Revised: By the fact that proj is bijective constrained on the set H 1 and the property of proj , we have

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) is by change of variable ˜ x 0 =
Revised: where ( ∗ ) is by change of variable ˜ x 0 =

Original: proj( x 0 ) and J is the Jacobian matrix
Revised: proj( x 0 ) and J is the Jacobian matrix

Original: J = 
Revised: J = 

Original:  − 1 0 · · ·  1 · · · ...
Revised:  − 1 0 · · ·  1 · · · ...

Original: 0 · · ·   and here we have the fact that ˜ x 1 − δ 

Revised: 0 · · ·   and here we have the fact that ˜ x 1 − δ

Original: =
Revised: =

Original: (2 δ 1 − x 1 ) − δ 1 = − ( x 1 − δ 1 ) .
Revised: (2 δ 1 − x 1 ) − δ 1 = − ( x 1 − δ 1 ) .

Original: Proof of
Revised: Proof of

Original: Inequality (1) This can be done by verifying that
Revised: Inequality (1) This can be done by verifying that

Original: H⊆ { x 0 : λπ 0 ( x 0 ) ≥ π δ ( x 0 ) , x 1 < δ 1 } .
Revised: H⊆ { x 0 : λπ 0 ( x 0 ) ≥ π δ ( x 0 ) , x 1 < δ 1 } .

Original: By the property of the projection, for any x 0 ∈ H 1 , let ˜ x 0 =
Revised: By the property of the projection, for any x 0 ∈ H 1 , let ˜ x 0 =

Original: proj
Revised: proj

Original: ( x 0 ) , then λπ 0 ( ˜ x 0 ) ≥ λπ 0 ( x 0 ) ≥ π δ ( x 0 ) =
Revised: ( x 0 ) , then λπ 0 ( ˜ x 0 ) ≥ λπ 0 ( x 0 ) ≥ π δ ( x 0 ) =

Original: π δ ( ˜ x 0 ) (by the fact that t φ 1 and φ 2 are monotone decreasing).
Revised: π δ ( ˜ x 0 ) (by the fact that t φ 1 and φ 2 are monotone decreasing).

Original: It implies that for any ˜ x 0 ∈ H 2 , we have λπ 0 ( ˜ x 0 ) ≥ π δ ( ˜ x 0 ) and thus
Revised: It implies that for any ˜ x 0 ∈ H 2 , we have λπ 0 ( ˜ x 0 ) ≥ π δ ( ˜ x 0 ) and thus

Original: H 2 ⊆ { x 0 : π 0 ( x 0 ) ≥ π δ ( x 0 ) ,
Revised: H 2 ⊆ { x 0 : π 0 ( x 0 ) ≥ π δ ( x 0 ) ,

Original: x 1 < δ 1 } .
Revised: x 1 < δ 1 } .

Original: Final statement By the above result, we have
Revised: Final statement By the above result, we have

Original: [Equation]
Revised: [Equation]

Original: and the same result holds for any ∂∂δ 1 (cid:82) ( λπ0 ( x 0 ) − π µ ( x 0 ))
Revised: and the same result holds for any ∂ ∂δ [Equation] ( λπ

Revised: [Equation]

Revised: ( x

Revised: [Equation]

Revised: ) − π µ ( x 0 ))

Original: + d x 0 , i ∈
Revised: + d x 0 , i ∈

Original: [ d ] , which implies our result.
Revised: [ d ] , which implies our result.

Original: A.3. P ROOF FOR T HEOREM
Revised: A.3. P ROOF FOR T HEOREM

Original: First notice that the distribution of z can be factorized by the following hierarchical scheme:
Revised: First notice that the distribution of z can be factorized by the following hierarchical scheme:

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, we assume δ ∗ =
Revised: Without loss of generality, we assume δ ∗ =

Original: [ r, ..., r ] (cid:62) .
Revised: [ r, ..., r ] (cid:62) .

Original: (see Theorem 6)
Revised: (see Theorem 6)

Original: [Equation]
Revised: [Equation]

Original: Notice that as the distribution is symmetry,
Revised: Notice that as the distribution is symmetry,

Original: [Equation]
Revised: [Equation]

Original: Deﬁne | z | ( i ) is the i -th order statistics of | z j | , j = 1 , ..., d conditioning on (cid:107) z (cid:107) ∞
Revised: Deﬁne | z | ( i ) is the i -th order statistics of | z j | , j = 1 , ..., d conditioning on (cid:107) z (cid:107) ∞

Original: =
Revised: =

Original: a .
Revised: a .

Original: By the factorization above and some algebra, we have, for any (cid:15) ∈ (0 , 1) , 
Revised: By the factorization above and some algebra, we have, for any (cid:15) ∈ (0 , 1) , [Equation]

Original: P
Revised: ≥ [Equation] − [Equation]

Original: [Equation]
Revised: [Equation]

Original: And | z | ( d − 1) | z | ( d ) ⊥ | z | ( d ) .
Revised: And | z | ( d − 1) | z | ( d ) ⊥ | z | ( d ) .

Original: Now we estimate D F [0 , 1] ( λπ 0 (cid:107) π δ ∗ ) .
Revised: Now we estimate D F [0 , 1] ( λπ 0 (cid:107) π δ ∗ ) .

Original: [Equation] (cid:34)(cid:18)
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Conditioning on (cid:107) z (cid:107) ∞ = a, (cid:107) z + δ ∗ (cid:107) ∞ =
Revised: Conditioning on (cid:107) z (cid:107) ∞ = a, (cid:107) z + δ ∗ [Equation] ∞ =

Original: a + r , we have
Revised: a + r , we have

Original: [Equation]
Revised: [Equation]

Original: Here the second equality is because we choose mode( (cid:107) z (cid:107) ∞ ) =
Revised: Here the second equality is because we choose mode( (cid:107) z (cid:107) ∞ ) =

Original: Cr , which implies that √ d − 1 − kσ =
Revised: Cr , which implies that √ d − 1 − kσ =

Original: Cr .
Revised: Cr .

Original: And thus we have
Revised: And thus we have

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: O

Original: [Equation]
Revised: [Equation]

Revised: t

Revised: [Equation]

Original:  for some t > 1 .
Revised: ) , for some t > 1 .

Original: Here the last equality is by  assumption that (cid:107) z (cid:107) ∞
Revised: Here the last equality is by the assumption that (cid:107) z (cid:107) ∞

Original: =
Revised: =

Original: O p ( [Equation]
Revised: O p (1) .

Original: Next we bound the second term 
Revised: Next we bound the second term E a E

Original: [Equation]
Revised: [Equation]

Original: By the property of uniform distribution, we have
Revised: By the property of uniform distribution

Original: [Equation]
Revised: [Equation]

Original: And thus
Revised: we have

Original: [Equation]
Revised: [Equation]

Original:  for any (cid:15) ∈ [0 , 1) ,P
Revised: And thus, for any (cid:15) ∈ [0 , 1) ,

Original: [Equation]
Revised: [Equation]

Original: It implies that
Revised: It implies that

Original: [Equation]
Revised: [Equation]

Original: + 
Revised: + .

Original: [Equation] For any (cid:15) (cid:48) ∈ (0 , 1) , by choosing (cid:15) = log(2 /(cid:15) (cid:48) )
Revised: For any (cid:15) (cid:48) ∈ (0 , 1) , by choosing (cid:15) = log(2 /(cid:15) (cid:48) )

Original: d − 1 , for large enough d , we have
Revised: d − 1 , for large enough d , we have

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]

Original: [Equation]

Original: λ − (cid:18) 1 − log(2 /(cid:15) (cid:48) )

Original: d −+ r a (cid:19) − k e − d − 1 − k 2 C 2 (2(1 − (cid:15) ) a/r +1)

Original: e a 2 log(2 /(cid:15) (cid:48) )

Original: C 2 r(cid:33) + ≥2 (1 − (cid:15) (cid:48) ) (cid:18)
Revised: ≥

Original: [Equation]

Original: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:15) [Equation] (cid:17) − k [Equation]
Revised: =  (cid:18)

Original: − d − 1 − k

Original: [Equation]
Revised: [Equation]

Original: C

Original: [Equation]

Original: [Equation]

Original:  (1 − (cid:15)) a/r [Equation] [Equation] a
Revised: ≥2 (1 − (cid:15)

Original: [Equation]
Revised: [Equation]

Original: log(2 /(cid:15) (cid:48) )

Original: C 2 r(cid:19)

Original: + .

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: Combine the bounds, for large d , we have
Revised: Combine the bounds, for large d , we have

Original: [Equation]
Revised: [Equation]

Original: B P RACTICAL A LGORITHM
Revised: B P RACTICAL A LGORITHM

Original: In this section, we give our algorithm for certiﬁcation.
Revised: In this section, we give our algorithm for certiﬁcation.

Original: Our target is to give a high probability bound for the solution of
Revised: Our target is to give a high probability bound for the solution of

Original: [Equation]
Revised: [Equation]

Original: δ ) (cid:9) given some classiﬁer f (cid:93) .
Revised: given some classiﬁer f (cid:93) .

Original: Following Cohen et al.
Revised: Following Cohen et al.

Original: (2019), the given classiﬁer here has a binary output { 0 , 1 } .
Revised: (2019), the given classiﬁer here has a binary output { 0 , 1 } .

Original: Computing the above quantity requires us to evaluate both f (cid:93)π 0 and D F [0 , 1] ( λπ 0 (cid:107) π δ ) .
Revised: Computing the above quantity requires us to evaluate both f (cid:93) π 0 and D F [0 , 1] ( λπ 0 (cid:107) π δ ) .

Original: A lower bound ˆ p 0 of the former term is obtained through binominal test as Cohen et al.
Revised: A lower bound ˆ p 0 of the former term is obtained through binominal test as Cohen et al.

Original: (2019) do, while the second term can be estimated with arbitrary accuracy using Monte Carlo samples.
Revised: (2019) do, while the second term can be estimated with arbitrary accuracy using Monte Carlo samples.

Original: We perform grid search to optimize λ and given λ , we draw N i .
Revised: We perform grid search to optimize λ and given λ , we draw N i .

Original: i .
Revised: i .

Original: d .
Revised: d .

Original: samples from the proposed smoothing distribution π 0 to estimate λf (cid:93)π 0 − D F [0 , 1] ( λπ 0 (cid:107) π δ ) .
Revised: samples from the proposed smoothing distribution π 0 to estimate λf (cid:93)π 0 − D F [0 , 1] ( λπ 0 (cid:107) π δ ) .

Original: This can be achieved by the following importance sampling manner:
Revised: This can be achieved by the following importance sampling manner

Original: [Equation]
Revised: [Equation]

Original: And we use reject sampling to obtain samples from π 0 .
Revised: And we use reject sampling to obtain samples from π 0 .

Original: Notice that, we restrict the search space of λ to a ﬁnite compact set so the importance samples is bounded.
Revised: Notice that, we restrict the search space of λ to a ﬁnite compact set so the importance samples is bounded.

Original: Since the Monte Carlo estimation is not exact with an error (cid:15) , we give a high probability concentration lower bound of the estimator.
Revised: Since the Monte Carlo estimation is not exact with an error (cid:15) , we give a high probability concentration lower bound of the estimator.

Original: Algorithm 1 summarized our algorithm.
Revised: Algorithm 1 summarized our algorithm.

Original: Λ =
Revised: Algorithm

Original:  range( λ start, λend , h) compute N2 : number of Monte Carlo estimation given (cid:15), α and Λ compute optimal disturb: δ depends on speciﬁc setting for λ in Λ do sample z 1 , · · · , z N 1 ∼ πcompute n 1 = 1 N 1 (cid:80) N 1 i =1 f (cid:93) ( x 0 + z i ) compute ˆ p 

Revised: Input compute search λ Λ = range λ start λ h compute N number of Monte Carlo estimation given (cid:15), α and Λ compute optimal disturb: δ depends on speciﬁc setting for λ in Λ do sample z 1 , · · · , z N 1 ∼ πcompute n 1 = 1 N 1 (cid:80) N 1 i =1 f (cid:93) ( x 0 + z i ) compute ˆ p

Original: = LowerConfBound ( n 1 , N 1 , 1 − α ) sample z 1 , · · · , z N 2 ∼ πcompute ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ )
Revised: = LowerConfBound ( n 1 , N 1 , 1 − α ) sample z 1 , · · · , z N 2 ∼ πcompute ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ )

Original: = 1 N 2 (cid:80) N 2 i =1 (cid:16) λ − π δ π 0 ( z i ) (cid:17) + compute conﬁdence lower bound b λ =
Revised: = 1 N 2 (cid:80) N 2 i =1 (cid:16) λ − π δ π 0 ( z i ) (cid:17) + compute conﬁdence lower bound b λ =

Original: λ ˆ p 0 − ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ ) − (cid:15) endif max λ ∈ Λ b λ ≥ 1 / 2 then x 0 can be certiﬁed else x 0 cannot be certiﬁed end
Revised: λ ˆ p 0 − ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ ) − (cid:15) endif max λ ∈ Λ b λ ≥ 1 / 2 then x 0 can be certiﬁed else x 0 cannot be certiﬁed end

Original: The LowerConfBound function performs a binominal test as described in Cohen et al.
Revised: The LowerConfBound function performs a binominal test as described in Cohen et al.

Original: The (cid:15) in Algorithm 1 is given by concentration inequality.
Revised: The (cid:15) in Algorithm 1 is given by concentration inequality.

Original: Theorem 7.
Revised: Theorem 7.

Original: Let h ( z 1 , · · · , z N ) = 1 N (cid:80) Ni
Revised: Let h ( z 1 , · · · , z N ) = 1 N (cid:80) Ni

Original: =1 (cid:16) λ − π δ ( z i ) π
Revised: =1 (cid:16) λ − π δ ( z i ) π

Original: ( z i ) (cid:17)
Revised: ( z i ) (cid:17)

Original: + , we yield
Revised: + , we yield

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Given McDiarmid’s Inequality , which sayssup
Revised: Given McDiarmid’s Inequality , which says

Original: [Equation]
Revised: [Equation]

Original: , x n ) | ≤ c i for 1 ≤ i ≤ n, we have c i = λN , and then obtain
Revised: for 1 ≤ i [Equation] n, we have c i = λN , and then obtain

Original: [Equation]
Revised: [Equation]

Original: The above theorem tells us that, once (cid:15), λ, N is given, we can yield a bound with high-probability 1 − α .
Revised: The above theorem tells us that, once (cid:15), λ, N is given, we can yield a bound with high-probability 1 − α .

Original: One can also get N when (cid:15), λ, α is provided.
Revised: One can also get N when (cid:15), λ, α is provided.

Original: Note that this is the same as the Hoeffding bound mentioned in Section 4.1 as Micdiarmid bound is a generalization of Hoeffding bound.
Revised: Note that this is the same as the Hoeffding bound mentioned in Section 4.1 as Micdiarmid bound is a generalization of Hoeffding bound.

Original: However, in practice we can use a small trick as below to certify with much less comupation:
Revised: However, in practice we can use a small trick as below to certify with much less comupation:

Original: Algorithm
Revised: λ

Original: Input: input image x 0 ; original classiﬁer: f (cid:93) ; smoothing distribution π 0 ; radius r ; search interval for λ : [ λ start , λ end ] ; search precision h for optimizing λ; number of Monte Carlo for ﬁrst estimation: N 01 , N 02 ; number of samples N 1 for a second test of p 0 ; pre-deﬁned error threshold (cid:15) ; signiﬁcant level α ; optimal perturbation δ ( δ = [ r, 0 , . . . , 0] (cid:62) for (cid:96) 2 attacking and δ = [ r, . . . , r ] (cid:62) for (cid:96) ∞ attacking).
Revised: λ

Original: for λ in Λ do sample z 1 , · · · , z N 01 ∼ πcompute n 0 1 = 1 N 01 (cid:80) N 01 i = 1 f (cid:93) ( x 0 + z i ) compute ˆ p 

Revised: for λ in Λ do sample z 1 , · · · , z N 01 ∼ πcompute n 01 = 1 N 01 (cid:80) N 01 i =1 f (cid:93) ( x 0 + z i ) compute ˆ p

Original: = LowerConfBound ( n 01 , N 01 , 1 − α ) sample z 1 , · · · , z N 02 ∼ πcompute ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ )
Revised: = LowerConfBound ( n 01 , N 01 , 1 − α ) sample z 1 , · · · , z N 02 ∼ πcompute ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ )

Original: = 1 N 02 (cid:80) N 02 i =1 (cid:16) λ − π δ π 0 ( z i ) (cid:17) + compute conﬁdence lower bound b λ =
Revised: = 1 N 02 (cid:80) N 02 i =1 (cid:16) λ − π δ π 0 ( z i ) (cid:17) + compute conﬁdence lower bound b λ =

Original: λ ˆ p 0 − ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ ) end compute ˆ λ = arg max λ ∈ Λ b λ compute N 2 : number of Monte Carlo estimation given (cid:15), α and ˆ λ sample z 1 , · · · , z N 1 ∼ πcompute n 1 = 1 N 1 (cid:80) N 1 i =
Revised: λ ˆ p 0 − ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ ) end compute ˆ λ = arg max λ ∈ Λ b λ compute N 2 : number of Monte Carlo estimation given (cid:15), α and ˆ λ sample z 1 , · · · , z N 1 ∼ πcompute n 1 = 1 N 1 (cid:80) N 1 i =

Original: 1 f (cid:93) ( x 0 + z i ) compute ˆ p 

Revised: 1 f (cid:93) ( x 0 + z i ) compute ˆ p

Original: = LowerConfBound ( n 1 , N 1 , 1 − α ) sample z 1 , · · · , z N 2 ∼ πcompute ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ )
Revised: = LowerConfBound ( n 1 , N 1 , 1 − α ) sample z 1 , · · · , z N 2 ∼ πcompute ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ )

Original: = 1 N 2 (cid:80) N 2 i =1 (cid:16) λ − π δ π 0 ( z i ) (cid:17) + compute b
Revised: = 1 N 2 (cid:80) N 2 i =1 (cid:16) λ − π δ π 0 ( z i ) (cid:17) + compute b

Original: = ˆ λ ˆ p 0 − ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ ) − (cid:15) if b ≥ 1 / 2 then x 0 can be certiﬁed else x 0 cannot be certiﬁed end
Revised: = ˆ λ ˆ p 0 − ˆ D F [0 , 1] ( λπ 0 (cid:107) π δ ) − [Equation] if b ≥ 1 / 2 then x 0 can be certiﬁed else x 0 cannot be certiﬁed end

Original: Algorithm 2 allow one to begin with small N 01 , N 02 to obtain the ﬁrst estimation and choose a ˆ λ .
Revised: Algorithm 2 allow one to begin with small N 01 , N 02 to obtain the ﬁrst estimation and choose a ˆ λ .

Original: Then a rigorous lower bound can be achieved with ˆ λ with enough (i.e. N 1 , N 2 ) Monte Carlo samples.
Revised: Then a rigorous lower bound can be achieved with ˆ λ with enough (i.e. N 1 , N 2 ) Monte Carlo samples.

Original: C A BALATION STUDY
Revised: C A BALATION STUDY

Original: On CIFAR10, we also do ablation study to show the inﬂuence of different k for the (cid:96) 2 certiﬁcation case as shown in Table 4. 
Revised: On CIFAR10, we also do ablation study to show the inﬂuence of different k for the (cid:96) 2 certiﬁcation case as shown in Table 4. [Table]

Original: D I LLUMINATION ABOUT BILATERAL CONDITION
Revised: D I LLUMINATION ABOUT BILATERAL CONDITION

Original: The results in the main context is obtained under binary classﬁcation setting.
Revised: The results in the main context is obtained under binary classﬁcation setting.

Original: Here we show it has a natural generalization to multi-class classiﬁcation setting.
Revised: Here we show it has a natural generalization to multi-class classiﬁcation setting.

Original: Suppose the given classiﬁer f (cid:93) classiﬁes an input x 0 correctly to class A, i.e.,
Revised: Suppose the given classiﬁer f (cid:93) classiﬁes an input x 0 correctly to class A, i.e.,

Original: [Equation]
Revised: [Equation]

Original: [Table] where f (cid:93)B ( x 0 ) denotes the prediction conﬁdence of any class B different from ground truth label
Revised: (11) where f (cid:93)B ( x 0 ) denotes the prediction conﬁdence of any class B different from ground truth label

Original: A .
Revised: A .

Original: Notice that f (cid:93)A ( x 0 ) + (cid:80) B (cid:54) =
Revised: Notice that f (cid:93)A ( x 0 ) + (cid:80) B (cid:54) =

Original: A f (cid:93)B ( x 0 ) = 1 , so the necessary and sufﬁcient condition for correct binary classiﬁcation f (cid:93)A ( x 0 ) > 1 / 2 becomes a sufﬁcient condition for multi-class prediction.
Revised: A f (cid:93)B ( x 0 ) = 1 , so the necessary and sufﬁcient condition for correct binary classiﬁcation f (cid:93)A ( x 0 ) > 1 / 2 becomes a sufﬁcient condition for multi-class prediction.

Original: Similarly, the necessary and sufﬁcient condition for correct classiﬁcation of the smoothed classiﬁer is
Revised: Similarly, the necessary and sufﬁcient condition for correct classiﬁcation of the smoothed classiﬁer is

Original: [Equation]
Revised: [Equation]

Original: for ∀ B (cid:54)
Revised: for ∀ B (cid:54)

Original: =
Revised: =

Original: A and any perturbation δ ∈ B .
Revised: A and any perturbation δ ∈ B .

Original: Writing out their Langragian forms makes things clear:
Revised: Writing out their Langragian forms makes things clear:

Original: [Equation]
Revised: [Equation]

Original: Thus the overall necessary and sufﬁcient condition is min
Revised: Thus the overall necessary and sufﬁcient condition is min

Original: [Equation]
Revised: [Equation]

Original: Optimizing this bilateral object will theoretically give a better certiﬁcation result than our method in main context, especially when the number of classes is large.
Revised: Optimizing this bilateral object will theoretically give a better certiﬁcation result than our method in main context, especially when the number of classes is large.

Original: But we do not use this bilateral formulation as reasons stated below.
Revised: But we do not use this bilateral formulation as reasons stated below.

Original: When both π 0 and π δ are gaussian, which is Cohen et al.
Revised: When both π 0 and π δ are gaussian, which is Cohen et al.

Original: (2019)’s setting, this condition is equivalent to:
Revised: (2019)’s setting, this condition is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: with a similar derivation process like Appendix A.2.
Revised: with a similar derivation process like Appendix A.2.

Original: This is exactly the same bound in the (restated) theorem 1 of Cohen et al.
Revised: This is exactly the same bound in the (restated) theorem 1 of Cohen et al.

Original: Cohen et al.
Revised: Cohen et al.

Original: (2019) use 1 − p
Revised: (2019) use 1 − p

Original: A as a naive estimate of the upper bound of f (cid:93)π 0 ,B ( x 0 ) , where p A is a lower bound of f (cid:93)π 0 ,A ( x 0 ) .
Revised: A as a naive estimate of the upper bound of f (cid:93)π 0 ,B ( x 0 ) , where p A is a lower bound of f (cid:93)π 0 ,A ( x 0 ) .

Original: This leads the conﬁdence bound decay to the bound one can get in binary case, i.e., r ≤ σ Φ − 1 ( f (cid:93)π 0 ,A ( x 0 )) .
Revised: This leads the conﬁdence bound decay to the bound one can get in binary case, i.e., r ≤ σ Φ − 1 ( f (cid:93)π 0 ,A ( x 0 )) .

Original: As the two important baselines (Cohen et al., 2019; Salman et al., 2019) do not take the bilateral form, we also do not use this form in experiments for fairness.
Revised: As the two important baselines (Cohen et al., 2019; Salman et al., 2019) do not take the bilateral form, we also do not use this form in experiments for fairness.