Original: Data-Dependent Randomized Smoothing
Revised: Data-Dependent Randomized Smoothing

Original: Abstract
Revised: Abstract

Original: Randomized smoothing is a recent technique that achieves state-of-art performance in training certiﬁably robust deep neural networks.
Revised: Randomized smoothing is a recent technique that achieves state-of-art performance in training certifiably robust deep neural networks.

Original: While the smoothing family of distributions is often connected to the choice of the norm used for certiﬁcation, the parameters of these distributions are always set as global hyper parameters independent from the input data on which a network is certiﬁed.
Revised: While the smoothing family of distributions is often connected to the choice of the norm used for certification, the parameters of these distributions are always set as global hyper parameters independent from the input data on which a network is certified.

Original: In this work, we revisit Gaussian randomized smoothing and show that the variance of the Gaussian distribution can be optimized at each input so as to maximize the certiﬁcation radius for the construction of the smooth classiﬁer.
Revised: In this work, we revisit Gaussian randomized smoothing and show that the variance of the Gaussian distribution can be optimized at each input so as to maximize the certification radius for the construction of the smooth classifier.

Original: Since the data dependent classiﬁer does not directly enjoy sound certiﬁcation with existing approaches, we propose a memory-enhanced data dependent smooth classiﬁer that is certiﬁable by construction.
Revised: Since the data dependent classifier does not directly enjoy sound certification with existing approaches, we propose a memory-enhanced data dependent smooth classifier that is certifiable by construction.

Original: This new approach is generic, parameter-free, and easy to implement.
Revised: This new approach is generic, parameter-free, and easy to implement.

Original: In fact, we show that our data dependent framework can be seamlessly incorporated into 3 randomized smoothing approaches, leading to consistent improved certiﬁed accuracy.
Revised: In fact, we show that our data dependent framework can be seamlessly incorporated into 3 randomized smoothing approaches, leading to consistent improved certified accuracy.

Original: When this framework is used in the training routine of these approaches followed by a data dependent certiﬁcation, we achieve 9% and 6% improvement over the certiﬁed accuracy of the strongest baseline for a radius of 0.5 on CIFAR10 and ImageNet.
Revised: When this framework is used in the training routine of these approaches followed by a data dependent certification, we achieve 9% and 6% improvement over the certified accuracy of the strongest baseline for a radius of 0.5 on CIFAR10 and ImageNet.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Revised: ing a fixed σ for all inputs to smooth f θ may under certify (results in smaller certification radius) inputs far from decision boundary e.g. x 1 , decrease in prediction confidence as for x 2 or produce incorrect predictions as for x 3 .

Original: Despite the success of Deep Neural Networks (DNNs) in various learning tasks [Krizhevsky et al., 2012, Long et al., 2015] they were shown to be vulnerable to small carefully crafted adversarial perturbations [Goodfellow et al., 2015,
Revised: Thus, smoothing should vary per input (right figure) to alleviate the aforementioned issues. 2015], they were shown to be vulnerable to small carefully crafted adversarial perturbations [Goodfellow et al., 2015,

Original: Szegedy et al., 2013].
Revised: Szegedy et al., 2013].

Original: For a DNN f that correctly classiﬁes an image x , f can be fooled to produce an incorrect predic- tion for x + η even when the adversary η is so small that x and x + η are indistinguishable to the human eye.
Revised: For a DNN f that correctly classifies an image x , f can be fooled to produce an incorrect prediction for x + η even when the adversary η is so small that x and x + η are indistinguishable to the human eye.

Original: To circumvent this nuisance, there have been several works proposing heuristic training procedures to build networks that are robust against such perturbations [Cisse et al., 2017, Madry et al., 2018].
Revised: To circumvent this nuisance, there have been several works proposing heuristic training procedures to build networks that are robust against such perturbations [Cisse et al., 2017, Madry et al., 2018].

Original: However, many of these works provided a false sense of security as they were subsequently broken, i.e. shown to be ineffective against stronger adversaries [Athalye et al., 2018, Tramer et al., 2020, Uesato et al., 2018].
Revised: However, many of these works provided a false sense of security as they were subsequently broken, i.e. shown to be ineffective against stronger adversaries [Athalye et al., 2018, Tramer et al., 2020, Uesato et al., 2018].

Original: This has inspired researchers to develop networks that are certiﬁably robust , i.e. networks that provably output constant predictions over a characterized region around every input.
Revised: This has inspired researchers to develop networks that are certifiably robust , i.e. networks that provably output constant predictions over a characterized region around every input.

Original: Among many certiﬁcation methods, a probabilistic approach to certiﬁcation called randomized smoothing has demonstrated impressive state-of-the-art certiﬁable robustness results [Cohen et al., 2019, Lecuyer et al., 2019, Li et al., 2019].
Revised: Among many certification methods, a probabilistic approach to certification called randomized smoothing has demonstrated impressive state-of-the-art certifiable robustness results [Cohen et al., 2019, Lecuyer et al., 2019, Li et al., 2019].

Original: In a nutshell, given an input x and a base classiﬁer f , e.g. a DNN, randomized smoothing constructs a “smooth classiﬁer” g ( x ) =
Revised: In a nutshell, given an input x and a base classifier f , e.g. a DNN, randomized smoothing constructs a “smooth classifier” g ( x ) =

Original: E (cid:15) ∼D [ f ( x + (cid:15) )] such that, and under some choices of D , g ( x ) 
Revised: E ϵ ∼D [ f ( x + ϵ )] such that, and under some choices of D , g ( x ) =

Original: =

Original: g ( x + δ ) ∀ δ ∈ R .
Revised: g ( x + δ ) ∀ δ ∈ R .

Original: As such, g is certiﬁable within the certiﬁcation region R characterized by x and the smoothing distribution D .
Revised: As such, g is certifiable within the certification region R characterized by x and the smoothing distribution D .

Original: While there has been considerable progress in devising a notion of “optimal” smoothing distribution D for when R is characterized by an (cid:96) p certiﬁcate [Yang et al., 2020], a common trait among all works in the literature is that the choice of D is independent from the input x .
Revised: While there has been considerable progress in devising a notion of “optimal” smoothing distribution D for when R is characterized by an ℓ p certificate [Yang et al., 2020], a common trait among all works in the literature is that the choice of D is independent from the input x .

Original: For example, one of the earliest works on randomized smoothing grants (cid:96) 2 certiﬁcates under
Revised: For example, one of the earliest works on randomized smoothing grants ℓ 2 certificates under

Original: D = N (0 , σ 2 I ) , where σ is a free parameter that is constant for all x [Cohen et al., 2019].
Revised: D = N (0 , σ 2 I ) , where σ is a free parameter that is constant for all x [Cohen et al., 2019].

Original: That is to say, the classiﬁer f is smoothed to a classiﬁer g uniformly (same variance σ 2 ) over the entire input space of x .
Revised: That is to say, the classifier f is smoothed to a classifier g uniformly (same variance σ 2 ) over the entire input space of x .

Original: The choice of σ used for certiﬁcation is often set either arbitrarily or via cross validation to obtain best certiﬁcation results [Salman et al., 2019a].
Revised: The choice of σ used for certification is often set either arbitrarily or via cross validation to obtain best certification results [Salman et al., 2019a].

Original: We believe this is suboptimal and that σ should vary with the input x (data dependent), since using a ﬁxed σ may under-certify inputs ( i.e. the constructed smooth classiﬁer g produces smaller certiﬁcation radii), which are far from the decision boundaries as exempliﬁed by x 1 in Figure 1.
Revised: We believe this is suboptimal and that σ should vary with the input x (data dependent), since using a fixed σ may under-certify inputs ( i.e. the constructed smooth classifier g produces smaller certification radii), which are far from the decision boundaries as exemplified by x 1 in Figure 1.

Original: Moreover, this ﬁxed σ could be large for inputs x close to the decision boundaries resulting in a smooth classiﬁer g that incorrectly classiﬁes x (refer to x 3 in Figure 1).
Revised: Moreover, this fixed σ could be large for inputs x close to the decision boundaries resulting in a smooth classifier g that incorrectly classifies x (refer to x 3 in Figure 1).

Original: In this paper, we aim to introduce more structure to the smoothing distribution D by rendering its parameters data dependent.
Revised: In this paper, we aim to introduce more structure to the smoothing distribution D by rendering its parameters data dependent.

Original: That is to say, the base classiﬁer f is smoothed with a family of smoothing distributions to produce: g ( x ) =
Revised: That is to say, the base classifier f is smoothed with a family of smoothing distributions to produce: g ( x ) =

Original: E (cid:15) ∼N (0 ,σ 2 x I )
Revised: E ϵ ∼N (0 ,σ 2 x I )

Original: [ f ( x + (cid:15) )] 1 .
Revised: [ f ( x + ϵ )] 2 .

Original: Note here that the variance of the Gaussian is now dependent on the data input x .
Revised: Note here that the variance of the Gaussian is now dependent on the data input x .

Original: Moreover, given that σ x varies with x , classical randomized smoothing based certiﬁcation does not apply directly.
Revised: Moreover, given that σ x varies with x , classical randomized smoothing based certification does not apply directly.

Original: We propose a simple memory-based approach to certify the resultant data dependent smooth classiﬁer g .
Revised: We propose a simple memory-based approach to certify the resultant data dependent smooth classifier g .

Original: We show that our memory-enhanced data dependent smooth classiﬁer can boost certiﬁcation performance of several randomized smoothing techniques.
Revised: We show that our memory-enhanced data dependent smooth classifier can boost certification performance of several randomized smoothing techniques.

Original: Our contributions can thus be summarized in three folds.
Revised: Our contributions can thus be summarized in three folds.

Original: ( i )
Revised: ( i )

Original: We propose a parameter free and generic framework that can easily turn several randomized smoothing techniques into their data dependent variants.
Revised: We propose a parameter free and generic framework that can easily turn several randomized smoothing techniques into their data dependent variants.

Original: In particular, given a network f and an input x , we propose to optimize the smoothing distribution parameters for every x , e.g. σ ∗ x , so they maximize the certiﬁcation radius.
Revised: In particular, given a network f and an input x , we propose to optimize the smoothing distribution parameters for every x , e.g. σ ∗ x , so they maximize the certification radius.

Original: This choice of σ ∗ x is then used to smooth f at x and construct a smoothed classiﬁer g .
Revised: This choice of σ ∗ x is then used to smooth f at x and construct a smoothed classifier g .

Original: Moreover, as the data dependent smooth classiﬁer is not directly certiﬁable using Cohen et al.
Revised: Moreover, as the data dependent smooth classifier is not directly certifiable using Cohen et al.

Original: [2019] MCMC approaches, we propose a memory-enhanced data dependent smooth classiﬁer for certiﬁcation.
Revised: [2019] MCMC approaches, we propose a memory-enhanced data dependent smooth classifier for certification.

Original: ( ii )
Revised: ( ii )

Original: We demonstrate the effectiveness of our memory-enhanced data dependent smoothing by showing that we can improve the certiﬁed accuracy of several models, speciﬁcally models trained with Gaussian augmentation (C OHEN )
Revised: We demonstrate the effectiveness of our memory-enhanced data dependent smoothing by showing that we can improve the certified accuracy of several models, specifically models trained with Gaussian augmentation (C OHEN )

Original: [Cohen et al., 2019], adversaries on the smoothed classiﬁer (
Revised: [Cohen et al., 2019], adversaries on the smoothed classifier (

Original: S MOOTH A DV )
Revised: S MOOTH A DV )

Original: [Salman et al., 2019a], and radius regularization (MACER) [Zhai et al., 2020] without any model retraining .
Revised: [Salman et al., 2019a], and radius regularization (MACER) [Zhai et al., 2020] without any model retraining .

Original: We boost the certiﬁed accuracy of the best baseline by 5.4% on CIFAR10 and by 2.8% on ImageNet for (cid:96) 2 perturbations with less than 0.5 (=127/255) ball radius.
Revised: We boost the certified accuracy of the best baseline by 5.4% on CIFAR10 and by 2.8% on ImageNet for ℓ 2 perturbations with less than 0.5 (=127/255) ball radius.

Original: ( iii )
Revised: ( iii )

Original: We show that incorporating the proposed data dependent smoothing in the training pipeline of C OHEN ,
Revised: We show that incorporating the proposed data dependent smoothing in the training pipeline of C OHEN ,

Original: S MOOTH A DV and MACER can further boost results to get certiﬁed accuracies of 68.3% on CIFAR10 and 64.2% on
Revised: S MOOTH A DV and MACER can further boost results to get certified accuracies of 68.3% on CIFAR10 and 64.2% on

Original: ImageNet at (cid:96) 2 perturbations less than 0.25.
Revised: ImageNet at ℓ 2 perturbations less than 0.25.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Certiﬁed Defenses.
Revised: Certified Defenses.

Original: Certiﬁed defenses aim to guarantee that an adversary does not exist in a certain region around a given input.
Revised: Certified defenses aim to guarantee that an adversary does not exist in a certain region around a given input.

Original: Certiﬁed defenses can be divided into exact [Cheng et al., 2017, Lomuscio and Maganti, 2017, Huang et al., 2017, Ehlers, 2017] and relaxed certiﬁcation [Salman et al., 2019b, Wong and Kolter, 2018].
Revised: Certified defenses can be divided into exact [Cheng et al., 2017, Lomuscio and Maganti, 2017, Huang et al., 2017, Ehlers, 2017] and relaxed certification [Salman et al., 2019b, Wong and Kolter, 2018].

Original: Generally, exact certiﬁcation suffers from poor scalability with networks that are at most 3 hidden layers deep [Tjeng et al., 2019].
Revised: Generally, exact certification suffers from poor scalability with networks that are at most 3 hidden layers deep [Tjeng et al., 2019].

Original: On the other hand, relaxed methods resolve this issue by aiming at ﬁnding an upper bound to the worst adversarial loss over all possible bounded perturbations around a given input [Weng et al., 2018].
Revised: On the other hand, relaxed methods resolve this issue by aiming at finding an upper bound to the worst adversarial loss over all possible bounded perturbations around a given input [Weng et al., 2018].

Original: However, the latter is too expensive for any mixed certiﬁcation-training routine.
Revised: However, the latter is too expensive for any mixed certification-training routine.

Original: Randomized Smoothing.
Revised: Randomized Smoothing.

Original: The earliest work on randomized smoothing [Lecuyer et al., 2019] was from a differential privacy perspective, where it was demonstrated that adding
Revised: The earliest work on randomized smoothing [Lecuyer et al., 2019] was from a differential privacy perspective, where it was demonstrated that adding

Original: Laplacian noise enjoys an (cid:96) 1 certiﬁcation radius in which the average classiﬁer prediction under this noise is constant.
Revised: Laplacian noise enjoys an ℓ 1 certification radius in which the average classifier prediction under this noise is constant.

Original: This work was later followed by the tight (cid:96) 2 certiﬁcate radius for Gaussian smoothing [Cohen et al., 2019].
Revised: This work was later followed by the tight ℓ 2 certificate radius for Gaussian smoothing [Cohen et al., 2019].

Original: Since then, there has been a body of work on randomized smoothing with empirical defenses [Salman et al., 2019a] to certify black box classiﬁers [Salman et al., 2020].
Revised: Since then, there has been a body of work on randomized smoothing with empirical defenses [Salman et al., 2019a] to certify black box classifiers [Salman et al., 2020].

Original: Other works derived certiﬁcation guarantees for (cid:96) 1 bounded
Revised: Other works derived certification guarantees for ℓ 1 bounded

Original: [Teng et al., 2019], (cid:96) ∞ bounded [Zhang et al., 2019], and (cid:96) 0 bounded [Levine and Feizi, 2020] perturbations.
Revised: [Teng et al., 2019], ℓ ∞ bounded [Zhang et al., 2019], and ℓ 0 bounded [Levine and Feizi, 2020] perturbations.

Original: Even more recently, a novel framework that ﬁnds the optimal smoothing distribution for a given (cid:96) p norm [Yang et al., 2020] was proposed showing state-of-art certiﬁcation results on (cid:96) 1 perturbations.
Revised: Even more recently, a novel framework that finds the optimal smoothing distribution for a given ℓ p norm [Yang et al., 2020] was proposed showing state-of-art certification results on ℓ 1 perturbations.

Original: We deviate from the common literature by introducing the notion of smoothing, particularly Gaussian smoothing for (cid:96)perturbations, which varies depending on the input.
Revised: We deviate from the common literature by introducing the notion of smoothing, particularly Gaussian smoothing for ℓperturbations, which varies depending on the input.

Original: In particular, since an input x that is far from the decision boundaries should tolerate larger smoothing (and equivalently have a larger certiﬁcation radius) as compared to inputs closer to these boundaries, we optimize for the amount of smoothing per input (speciﬁcally σ x ) that maximizes the certiﬁcation radius.
Revised: In particular, since an input x that is far from the decision boundaries should tolerate larger smoothing (and equivalently have a larger certification radius) as compared to inputs closer to these boundaries, we optimize for the amount of smoothing per input (specifically σ x ) that maximizes the certification radius.

Original: This proposed process is denoted as data dependent smoothing where we provide a procedure for certifying the resultant smooth classiﬁer.
Revised: This proposed process is denoted as data dependent smoothing where we provide a procedure for certifying the resultant smooth classifier.

Original: DATA DEPENDENT SMOOTHING 3.1 PRELIMINARIES AND NOTATIONS
Revised: 3.1 PRELIMINARIES AND NOTATIONS

Original: Let x ∈ R d and the labels y ∈ Y = { 1 , . .
Revised: Let x ∈ R d and the labels y ∈ Y = { 1 , . .

Original: , k } be the input-label pairs ( x, y ) sampled from an unknown data distribution.
Revised: , k } be the input-label pairs ( x, y ) sampled from an unknown data distribution.

Original: Unless explicitly mentioned, we consider a classiﬁer f θ : R d → P ( Y ) parameterized by θ where
Revised: Unless explicitly mentioned, we consider a classifier f θ : R d → P ( Y ) parameterized by θ where

Original: P ( Y ) is a probability simplex over k labels.
Revised: P ( Y ) is a probability simplex over k labels.

Original: We say that f θ is (cid:96) rp certiﬁably accurate for an input x , if and only if,
Revised: We say that f θ is ℓ rp certifiably accurate for an input x , if and only if,

Original: [Equation]
Revised: [Equation]

Original: where f cθ is the c th element of f θ .
Revised: where f cθ is the c th element of f θ .

Original: That is to say, the classiﬁer correctly predicts the label of x and enjoys a constant prediction for all perturbations δ that are in the (cid:96) p ball of radius r from x .
Revised: That is to say, the classifier correctly predicts the label of x and enjoys a constant prediction for all perturbations δ that are in the ℓ p ball of radius r from x .

Original: As such, the overall (cid:96) rp certiﬁcation accuracy is deﬁned as the average certiﬁed accuracy over the data distribution.
Revised: As such, the overall ℓ rp certification accuracy is defined as the average certified accuracy over the data distribution.

Original: Following prior art [Cohen et al., 2019, Salman et al., 2019a, Zhai et al., 2020], we focus on (cid:96) r 2 certiﬁcation.
Revised: Following prior art [Cohen et al., 2019, Salman et al., 2019a, Zhai et al., 2020], we focus on ℓ r 2 certification.

Original: 3.2 OVERVIEW OF RANDOMIZED SMOOTHING
Revised: 3.2 OVERVIEW OF RANDOMIZED SMOOTHING

Original: Randomized smoothing constructs a certiﬁable classiﬁer g θ by smoothing a base classiﬁer f θ .
Revised: Randomized smoothing constructs a certifiable classifier g θ by smoothing a base classifier f θ .

Original: For any σ > 0 , the smooth classiﬁer is deﬁned as: g θ ( x ) =
Revised: For any σ > 0 , the smooth classifier is defined as: g θ ( x ) =

Original: E (cid:15) ∼N (0 ,σ 2 I )
Revised: E ϵ ∼N (0 ,σ 2 I )

Original: [ f θ ( x + (cid:15) )] .
Revised: [ f θ ( x + ϵ )] .

Original: Let g θ predict label c A for input x with some conﬁdence, i.e. E 
Revised: Let g θ predict label c A for input x with some confidence, i.e. E ϵ

Original: [Equation]
Revised: [ f c A θ ( x + ϵ )] = p

Original: [Equation]
Revised: [Equation]

Original: =
Revised: ̸

Original: [Equation]
Revised: [Equation]

Original: ] , then, g θ is certiﬁably robust at x with certiﬁcation radius:
Revised: E ϵ [ f cθ ( x + ϵ )] , then, g θ is certifiably robust at x with certification radius:

Original: [Equation]
Revised: [Equation]

Original: Here, g ( x + δ ) =
Revised: Here, g ( x + δ ) =

Original: g ( x ) ∀(cid:107) δ (cid:107) 2 ≤ R , where Φ is the CDF of the standard Gaussian.
Revised: g ( x ) ∀∥ δ ∥ 2 ≤ R , where Φ is the CDF of the standard Gaussian.

Original: 3.3 ROBUSTNESS-ACCURACY TRADE-OFF
Revised: 3.3 ROBUSTNESS-ACCURACY TRADE

Revised: -OFF

Original: Note that Equation 1 holds regardless of the prediction c
Revised: Note that Equation 1 holds regardless of the prediction c

Original: A made by the smooth classiﬁer g θ .
Revised: A made by the smooth classifier g θ .

Original: This suggests that one can perhaps improve the robustness of g θ , i.e. increase certiﬁcation radius R where g θ is constant, by increasing the hyper parameter σ in Equation 1.
Revised: This suggests that one can perhaps improve the robustness of g θ , i.e. increase certification radius R where g θ is constant, by increasing the hyper parameter σ in Equation 1.

Original: However, to reason about (cid:96) r 2 certiﬁcation accuracy, it is not enough to increase the certiﬁcation radius R , as this requires c A to be the correct prediction for x by g θ .
Revised: However, to reason about ℓ r 2 certification accuracy, it is not enough to increase the certification radius R , as this requires c A to be the correct prediction for x by g θ .

Original: This reveals the robustnessaccuracy trade-off as one cannot improve (cid:96) r 2 certiﬁed accuracy by only increasing the certiﬁcation radius R (robustness) through the increase in σ .
Revised: This reveals the robustnessaccuracy trade-off as one cannot improve ℓ r 2 certified accuracy by only increasing the certification radius R (robustness) through the increase in σ .

Original: This is because it comes at the expense of requiring a classiﬁer g θ that correctly classiﬁes x with correct label y under large Gaussian perturbations (accuracy).
Revised: This is because it comes at the expense of requiring a classifier g θ that correctly classifies x with correct label y under large Gaussian perturbations (accuracy).

Original: As such, the following inequality should hold E(cid:15)
Revised: As such, the following inequality should hold E

Original: [ f yθ ( x + (cid:15) )] ≥ p A ≥ p B ≥ max c (cid:54) = y E (cid:15) [ f c ( x + (cid:15) )] .
Revised: [Equation]

Original: The certiﬁcation region R = { δ : (cid:107) δ (cid:107) 2 ≤ R } at an input x is fully characterized by the classiﬁer f θ and the standard deviation of the Gaussian distribution σ . Moreover, for a given f θ , the certiﬁcation region R varies at different x , when σ is ﬁxed, due to the nonlinear dependence of the prediction gap Φ − 1 ( p A ( x ; σ )) − Φ − 1 ( p B ( x ; σ )) on x . This hints that, for a given f θ , different inputs x may enjoy a
Revised: Algorithm

Original: Function OptimizeSigma
Revised: Function OptimizeSigma

Original: ( f θ , x , α , σ 0 , n ) :
Revised: ( f θ , x , α , σ 0 , n ) :

Original: Initialize: σ 0 x ← σ 0 , K for k = 0 . . .
Revised: Initialize: σ 0 x ← σ 0 , K for k = 0 . . .

Original: K − 1 do sample ˆ (cid:15) 1 , . . .
Revised: K − 1 do sample ˆ ϵ 1 , . . .

Original: ˆ (cid:15) n ∼ N (0 , I )ψ
Revised: ˆ ϵ n ∼ N (0 , I )

Original: ( σ kx ) = 1 n (cid:80) ni =1 f θ ( x + σ kx ˆ (cid:15) i )
Revised: [Equation]

Original: E A ( σ kx ) =
Revised: return σ ∗ x

Original: max c ψ c ; y A = arg max c ψ c ; E B ( σ kx ) = max c (cid:54) = y
Revised: The certification region R = { δ : ∥ δ ∥ 2 ≤ R } at an input x is fully characterized by the classifier f θ and the standard deviation of the Gaussian distribution σ .

Original: A ψ c R ( σ kx ) = σ kx 2 (cid:0) Φ − 1 ( E A ) − Φ − 1 ( E B) (cid:1) σ k +1 x ← σ kx + α ∇ σ kx R (σ kx ) σ ∗ x ← σ Kx return σ ∗ x  different optimal σ ∗ x that maximizes the certiﬁcation region.
Revised: Moreover, for a given f θ , the certification region R varies at different x , when σ is fixed, due to the nonlinear dependence of the prediction gap Φ − 1 ( p A ( x ; σ )) − Φ − 1 ( p B ( x ; σ )) on x . This hints that, for a given f θ , different inputs x may enjoy a different optimal σ ∗ x that maximizes the certification region.

Original: To see this, consider the three inputs x 1 , x 2 and x 3 all correctly classiﬁed by the binary classiﬁer f θ as C 1 in Figure 1.
Revised: To see this, consider the three inputs x 1 , x 2 and x 3 all correctly classified by the binary classifier f θ as C 1 in Figure 1.

Original: Using a ﬁxed σ to smooth the predictions of f θ , i.e. predict with g θ , reveals that inputs, depending on how close they are from the decision boundaries, can enjoy different levels of smoothing without affecting the prediction of g θ .
Revised: Using a fixed σ to smooth the predictions of f θ , i.e. predict with g θ , reveals that inputs, depending on how close they are from the decision boundaries, can enjoy different levels of smoothing without affecting the prediction of g θ .

Original: For instance, as shown in Figure 1 for constant σ , the input far from the decision boundary x 1 could have still been classiﬁed correctly with similarly large prediction gap even if f θ were to be smoothed with a larger σ .
Revised: For instance, as shown in Figure 1 for constant σ , the input far from the decision boundary x 1 could have still been classified correctly with similarly large prediction gap even if f θ were to be smoothed with a larger σ .

Original: This indicates that perhaps the certiﬁcation radius at x 1 could have been enlarged with a larger smoothing σ .
Revised: This indicates that perhaps the certification radius at x 1 could have been enlarged with a larger smoothing σ .

Original: As for x 2 , we can observe that while the prediction under this choice of σ by g θ is still correct, the prediction gap Φ − 1 ( p A ( x ; σ )) − Φ − 1 ( p B ( x ; σ )) drops, due to having more Gaussian samples fall in the Cregion.
Revised: As for x 2 , we can observe that while the prediction under this choice of σ by g θ is still correct, the prediction gap Φ − 1 ( p A ( x ; σ )) − Φ − 1 ( p B ( x ; σ )) drops, due to having more Gaussian samples fall in the Cregion.

Original: Thus, a different choice of σ could have been used to trade-off the drop in prediction gap and certiﬁcation radius.
Revised: Thus, a different choice of σ could have been used to trade-off the drop in prediction gap and certification radius.

Original: Last, for the input x 3 that is very close to the decision boundary, the sub optimal choice of σ (too large for x 3 ) could result in an incorrect prediction by g θ .
Revised: Last, for the input x 3 that is very close to the decision boundary, the sub optimal choice of σ (too large for x 3 ) could result in an incorrect prediction by g θ .

Original: Despite the observations that σ plays a signiﬁcant role in (cid:96) r 2 certiﬁcation accuracy, certiﬁcation methods generally ( i ) choose σ arbitrarily and ( ii ) set it to be constant for all x .
Revised: Despite the observations that σ plays a significant role in ℓ r 2 certification accuracy, certification methods generally ( i ) choose σ arbitrarily and ( ii ) set it to be constant for all x .

Original: Based on this observation, for a given smooth classiﬁer with a speciﬁc σ 0 , where σ 0 can be zero reducing the smooth classiﬁer to f θ , we seek to construct another smooth classiﬁer with parameter σ ∗ x for every input x such that: ( i ) the prediction of both smooth classiﬁers (smoothing with σ 0 and σ ∗ x ) is identical for all x .
Revised: Based on this observation, for a given smooth classifier with a specific σ 0 , where σ 0 can be zero reducing the smooth classifier to f θ , we seek to construct another smooth classifier with parameter σ ∗ x for every input x such that: ( i ) the prediction of both smooth classifiers (smoothing with σ 0 and σ ∗ x ) is identical for all x .

Original: ( ii )
Revised: ( ii )

Original: The certiﬁcation radius of the new smooth classiﬁer at every x is maximized.
Revised: The certification radius of the new smooth classifier at every x is maximized.

Original: To construct a classiﬁer smoothed with σ ∗ x enjoying the two previous properties, let c A be the prediction under σ 0 smoothing, i.e. 
Revised: To construct a classifier smoothed with σ ∗ x enjoying the two previous properties, let c A be the prediction under σ 0 smoothing, i.e. c A = arg max c E ϵ ∼N (0 ,σ 0 I )

Original: [Equation] [Equation]
Revised: [ f c ( x + ϵ )] .

Original: We maximize R in Equation 1 over σ for every x by solving:
Revised: We maximize R in Equation 1 over σ for every x by solving:

Original: [Equation]
Revised: [Equation]

Original: Since Φ − 1 is a strictly increasing function, it is important to note that solving Equation 2 for a ﬁxed c A can at worst yield a smooth classiﬁer of an identical radius to when the classiﬁer is smoothed with σ 0 both predicting c A for x .
Revised: Since Φ − 1 is a strictly increasing function, it is important to note that solving Equation 2 for a fixed c A can at worst yield a smooth classifier of an identical radius to when the classifier is smoothed with σ 0 both predicting c A for x .

Original: Solver .
Revised: Solver .

Original: While our proposed Objective 2 has a similar form to the MACER regularizer [Zhai et al., 2020] used during training, ours differs in that we optimize σ for every x and not the network parameters θ , which are ﬁxed here.
Revised: While our proposed Objective 2 has a similar form to the MACER regularizer [Zhai et al., 2020] used during training, ours differs in that we optimize σ for every x and not the network parameters θ , which are fixed here.

Original: A natural solver for 2 is stochastic gradient ascent with the expectation approximated with n Monte
Revised: A natural solver for 2 is stochastic gradient ascent with the expectation approximated with n Monte

Original: Carlo samples.
Revised: Carlo samples.

Original: As such, the gradient of the objective at the k th iteration will be approximated as follows:
Revised: As such, the gradient of the objective at the k th iteration will be approximated as follows:

Original: [Equation]
Revised: [Equation]

Original: where γ c ( σ k ) = 1 n (cid:80) ni =1 f c ( x + (cid:15) i ) for (cid:15) 1 , . . .
Revised: where γ c ( σ k ) = 1 n (cid:80) ni =1 f c ( x + ϵ i ) for ϵ 1 , . . .

Original: , (cid:15) n ∼
Revised: , ϵ n ∼

Original: N (0 , ( σ k ) 2 I ) .
Revised: N (0 , ( σ k ) 2 I ) .

Original: However, this estimation of the gradient suffers from high variance due to the dependence of the expectation on the optimization variable σ that parameterizes the smoothing distribution N (0 , σ 2 I )
Revised: However, this estimation of the gradient suffers from high variance due to the dependence of the expectation on the optimization variable σ that parameterizes the smoothing distribution N (0 , σ 2 I )

Original: [Williams, 1992].
Revised: [Williams, 1992].

Original: To alleviate this, we use the reparameterization trick suggested by Kingma and Welling [2014], Rezende et al.
Revised: To alleviate this, we use the reparameterization trick suggested by Kingma and Welling [2014], Rezende et al.

Original: [2014] to compute a lower variance gradient estimate for our Objective 2.
Revised: [2014] to compute a lower variance gradient estimate for our Objective 2.

Original: In particular, with the change of variable (cid:15) =
Revised: In particular, with the change of variable ϵ =

Original: σ ˆ (cid:15) where ˆ (cid:15) ∼ N (0 , I ) , Objective 2 isnow equivalent to :
Revised: σ ˆ ϵ where ˆ ϵ ∼ N (0 , I ) , Objective 2 is equivalent to :

Original: [Equation]
Revised: [Equation]

Original: Note that, unlike before, the expectation over the distribution ˆ (cid:15) ∼ N (0 , I ) no longer depends on the optimization variable σ .
Revised: Note that, unlike before, the expectation over the distribution ˆ ϵ ∼ N (0 , I ) no longer depends on the optimization variable σ .

Original: This allows the gradient of 3 to enjoy a lower variance compared to the gradient of 2 [Kingma and Welling, 2014,
Revised: This allows the gradient of 3 to enjoy a lower variance compared to the gradient of 2 [Kingma and Welling, 2014,

Original: Rezende et al., 2014].
Revised: Rezende et al., 2014].

Original: Algorithm 1 summarizes the updates for optimizing σ for each x by solving 3 with K steps of stochastic gradient ascent.
Revised: Algorithm 1 summarizes the updates for optimizing σ for each x by solving 3 with K steps of stochastic gradient ascent.

Original: It is worthwhile to mention that the function OptimizeSigma in Algorithm 1 is agnostic of the choice of architecture f θ and of the training procedure that constructed f θ .
Revised: It is worthwhile to mention that the function OptimizeSigma in Algorithm 1 is agnostic of the choice of architecture f θ and of the training procedure that constructed f θ .

Original: 3.5 MEMORY-BASED CERTIFICATION FOR DATA DEPENDENT CLASSIFIERS

Original: Unlike previous approaches where σ is constant for all inputs, the data dependent classiﬁer g θ with varying σ per input can not be directly certiﬁed by the classical Monte Carlo algorithms proposed by Cohen et al. This is since the data dependent classiﬁer g θ does not enjoy a constant σ within the given certiﬁcation region, i.e. g θ tailors a new σ x for every input x including within the certiﬁed region of x . Informally, let R ( σ∗ x 1 ) be the radius of certiﬁcation at x 1 granted by the data dependent classiﬁer g θ .
Revised: TrainBatch( f θ , { x i , y i } Bi =1 , { σ x i } Bi =1 , α , n ) : for i = 1 , . . .

Original: The data dependent classiﬁer does not guarantee that there TrainBatch( f θ { x y } Bi
Revised: , B do σ ∗

Original: = σ x i} =
Revised: [Equation] i =

Original: B do σ ∗ x i =

Original: OptimizeSigma
Revised: OptimizeSigma

Original: ( f θ , x i , α, σ x i , n )
Revised: ( f [Equation] , x i , α, σ x i , n )

Original: TrainFunction (cid:0) { x i , y i } Bi =1 , { σ ∗ x i } Bi
Revised: TrainFunction (cid:0) { x i , y i } Bi =1 , { σ ∗ x i } Bi

Original: =1 (cid:1) //
Revised: =1 (cid:1) //

Original: any 
Revised: any training routine e.g. MACER 3.5 MEMORY-BASED CERTIFICATION FOR DATA DEPENDENT CLASSIFIERS Unlike previous approaches where σ is constant for all inputs, the data dependent classifier g θ with varying σ per input can not be directly certified by the classical Monte Carlo algorithms proposed by Cohen et al.

Original: [Table]
Revised: This is since the data dependent classifier g θ does not enjoy a constant σ within the given certification region, i.e. g θ tailors a new σ x for every input x including within the certified region of x . Informally, let R ( σ ∗ x 1 ) be the radius of certification at x 1 granted by the data dependent classifier g θ .

Original:  can not exist x 2 within the region of certiﬁcation of x 1 , i.e. (cid:107) x 1 − x 2 (cid:107) 2 ≤ R ( σ ∗ x 1 ) , where g θ with σ ∗ x 2 predicts xdifferently from x 1 breaking the soundness of certiﬁcation.
Revised: The data dependent classifier does not guarantee that there can not exist x 2 within the region of certification of x 1 , i.e. ∥ x 1 − x 2 ∥ 2 ≤ R ( σ ∗ x 1 ) , where g θ with σ ∗ x 2 predicts xdifferently from x 1 breaking the soundness of certification.

Original: To circumvent this problem, we propose a memory-based procedure to certifying our proposed data dependent classiﬁer.
Revised: To circumvent this problem, we propose a memory-based procedure to certifying our proposed data dependent classifier.

Original: Let { x i } Ni =1 be a set of previously predicted inputs and {C i } Ni =1 be their corresponding predictions with mutually exclusive (cid:96) 2 certiﬁed regions R i for differently predicted inputs, i.e. R i ∩ R j = ∅ ∀
Revised: Let { x i } Ni =1 be a set of previously predicted inputs and {C i } Ni =1 be their corresponding predictions with mutually exclusive ℓ 2 certified regions R i for differently predicted inputs, i.e. R i ∩ R j = ∅ ∀

Original: i (cid:54) =
Revised: i ̸

Original:  j, C i (cid:54) =
Revised: = j, C i ̸ =

Original: C j .
Revised: C j .

Original: Let x N +1 be a new input with a certiﬁed region R N +1 computed by the Monte
Revised: Let x N +1 be a new input with a certified region R N +1 computed by the Monte

Original: Carlo algorithms of Cohen et al.
Revised: Carlo algorithms of Cohen et al.

Original: [2019] for the data dependent classiﬁer g θ with prediction C N +1 .
Revised: [2019] for the data dependent classifier g θ with prediction C N +1 .

Original: If there exists an i such that R N +1 ∩ R i (cid:54) = ∅ , x N +1 ∈ R i , and C N +1 (cid:54) = C i , we adjust the prediction of the data dependent classiﬁer g θ to be C i and update R N +1 to be the largest subset of
Revised: If there exists an i such that R N +1 ∩ R i ̸ = ∅ , x N +1 ∈ R i , and C N +1 ̸ = C i , we adjust the prediction of the data dependent classifier g θ to be C i and update R N +1 to be the largest subset of

Original: R N +1 that is a subset of R
Revised: R N +1 that is a subset of R

Original: i
Revised: i

Original: (see middle example in Figure 2).
Revised: (see middle example in Figure 2).

Original: On the other hand, if R N +1 ∩ R i (cid:54) = ∅ , x N +1 / ∈ R i , and
Revised: On the other hand, if R N +1 ∩ R i ̸ = ∅ , x N +1 / ∈ R i , and

Original: that C N [-+
-]
Revised: that C N [-+
-]+1 ̸ = C

Original: (cid:54) =

Original: C i , we update R N +1 to be the largest subset of R N +1 not intersecting with R
Revised: i , we update R N +1 to be the largest subset of R N +1 not intersecting with R

Original: i (see right example in Figure 2).
Revised: i (see right example in Figure 2).

Original: We perform the previous operations for all elements in the memory and add x N +1 , C N +1 , R N + to memory.
Revised: We perform the previous operations for all elements in the memory and add x N +1 , C N +1 , R N + to memory.

Original: The aforementioned procedure grants a sound certiﬁcation for the data dependent classiﬁer preventing by construction overlapping certiﬁed regions with different predictions.
Revised: The aforementioned procedure grants a sound certification for the data dependent classifier preventing by construction overlapping certified regions with different predictions.

Original: While the memory-based certiﬁcation is essential for a sound certiﬁcation, empirically, we never found in any of the later experiments a case where two inputs predicted differently suffer from intersecting certiﬁed regions.
Revised: While the memory-based certification is essential for a sound certification, empirically, we never found in any of the later experiments a case where two inputs predicted differently suffer from intersecting certified regions.

Original: That is to say while our sound certiﬁcate works on the memoryenhanced data dependent smooth classiﬁer, we found that the certiﬁed radius of the memory classiﬁer for every input is the radius granted by the Monte Carlo certiﬁcates of Cohen et al.
Revised: That is to say while our sound certificate works on the memoryenhanced data dependent smooth classifier, we found that the certified radius of the memory classifier for every input is the radius granted by the Monte Carlo certificates of Cohen et al.

Original: [2019] for the data dependent classiﬁer.
Revised: [2019] for the data dependent classifier.

Original: Therefore and throughout, we refer to the memory-enhanced data dependent smooth classiﬁer and data dependent smooth classiﬁer interchangeably.
Revised: Therefore and throughout, we refer to the memory-enhanced data dependent smooth classifier and data dependent smooth classifier interchangeably.

Original: We elaborate more on this and provide an algorithm in the Appendix .
Revised: We elaborate more on this and provide an algorithm in the Appendix .

Original: 3.6 TRAINING WITH DATA DEPENDENT SMOOTHING
Revised: 3.6 TRAINING WITH DATA DEPENDENT SMOOTHING

Original: Models that enjoy a large (cid:96) r 2 certiﬁcation accuracy under the randomized smoothing framework need to enjoy a large certiﬁcation radius R in Equation 1 for all x and be able to correctly classify inputs corrupted with Gaussian noise, i.e. g θ ( x ) = y .
Revised: Models that enjoy a large ℓ r 2 certification accuracy under the randomized smoothing framework need to enjoy a large certification radius R in Equation 1 for all x and be able to correctly classify inputs corrupted with Gaussian noise, i.e. g θ ( x ) = y .

Original: While there are several approaches to train f θ (or directly g θ ) so as to output correct predictions for inputs corrupted with noise sampled from N (0 , σ 2 I ) , all existing works ﬁx σ for all inputs during training.
Revised: While there are several approaches to train f θ (or directly g θ ) so as to output correct predictions for inputs corrupted with noise sampled from N (0 , σ 2 I ) , all existing works fix σ for all inputs during training.

Original: We are interested in complementing these approaches with smoothing distributions that are data dependent.
Revised: We are interested in complementing these approaches with smoothing distributions that are data dependent.

Original: As such, we can employ the training procedure of these approaches but with σ ∗ x computed by OptimizeSigma .
Revised: As such, we can employ the training procedure of these approaches but with σ ∗ x computed by OptimizeSigma .

Original: Algorithmsummarizes this proposed training pipeline.
Revised: Algorithmsummarizes this proposed training pipeline.

Original: The function
Revised: The function

Original: TrainFunction proceeds by performing backpropagation using any training scheme, given the estimated σ ∗ x i for each x i .
Revised: TrainFunction proceeds by performing backpropagation using any training scheme, given the estimated σ ∗ x i for each x i .

Original: We note that whenever Algorithm 2 is used, we initialize σ x i at each epoch with σ ∗ x i computed at the previous epoch.
Revised: We note that whenever Algorithm 2 is used, we initialize σ x i at each epoch with σ ∗ x i computed at the previous epoch.

Original: Since C OHEN , S MOOTH A DV and MACER are among the most popular approaches that embed randomized smoothing certiﬁcates as part of the training routine,
Revised: Since C OHEN , S MOOTH A DV and MACER are among the most popular approaches that embed randomized smoothing certificates as part of the training routine,

Original: TrainFunction refers here to any of these three training methods.
Revised: TrainFunction refers here to any of these three training methods.

Original: Empirically, we show that we can boost all three methods even further when models are trained with
Revised: Empirically, we show that we can boost all three methods even further when models are trained with

Original: Algorithm 2.
Revised: Algorithm 2.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: We conduct two sets of experiments to validate our key contributions.
Revised: We conduct two sets of experiments to validate our key contributions.

Original: ( i )
Revised: ( i )

Original: We show that we can boost certiﬁed accuracy for several pre-trained models by using Algorithm 1 for data dependent smoothing only during certiﬁcation, i.e. without employing any additional training.
Revised: We show that we can boost certified accuracy for several pre-trained models by using Algorithm 1 for data dependent smoothing only during certification, i.e. without employing any additional training.

Original: ( ii ) Once data dependent smoothing is employed during training, we can improve the certiﬁed accuracy even further.
Revised: ( ii ) Once data dependent smoothing is employed during training, we can improve the certified accuracy even further.

Original: Since our framework is agnostic to the training routine, we incorporate it into ( i )
Revised: Since our framework is agnostic to the training routine, we incorporate it into ( i )

Original: C OHEN
Revised: C OHEN

Original: [Cohen et al., 2019], ( ii ) S MOOTH A DV [Salman et al., 2019a] and ( iii ) MACER
Revised: [Cohen et al., 2019], ( ii ) S MOOTH A DV [Salman et al., 2019a] and ( iii ) MACER

Original: [Zhai et al., 2020].
Revised: [Zhai et al., 2020].

Original: Throughout, we use DS to refer to when data dependent smoothing is used only in certiﬁcation and DS 2 when it is used during both training and certiﬁcation.
Revised: Throughout, we use DS to refer to when data dependent smoothing is used only in certification and DS 2 when it is used during both training and certification.

Original: Setup.
Revised: Setup.

Original: We conduct experiments with ResNet-18 and ReNet50 [He et al., 2016] on CIFAR

Revised: We conduct experiments with ResNet-18 and ReNet50 [He et al., 2016] on CIFAR

Original: [Krizhevsky and Hinton, 2009] and ImageNet [Russakovsky et al., 2015], respectively.
Revised: [Krizhevsky and Hinton, 2009] and ImageNet [Russakovsky et al., 2015], respectively.

Original: For CIFAR10 experiments, we train from scratch for 200 epochs.
Revised: For CIFAR10 experiments, we train from scratch for 200 epochs.

Original: For ImageNet, we initialize using the network parameters provided by the authors.
Revised: For ImageNet, we initialize using the network parameters provided by the authors.

Original: When σ is ﬁxed and following prior art, e.g. C OHEN , S MOOTH A DV , and MACER, we set σ ∈ { 0 .  50 } and σ ∈ { 0 . 25 , 0 .  for CIFAR10 and ImageNet, respectively, for training and certiﬁcation.
Revised: When σ is fixed and following prior art, e.g. C OHEN , S MOOTH A DV , and MACER, we set σ ∈ { 0 . 12 , 0 . 25 , 0 . 50 } and σ ∈ { 0 . 25 , 0 . 50 , 1 . 0 } for CIFAR10 and ImageNet, respectively, for training and certification.

Original: We set α = 10 − 4 in Algorithm 1 and the initial σ 0 to the σ used in training the respective model.
Revised: We set α = 10 − 4 in Algorithm 1 and the initial σ 0 to the σ used in training the respective model.

Original: Unless stated otherwise, we set n =in Algorithm 1.
Revised: Unless stated otherwise, we set n =in Algorithm 1.

Original: Following C OHEN and S MOOTH A DV , we compare models using the approximate certiﬁed accuracy curve (simply referred to as certiﬁed accuracy) followed by the envelope curve over all σ .
Revised: Following C OHEN and S MOOTH A DV , we compare models using the approximate certified accuracy curve (simply referred to as certified accuracy) followed by the envelope curve over all σ .

Original: We also report the Average Certiﬁed Radius (ACR) proposed by MACER
Revised: We also report the Average Certified Radius (ACR) proposed by MACER

Original: [Equation]
Revised: [Equation]

Original: where 1 { . } is an indicator function.
Revised: where 1 { . } is an indicator function.

Original: Following C OHENand all randomized smoothing methods, we certify all results using N 0 = 100 Monte Carlo samples for prediction and N = 100 , 000 estimation samples to estimate the radius with a failure probability of 0 .
Revised: Following C OHEN and all randomized smoothing methods, we certify all results using N 0 = 100 Monte Carlo samples for prediction and N = 100 , 000 estimation samples to estimate the radius with a failure probability of 0 .

Original: 001 given a smoothing σ .
Revised: 001 given a smoothing σ .

Original: 4. COHEN + DS
Revised: 4. COHEN + DS

Original: We combine data dependent smoothing with C OHEN .
Revised: We combine data dependent smoothing with C OHEN .

Original: Following Gaussian augmentation, this method trains f θ on ( x + (cid:15) ) , where (cid:15) ∼ N (0 , σ 2 I ) , with the cross entropy loss. 
Revised: Following Gaussian augmentation, this method trains f θ on ( x + ϵ ) , where ϵ ∼ N (0 , σ 2 I ) , with the cross entropy loss. [Table]

Original: DS for certiﬁcation only.
Revised: DS for certification only.

Original: We ﬁrst certify the trained models with the same ﬁxed σ used in training for all inputs, dubbed C OHEN .
Revised: We first certify the trained models with the same fixed σ used in training for all inputs, dubbed C OHEN .

Original: Then, we certify using the memory based certiﬁcation the same trained models with the proposed data dependent σ ∗ x produced by Algorithm 1, which we refer to as C OHEN -DS.
Revised: Then, we certify using the memory based certification the same trained models with the proposed data dependent σ ∗ x produced by Algorithm 1, which we refer to as C OHEN -DS.

Original: Figure 3 plots the certiﬁed accuracy for
Revised: Figure 3 plots the certified accuracy for

Original: CIFAR10 and ImageNet in the ﬁrst and second rows, respectively.
Revised: CIFAR10 and ImageNet in the first and second rows, respectively.

Original: Even though the base classiﬁer f θ is identical for
Revised: Even though the base classifier f θ is identical for

Original: C OHEN and C OHEN -DS, Figure 3 shows that C OHEN -DS is superior to C OHEN in certiﬁed accuracy across almost all radii and for all training σ on both datasets.
Revised: C OHEN and C OHEN -DS, Figure 3 shows that C OHEN -DS is superior to C OHEN in certified accuracy across almost all radii and for all training σ on both datasets.

Original: This is also evident from the envelope plots in the last column of Figure 3.
Revised: This is also evident from the envelope plots in the last column of Figure 3.

Original: In Table 1, we report the best certiﬁed accuracy per radius over all training σ for C OHEN (envelope ﬁgure) against our best C OHEN -DS, cross-validated over all training σ and the number of iterations in Algorithm 1 K , accompanied with the corresponding ACR score.
Revised: In Table 1, we report the best certified accuracy per radius over all training σ for C OHEN (envelope figure) against our best C OHEN -DS, cross-validated over all training σ and the number of iterations in Algorithm 1 K , accompanied with the corresponding ACR score.

Original: For instance, we observe that data dependent certiﬁcation C OHEN -DS can signiﬁcantly [Table] boost certiﬁed accuracy at radii 0 .
Revised: For instance, we observe that data dependent certification C OHEN -DS can significantly boost certified accuracy at radii 0 .

Original: 5 and 0 .
Revised: 5 and 0 .

Original: 75 by 7 .
Revised: 75 by 7 .

Original: 7% (from 40 .
Revised: 7% (from 40 .

Original: 1 to 47 .
Revised: 1 to 47 .

Original: 8 ) and 9 .
Revised: 8 ) and 9 .

Original: 1% (from 29 .
Revised: 1% (from 29 .

Original: 2% to 38 .
Revised: 2% to 38 .

Original: 3% ), respectively, and by 0 .
Revised: 3% ), respectively, and by 0 .

Original: 193 ACR points on CIFAR10.
Revised: 193 ACR points on CIFAR10.

Original: Moreover, we boost the certiﬁed accuracy on ImageNet by 4 . 6% and 3 .
Revised: Moreover, we boost the certified accuracy on ImageNet by 4 . 6% and 3 .

Original: 2% at 0 .
Revised: 2% at 0 .

Original: 5 and 0 .
Revised: 5 and 0 .

Original: 75 radii, respectively, and by 0 .
Revised: 75 radii, respectively, and by 0 .

Original: 159 ACR points.
Revised: 159 ACR points.

Original: DS for training and certiﬁcation.
Revised: DS for training and certification.

Original: We employ data dependent smoothing in both training and certiﬁcation for C OHEN models (denoted as C OHEN -DS 2 ) by running Algorithm 2.
Revised: We employ data dependent smoothing in both training and certification for C OHEN models (denoted as C OHEN -DS 2 ) by running Algorithm 2.

Original: For CIFAR10, we train C OHEN ﬁrst with ﬁxed σ forepochs, i.e. K = 0 in Algorithm 1, and then we perform data dependent smoothing with K = 1 for the remainingepochs.
Revised: For CIFAR10, we train C OHEN first with fixed σ forepochs, i.e. K = 0 in Algorithm 1, and then we perform data dependent smoothing with K = 1 for the remainingepochs.

Original: For ImageNet experiments, we only ﬁnetune the provided models for 30 epochs using Algorithm 2 with K = 1 .
Revised: For ImageNet experiments, we only finetune the provided models for 30 epochs using Algorithm 2 with K = 1 .

Original: Once training is complete, we certify all trained models with
Revised: Once training is complete, we certify all trained models with

Original: Algorithm 1 using the memory based certiﬁcation.
Revised: Algorithm 1 using the memory based certification.

Original: In Figure 3, we observe that C OHEN -DS 2 can further improve certiﬁed accuracy across all trained models on both CIFAR and ImageNet.
Revised: In Figure 3, we observe that C OHEN -DS 2 can further improve certified accuracy across all trained models on both CIFAR and ImageNet.

Original: This is also evident in the last column of
Revised: This is also evident in the last column of

Original: Figure 3 that shows the best certiﬁed accuracy per radius  (envelope) over all training σ .
Revised: Figure 3 that shows the best certified accuracy per radius [Table] (envelope) over all training σ .

Original: We note that C OHEN -DSimproves the certiﬁcation accuracy of C OHEN -DS by 2 . 6% and by 0 .
Revised: We note that C OHEN -DSimproves the certification accuracy of C OHEN -DS by 2 . 6% and by 0 .

Original: 9% at radii 0 .
Revised: 9% at radii 0 .

Original: 5 and 0 .
Revised: 5 and 0 .

Original: 75 respectively on CIFAR10, and by 4 . 8% and 1 . 8% at radii 0 .
Revised: 75 respectively on CIFAR10, and by 4 . 8% and 1 . 8% at radii 0 .

Original: 5 and 0 .
Revised: 5 and 0 .

Original: 75 respectively on ImageNet.
Revised: 75 respectively on ImageNet.

Original: The improvements are consistently present over a wide range of radii on both datasets.
Revised: The improvements are consistently present over a wide range of radii on both datasets.

Original: We do observe that the ACR score for C OHEN -DS 2 on CIFAR marginally drops compared to C OHEN -DS.
Revised: We do observe that the ACR score for C OHEN -DS 2 on CIFAR marginally drops compared to C OHEN -DS.

Original: We believe that this is due to the fact that some inputs that are classiﬁed correctly at the small radii have an overall larger certiﬁcation radius for C OHEN -DS compared to C OHEN -DS 2 on CIFAR10.
Revised: We believe that this is due to the fact that some inputs that are classified correctly at the small radii have an overall larger certification radius for C OHEN -DS compared to C OHEN -DS 2 on CIFAR10.

Original: Regardless, C OHEN -DS 2 substantially outperforms
Revised: Regardless, C OHEN -DS 2 substantially outperforms

Original: C OHEN by 0 .
Revised: C OHEN by 0 .

Original: ACR points.
Revised: ACR points.

Original: As compared to C OHEN -DS,
Revised: As compared to C OHEN -DS,

Original: C OHEN -DS 2 improves the ACR on ImageNet from 1 .to 1 . 319 .
Revised: C OHEN -DS 2 improves the ACR on ImageNet from 1 .to 1 . 319 .

Original:  SMOOTHADV + DS
Revised: 4. SMOOTHADV + DS

Original: We combine our data dependent smoothing strategy with the more effective S MOOTH A DV , which trains the smoothed classiﬁer for every x on the adversarial example ˆ x that maximizes − log E (cid:15) ∼N (0 ,σ 2 I ) [ f yθ ( x (cid:48) + (cid:15) )] , where (cid:107) x (cid:48) − x (cid:107) ≤ ζ .For CIFAR10 experiments, we follow the training procedure of S MOOTH A DV , where the adversary ˆ x is computed with 2 PGD (proximal gradient descent) steps with ζ = 0 .and
Revised: We combine our data dependent smoothing strategy with the more effective S MOOTH A DV , which trains the smoothed classifier for every x on the adversarial example ˆ x that maximizes − log E ϵ ∼N (0 ,σ 2 I ) [ f yθ ( x ′ + ϵ )] , where ∥ x ′ − x ∥ ≤ ζ .For CIFAR10 experiments, we follow the training procedure of S MOOTH A DV , where the adversary ˆ x is computed with 2 PGD (proximal gradient descent) steps with ζ = 0 .and

Original: one augmented sample to estimate the expectation.
Revised: one augmented sample to estimate the expectation.

Original: For
Revised: For

Original: ImageNet experiments, we use the best reported models, in terms of certiﬁed accuracy, provided by the authors, which correspond to ζ = 0 .
Revised: ImageNet experiments, we use the best reported models, in terms of certified accuracy, provided by the authors, which correspond to ζ = 0 .

Original: 5 for σ = 0 .
Revised: 5 for σ = 0 .

Original: 25 and ζ = 1 . 

Revised: 25 and ζ = 1 .

Original: for σ ∈ { 0 . 5 , 1 . 0 } .
Revised: for σ ∈ { 0 . 5 , 1 . 0 } .

Original: [Table]

Original: DS for certiﬁcation only.
Revised: DS for certification only.

Original: Similar to C OHEN , we ﬁrst certify S MOOTH A DV models trained with the same ﬁxed σ .
Revised: Similar to C OHEN , we first certify S MOOTH A DV models trained with the same fixed σ .

Original: Then, we certify the proposed data dependent σ ∗ x models using the memory-based certiﬁcation, which we refer to as S MOOTH A DV -DS.
Revised: Then, we certify the proposed data dependent σ ∗ x models using the memory-based certification, which we refer to as S MOOTH A DV -DS.

Original: In Figure 4, we show the certiﬁed accuracy for both CIFAR10 and ImageNet in the ﬁrst and second rows, respectively.
Revised: In Figure 4, we show the certified accuracy for both CIFAR10 and ImageNet in the first and second rows, respectively.

Original: The last column shows the envelopes per radius.
Revised: The last column shows the envelopes per radius.

Original: Even though they both share the same classiﬁer f θ , S MOOTH A DV -DS signiﬁcantly improves upon
Revised: Even though they both share the same classifier f θ , S MOOTH A DV -DS significantly improves upon

Original: S MOOTH A DV over all radii and all values of σ in training for both CIFAR10 and ImageNet.
Revised: S MOOTH A DV over all radii and all values of σ in training for both CIFAR10 and ImageNet.

Original: In particular, for models trained with σ = 0 .
Revised: In particular, for models trained with σ = 0 .

Original: S MOOTH A DV achieves a zero certiﬁed accuracy for large certiﬁcation radii ( ≥ 1 . 0 ), while
Revised: S MOOTH A DV achieves a zero cer- [Table] tified accuracy for large certification radii ( ≥ 1 . 0 ), while

Original: S MOOTH A DV -DS achieves non-trivial certiﬁed accuracy in these cases.
Revised: S MOOTH A DV -DS achieves non-trivial certified accuracy in these cases.

Original: Similar to the earlier setup, we report the best certiﬁed accuracy along with the ACR scores in Table 2.
Revised: Similar to the earlier setup, we report the best certified accuracy along with the ACR scores in Table 2.

Original: We improve over S MOOTH A DV by large margins.
Revised: We improve over S MOOTH A DV by large margins.

Original: For example, the certiﬁed accuracy at 0 .
Revised: For example, the certified accuracy at 0 .

Original: 5 radius increases by 5 . 4% and 2 .
Revised: 5 radius increases by 5 . 4% and 2 .

Original: 8% on CIFAR10 and Imagenet, respectively.
Revised: 8% on CIFAR10 and Imagenet, respectively.

Original: The improvement is consistent over all radii.
Revised: The improvement is consistent over all radii.

Original: The ACR also improves by 0 .
Revised: The ACR also improves by 0 .

Original: 118 and 0 .
Revised: 118 and 0 .

Original: 158 on CIFAR10 and ImageNet, respectively.
Revised: 158 on CIFAR10 and ImageNet, respectively.

Original: DS for training and certiﬁcation.
Revised: DS for training and certification.

Original: We ﬁne tune the
Revised: We fine tune the

Original: S MOOTH A DV trained models (either the retrained CIFAR models or the ImageNet models provided by S MOOTH A DV ) using Algorithm 2, where σ ∗ x is computed using Algorithm 1.
Revised: S MOOTH A DV trained models (either the retrained CIFAR models or the ImageNet models provided by S MOOTH A DV ) using Algorithm 2, where σ ∗ x is computed using Algorithm 1.

Original: We report the per σ certiﬁcation accuracy comparing S MOOTH A DV -DS 2 (certiﬁed also using memory based certiﬁcation) to both S MOOTH A DV -DS and S MOOTH A DV .
Revised: We report the per σ certification accuracy comparing S MOOTH A DV -DS 2 (certified also using memory based certification) to both S MOOTH A DV -DS and S MOOTH A DV .

Original: S MOOTH A DV -DS 2 further improves the certiﬁed accuracy as compared to S MOOTH A DV -DS with performance gains more prominent on ImageNet.
Revised: S MOOTH A DV -DS 2 further improves the certified accuracy as compared to S MOOTH A DV -DS with performance gains more prominent on ImageNet.

Original: While the improvement of
Revised: While the improvement of

Original: S MOOTH A DV -DS 2 over S MOOTH
Revised: S MOOTH A DV -DS 2 over S MOOTH

Original: A DV -DS is indeed small, e.g. 0 .
Revised: A DV -DS is indeed small, e.g. 0 .

Original: 7% at radius 0 .
Revised: 7% at radius 0 .

Original: 5 on CIFAR10, we observe that the performance gaps are much larger on ImageNet reaching 1 . 4% at 0 .
Revised: 5 on CIFAR10, we observe that the performance gaps are much larger on ImageNet reaching 1 . 4% at 0 .

Original: 5 radius as shown in Table 2.
Revised: 5 radius as shown in Table 2.

Original: We see a similar trend in ACR with improvements of 0 .
Revised: We see a similar trend in ACR with improvements of 0 .

Original: 013 and 0 .
Revised: 013 and 0 .

Original: 069 on
Revised: 069 on

Original: CIFAR10 and ImageNet, respectively.
Revised: CIFAR10 and ImageNet, respectively.

Original: S MOOTH
Revised: S MOOTH

Original: A DV -DSboosts the certiﬁed accuracy of S MOOTH A DV at radius 0.
Revised: A DV -DSboosts the certified accuracy of S MOOTH A DV at radius 0.

Original: by 6 . 1% and 4 .
Revised: by 6 . 1% and 4 .

Original: 2% on CIFAR10 and ImageNet, respectively.
Revised: 2% on CIFAR10 and ImageNet, respectively.

Original: 4. MACER + DS
Revised: 4. MACER + DS

Original: We integrate data dependent smoothing within
Revised: We integrate data dependent smoothing within

Original: MACER which trains g θ by minimizing over the parameters θ the following objective − log g θ ( x ) 
Revised: MACER which trains g θ by minimizing over the parameters θ the following objective − log g [Equation] ( x ) [Equation]

Original: + λσ
Revised: λσ

Original: [Equation]
Revised: [Equation]

Original: where R also depends on θ .
Revised: where R also depends on θ .

Original: While this seems to be similar in spirit to our approach, we in fact maximize the certiﬁcation radius over σ with ﬁxed parameters θ for every x .
Revised: While this seems to be similar in spirit to our approach, we in fact maximize the certification radius over σ with fixed parameters θ for every x .

Original: We conduct experiments on CIFAR103 following the training procedure of MACER estimating the expectation with 64 samples, λ = 12 , and γ = 8 .We set n = 8 in Algorithm 1 with [Figure] [Figure]
Revised: We conduct experiments on CIFAR10 following the training procedure of MACER estimating the expectation with 64 samples, λ = 12 , and γ = 8 .

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] ablations on n = 1 in the appendix .
Revised: We set n = 8 in Algorithm 1 with ablations on n = 1 in the appendix .

Original: DS for certiﬁcation only.
Revised: DS for certification only.

Original: Similar to the earlier setup in
Revised: Similar to the earlier setup in

Original: C OHEN and S MOOTH
Revised: C OHEN and S MOOTH

Original: A DV , we certify models with ﬁxed σ and then with data dependent σ ∗ x using the memory based certiﬁcation, referred to as MACER-DS.
Revised: A DV , we certify models with fixed σ and then with data dependent σ ∗ x using the memory based certification, referred to as MACER-DS.

Original: In Figure 5, we observe that MACER-DS signiﬁcantly outperforms MACER particularly in the large radius region.
Revised: In Figure 5, we observe that MACER-DS significantly outperforms MACER particularly in the large radius region.

Original: This can also be seen in the envelope ﬁgure reporting the best certiﬁed accuracy per radius over σ .
Revised: This can also be seen in the envelope figure reporting the best certified accuracy per radius over σ .

Original: Similarly, Table 3 demonstrates the beneﬁts of data dependent smoothing, where it boosts certiﬁed accuracy by 7 . 4% (from 59 .
Revised: Similarly, Table 3 demonstrates the benefits of data dependent smoothing, where it boosts certified accuracy by 7 . 4% (from 59 .

Original: 3% to 66 .
Revised: 3% to 66 .

Original: 7% ) and 8 .
Revised: 7% ) and 8 .

Original: 7% ( 43 .to 52 . 3 ) at 0 .
Revised: 7% ( 43 .to 52 . 3 ) at 0 .

Original: 25 and 0 .
Revised: 25 and 0 .

Original: 5 radii, respectively.
Revised: 5 radii, respectively.

Original: Moreover, we improve ACR by 0 .
Revised: Moreover, we improve ACR by 0 .

Original: 139 points.
Revised: 139 points.

Original: DS for training and certiﬁcation.
Revised: DS for training and certification.

Original: We incorporate data dependent smoothing as part of MACER training and certiﬁcation in a similar fashion to the earlier setup, dubbed MACER-DS 2 .
Revised: We incorporate data dependent smoothing as part of MACER training and certification in a similar fashion to the earlier setup, dubbed MACER-DS 2 .

Original: Figure 5 shows the improvement of
Revised: Figure 5 shows the improvement of

Original: MACER-DS 2 over the certiﬁcation only MACER-DS over all trained models.
Revised: MACER-DS 2 over the certification only MACER-DS over all trained models.

Original: Table 3 summarizes the best certiﬁed accuracy per radius.
Revised: Table 3 summarizes the best certified accuracy per radius.

Original: Overall, we ﬁnd that the performance is comparable or slightly better than MACER-DS, which is still signiﬁcantly better than MACER by 8 .
Revised: Overall, we find that the performance is comparable or slightly better than MACER-DS, which is still significantly better than MACER by 8 .

Original: 67% at radius 0 .
Revised: 67% at radius 0 .

Original: We also observe that MACER-DS enjoys better
Revised: We also observe that MACER-DS enjoys better

Original: ACR than MACER-DS 2 with both being far better than the
Revised: ACR than MACER-DS 2 with both being far better than the

Original: MACER baseline.
Revised: MACER baseline.

Original: 4.4 DISCUSSION AND ABLATION
Revised: 4.4 DS FOR ℓ 1 CERTIFICATES

Revised: At last, we extend our methodology to ℓ 1 certification.

Revised: We leveraged the results of Yang et al.

Revised: [2020] that derived the tightest ℓ 1 certificate using randomized smoothing with uniform distribution U [ − λ, λ ] d .

Revised: The certified radius in that case has the form R 1 =

Revised: λ ( p A − p B ) .

Revised: We replace our objective in Equation (3) with:

Revised: [Equation]

Revised: We solved our objective in Eq (3) in an identical fashion to our Algorithm 1 with the same hyperparameters for λ ∈ { 0 . 25 , 0 .

Revised: 5 , 1 . 0 } in certification on both CIFAR10 and

Revised: ImageNet.

Revised: Further, we combine our data-dependent smooth classifier with the memory based algorithm proposed in Section 3.5.

Revised: It is worthwhile mentioning that similar to the ℓcase, the memory based algorithm did not find any overlap between the certified regions of any pair of instances.

Revised: We report the results in Table 4.

Revised: We observe that, similar to our extensive experiments on the ℓ 2 certificate, our proposed

Revised: [Table]

Revised: memory-enhanced data-dependent smoothing yields consistent improvement in the ℓ 1 certified accuracy.

Revised: We report an improvement of 7% and 3% over the state of the art certified accuracy at ℓ 1 radius of 0.5 on CIFAR10 and ImageNet, respectively.

Revised: At last, we note similar improvement to the ℓ

Revised: ACR as reported in Table 4.

Revised: 4.5 DISCUSSION AND ABLATION

Original: Varying K .
Revised: Varying K .

Original: We pose the question: does attaining better solutions to our proposed Objective 3 improve certiﬁed ac- curacy?
Revised: We pose the question: does attaining better solutions to our proposed Objective 3 improve certified accuracy?

Original: To answer this, we control the solution quality of σ ∗ x by certifying trained models with a varying number of stochastic gradient ascent iterations K in Algorithm 1.
Revised: To answer this, we control the solution quality of σ ∗ x by certifying trained models with a varying number of stochastic gradient ascent iterations K in Algorithm 1.

Original: In particular, we certify the trained models S MOOTH
Revised: In particular, we certify the trained models S MOOTH

Original: A DV -DSand S MOOTH A DV -DS on CIFAR10 and ImageNet, respectively, with a varying K .
Revised: A DV -DSand S MOOTH A DV -DS on CIFAR10 and ImageNet, respectively, with a varying K .

Original: We leave the rest of the experiments for other models to the appendix .
Revised: We leave the rest of the experiments for other models to the appendix .

Original: We observe in
Revised: We observe in

Original: Figure 7 that the certiﬁed accuracy per radius consistently improves as K increases, particularly in the large radius regime.
Revised: Figure 7 that the certified accuracy per radius consistently improves as K increases, particularly in the large radius regime.

Original: This is expected, since Algorithm 1 produces better optimal smoothing σ ∗ x per input x with larger K , which in turn improves the certiﬁcation radius leaving room for improvements with more powerful optimizers 4 .
Revised: This is expected, since Algorithm 1 produces better optimal smoothing σ ∗ x per input x with larger K , which in turn improves the certification radius leaving room for improvements with more powerful optimizers.

Original: Visualizing σ ∗ x .
Revised: Visualizing σ ∗ x .

Original: We show the variation of σ ∗ x that maximizes the certiﬁcation radius over different inputs x .
Revised: We show the variation of σ ∗ x that maximizes the certification radius over different inputs x .

Original: Figure 6 shows two examples, where the ﬁrst and fourth columns contain the clean images.
Revised: Figure 6 shows two examples, where the first and fourth columns contain the clean images.

Original: In the second column, a choice of ﬁxed σ = 0 .
Revised: In the second column, a choice of fixed σ = 0 .

Original: 5 is too large compared to our estimated σ ∗ x = 0 .
Revised: 5 is too large compared to our estimated σ ∗ x = 0 .

Original: 368 that maximizes the certiﬁcation radius as per
Revised: 368 that maximizes the certification radius as per

Original: Algorithm 1.
Revised: Algorithm 1.

Original: As for the ﬁfth column, we observe that a constant σ = 0 .
Revised: As for the fifth column, we observe that a constant σ = 0 .

Original: 25 is far less than σ ∗ x = 0 .
Revised: 25 is far less than σ ∗ x = 0 .

Original: This indicates that indeed the σ ∗ x maximizing the certiﬁcation radius varies signiﬁcantly over inputs.  
Revised: This indicates that indeed the σ ∗ x maximizing the certification radius varies significantly over inputs.  CONCLUSION

Revised: In this work, we presented a simple and generic framework to equip randomized smoothing techniques with data dependency.

Revised: We demonstrated that combining data dependent smoothing with 3 randomized smoothing techniques provided substantial improvement in their certified accuracy.

Revised: Acknowledgements

Revised: This publication is based upon work supported by King

Revised: Abdullah University of Science and Technology (KAUST) under Award No. ORA-CRG10-2021-4648.

Revised: We thank Francisco Girbal Eiras for the help in the memory based certification and the discussions.