Original: E NHANCING C ERTIFIED R OBUSTNESS OF S MOOTHED C LASSIFIERS VIA W EIGHTED M ODEL E NSEMBLING
Revised: E NHANCING C ERTIFIED R OBUSTNESS OF S MOOTHED C LASSIFIERS VIA W EIGHTED M ODEL E NSEMBLING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Randomized smoothing has achieved state-of-the-art certiﬁed robustness against l 2 -norm adversarial attacks.
Revised: Randomized smoothing has achieved state-of-the-art certiﬁed robustness against l 2 -norm adversarial attacks.

Original: However, it is not wholly resolved on how to ﬁnd the optimal base classiﬁer for randomized smoothing.
Revised: However, it is not wholly resolved on how to ﬁnd the optimal base classiﬁer for randomized smoothing.

Original: In this work, we employ a
Revised: In this work, we employ a

Original: Smoothed WEighted ENsembling (SWEEN) scheme to improve the performance of randomized smoothed classiﬁers.
Revised: Smoothed WEighted ENsembling (SWEEN) scheme to improve the performance of randomized smoothed classiﬁers.

Original: We show the ensembling generality that
Revised: We show the ensembling generality that

Original: SWEEN can help achieve optimal certiﬁed robustness.
Revised: SWEEN can help achieve optimal certiﬁed robustness.

Original: Furthermore, theoretical analysis proves that the optimal SWEEN model can be obtained from training under mild assumptions.
Revised: Furthermore, theoretical analysis proves that the optimal SWEEN model can be obtained from training under mild assumptions.

Original: We also develop an adaptive prediction algorithm to reduce the prediction and certiﬁcation cost of SWEEN models.
Revised: We also develop an adaptive prediction algorithm to reduce the prediction and certiﬁcation cost of SWEEN models.

Original: Extensive experiments show that SWEEN models outperform the upper envelope of their corresponding candidate models by a large margin.
Revised: Extensive experiments show that SWEEN models outperform the upper envelope of their corresponding candidate models by a large margin.

Original: Moreover, SWEEN models constructed using a few small models can achieve comparable performance to a single large model with a notable reduction in training time.
Revised: Moreover, SWEEN models constructed using a few small models can achieve comparable performance to a single large model with a notable reduction in training time.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have achieved great success in image classiﬁcation tasks.
Revised: Deep neural networks have achieved great success in image classiﬁcation tasks.

Original: However, they are vulnerable to adversarial examples , which are small imperceptible perturbations on the original inputs that can cause misclassiﬁcation (Biggio et al., 2013; Szegedy et al., 2014).
Revised: However, they are vulnerable to adversarial examples , which are small imperceptible perturbations on the original inputs that can cause misclassiﬁcation (Biggio et al., 2013; Szegedy et al., 2014).

Original: To tackle this problem, researchers have proposed various defense methods to train classiﬁers robust to adversarial perturbations.
Revised: To tackle this problem, researchers have proposed various defense methods to train classiﬁers robust to adversarial perturbations.

Original: These defenses can be roughly categorized into empirical defenses and certiﬁed defenses.
Revised: These defenses can be roughly categorized into empirical defenses and certiﬁed defenses.

Original: One of the most successful empirical defenses is adversarial training (Kurakin et al., 2017;
Revised: One of the most successful empirical defenses is adversarial training (Kurakin et al., 2017;

Original: Madry et al., 2018) which optimizes the model by minimizing the loss over adversarial examples generated during training.
Revised: Madry et al., 2018) which optimizes the model by minimizing the loss over adversarial examples generated during training.

Original: Empirical defenses produce models robust to certain attacks without a theoretical guarantee.
Revised: Empirical defenses produce models robust to certain attacks without a theoretical guarantee.

Original: Most of the empirical defenses are heuristic and subsequently broken by more sophisticated adversaries (Carlini & Wagner, 2017; Athalye et al., 2018; Uesato et al., 2018;
Revised: Most of the empirical defenses are heuristic and subsequently broken by more sophisticated adversaries (Carlini & Wagner, 2017; Athalye et al., 2018; Uesato et al., 2018;

Original: Tramer et al., 2020).
Revised: Tramer et al., 2020).

Original: Certiﬁed defenses, either exact or conservative, are introduced to mitigate such deﬁciency in empirical defenses.
Revised: Certiﬁed defenses, either exact or conservative, are introduced to mitigate such deﬁciency in empirical defenses.

Original: In the context of l p norm-bounded perturbations, exact methods report whether an adversarial example exists within an l p ball with radius r centered at a given input x .
Revised: In the context of l p norm-bounded perturbations, exact methods report whether an adversarial example exists within an l p ball with radius r centered at a given input x .

Original: Exact methods are usually based on Satisﬁability Modulo Theories (Katz et al., 2017; Ehlers, 2017) or mixed-integer linear programming (Lomuscio & Maganti, 2017; Fischetti & Jo, 2017), which are computationally inefﬁcient and not scalable (Tjeng et al., 2019).
Revised: Exact methods are usually based on Satisﬁability Modulo Theories (Katz et al., 2017; Ehlers, 2017) or mixed-integer linear programming (Lomuscio & Maganti, 2017; Fischetti & Jo, 2017), which are computationally inefﬁcient and not scalable (Tjeng et al., 2019).

Original: Conservative methods are more computationally efﬁcient, but might mistakenly ﬂag a safe data point as vulnerable to adversarial examples (Raghunathan et al., 2018a; Wong & Kolter, 2018;
Revised: Conservative methods are more computationally efﬁcient, but might mistakenly ﬂag a safe data point as vulnerable to adversarial examples (Raghunathan et al., 2018a; Wong & Kolter, 2018;

Original: Wong et al., 2018; Gehr et al., 2018;
Revised: Wong et al., 2018; Gehr et al., 2018;

Original: Mirman et al., 2018; Weng et al., 2018; Zhang et al., 2018; Raghunathan et al., 2018b; Dvijotham et al., 2018b; Singh et al., 2018; Wang et al., 2018b;
Revised: Mirman et al., 2018; Weng et al., 2018; Zhang et al., 2018; Raghunathan et al., 2018b; Dvijotham et al., 2018b; Singh et al., 2018; Wang et al., 2018b;

Original: Salman et al., 2019b; Croce et al., 2019; Gowal et al., 2018; Dvijotham et al., 2018a; Wang et al., 2018a).
Revised: Salman et al., 2019b; Croce et al., 2019; Gowal et al., 2018; Dvijotham et al., 2018a; Wang et al., 2018a).

Original: However, both types of defenses are not scalable to practical networks that perform well on modern machine learning problems (e.g., the
Revised: However, both types of defenses are not scalable to practical networks that perform well on modern machine learning problems (e.g., the

Original: ImageNet (Deng et al., 2009) classiﬁcation task).
Revised: ImageNet (Deng et al., 2009) classiﬁcation task).

Original: Recently, a new certiﬁed defense technique called randomized smoothing has been proposed (Lecuyer et al., 2019; Cohen et al., 2019).
Revised: Recently, a new certiﬁed defense technique called randomized smoothing has been proposed (Lecuyer et al., 2019; Cohen et al., 2019).

Original: A (randomized) smoothed classiﬁer is constructed from a base classiﬁer, typically a deep neural network.
Revised: A (randomized) smoothed classiﬁer is constructed from a base classiﬁer, typically a deep neural network.

Original: It outputs the most probable class given by its base classiﬁer under a random noise perturbation of the input.
Revised: It outputs the most probable class given by its base classiﬁer under a random noise perturbation of the input.

Original: Randomized smoothing is scalable due to its independency over architectures and has achieved state-of-the-art certiﬁed l 2 -robustness.
Revised: Randomized smoothing is scalable due to its independency over architectures and has achieved state-of-the-art certiﬁed l 2 -robustness.

Original: In theory, randomized smoothing can apply to any classiﬁers.
Revised: In theory, randomized smoothing can apply to any classiﬁers.

Original: However, naively using randomized smoothing on standard-trained classiﬁers leads to poor robustness results.
Revised: However, naively using randomized smoothing on standard-trained classiﬁers leads to poor robustness results.

Original: It is still not wholly resolved on how a base classiﬁer should be trained so that the corresponding smoothed classiﬁer has good robustness properties.
Revised: It is still not wholly resolved on how a base classiﬁer should be trained so that the corresponding smoothed classiﬁer has good robustness properties.

Original: Recently, Salman et al. (2019a) employ adversarial training to train base classiﬁers and substantially improve the performance of randomized smoothing, which indicates that techniques originally proposed for empirical defenses can be useful in ﬁnding good base classiﬁers for randomized smoothing.
Revised: Recently, Salman et al. (2019a) employ adversarial training to train base classiﬁers and substantially improve the performance of randomized smoothing, which indicates that techniques originally proposed for empirical defenses can be useful in ﬁnding good base classiﬁers for randomized smoothing.

Original: In this paper, we take a step towards ﬁnding suitable base models for randomized smoothing by model ensembling.
Revised: In this paper, we take a step towards ﬁnding suitable base models for randomized smoothing by model ensembling.

Original: The idea of model ensembling has been used in various empirical defenses against adversarial examples and shows promising results for robustness (Liu et al., 2018; Strauss et al., 2018;
Revised: The idea of model ensembling has been used in various empirical defenses against adversarial examples and shows promising results for robustness (Liu et al., 2018; Strauss et al., 2018;

Original: Pang et al., 2019; Wang et al., 2019; Meng et al., 2020; Sen et al., 2020).
Revised: Pang et al., 2019; Wang et al., 2019; Meng et al., 2020; Sen et al., 2020).

Original: Moreover, an ensemble can combine the strengths of candidate models 1 to achieve superior clean accuracy (Hansen & Salamon, 1990; Krogh & Vedelsby, 1994).
Revised: Moreover, an ensemble can combine the strengths of candidate models 1 to achieve superior clean accuracy (Hansen & Salamon, 1990; Krogh & Vedelsby, 1994).

Original: Thus, we believe ensembling several smoothed models can help improve both the robustness and accuracy.
Revised: Thus, we believe ensembling several smoothed models can help improve both the robustness and accuracy.

Original: Speciﬁcally for randomized smoothing, the smoothing operator is commutative with the ensembling operator: ensembling several smoothed models is equivalent to smoothing an ensembled base model.
Revised: Speciﬁcally for randomized smoothing, the smoothing operator is commutative with the ensembling operator: ensembling several smoothed models is equivalent to smoothing an ensembled base model.

Original: This property makes the combination suitable and efﬁcient.
Revised: This property makes the combination suitable and efﬁcient.

Original: Therefore, we directly ensemble a base model by taking some pre-trained models as candidates and optimizing the optimal weights for randomized smoothing.
Revised: Therefore, we directly ensemble a base model by taking some pre-trained models as candidates and optimizing the optimal weights for randomized smoothing.

Original: We refer to the ﬁnal model as a Smoothed WEighted ENsembling (SWEEN) model.
Revised: We refer to the ﬁnal model as a Smoothed WEighted ENsembling (SWEEN) model.

Original: Moreover, SWEEN does not limit how individual candidate classiﬁers are trained, thus is compatible with most previously proposed training algorithms on randomized smoothing.
Revised: Moreover, SWEEN does not limit how individual candidate classiﬁers are trained, thus is compatible with most previously proposed training algorithms on randomized smoothing.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: We propose SWEEN to substantially improve the performance of smoothed models.
Revised: We propose SWEEN to substantially improve the performance of smoothed models.

Original: Theoretical analysis shows the ensembling generality and the optimization guarantee: SWEEN can achieve optimal certiﬁed robustness w.r.t.
Revised: Theoretical analysis shows the ensembling generality and the optimization guarantee: SWEEN can achieve optimal certiﬁed robustness w.r.t.

Original: the deﬁned γ -robustness index, which is an extension of previously proposed criteria of certiﬁed robustness (Lemma 1), and SWEEN can be easily trained to a near-optimal risk with a surrogate loss (Theorem 2).
Revised: the deﬁned γ -robustness index, which is an extension of previously proposed criteria of certiﬁed robustness (Lemma 1), and SWEEN can be easily trained to a near-optimal risk with a surrogate loss (Theorem 2).

Original: We develop an adaptive prediction algorithm for the weighted ensembling, which effectively reduces the prediction and certiﬁcation cost of the smoothed ensemble classiﬁer.
Revised: We develop an adaptive prediction algorithm for the weighted ensembling, which effectively reduces the prediction and certiﬁcation cost of the smoothed ensemble classiﬁer.

Original: We evaluate our proposed method through extensive experiments.
Revised: We evaluate our proposed method through extensive experiments.

Original: On all tasks, SWEEN models consistently outperform the upper envelopes of their respective candidate models in terms of the approximated certiﬁed accuracy by a large margin.
Revised: On all tasks, SWEEN models consistently outperform the upper envelopes of their respective candidate models in terms of the approximated certiﬁed accuracy by a large margin.

Original: In addition, SWEEN models can achieve comparable or superior performance to a large individual model using a few candidates with a notable reduction in total training time.
Revised: In addition, SWEEN models can achieve comparable or superior performance to a large individual model using a few candidates with a notable reduction in total training time.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In the past few years, numerous defenses have been proposed to build classiﬁers robust to adversarial examples.
Revised: In the past few years, numerous defenses have been proposed to build classiﬁers robust to adversarial examples.

Original: Our work typically involves randomized smoothing and model ensembling.
Revised: Our work typically involves randomized smoothing and model ensembling.

Original: Randomized smoothing
Revised: Randomized smoothing

Original: Randomized smoothing constructs a smoothed classiﬁer from a base classiﬁer via convolution between the input distribution and certain noise distribution.
Revised: Randomized smoothing constructs a smoothed classiﬁer from a base classiﬁer via convolution between the input distribution and certain noise distribution.

Original: It is ﬁrst proposed as a heuristic defense by (Liu et al., 2018; Cao & Gong, 2017).
Revised: It is ﬁrst proposed as a heuristic defense by (Liu et al., 2018; Cao & Gong, 2017).

Original: Lecuyer et al.
Revised: Lecuyer et al.

Original: ﬁrst prove robustness guarantees for randomized smoothing utilizing tools from differential privacy.
Revised: ﬁrst prove robustness guarantees for randomized smoothing utilizing tools from differential privacy.

Original: Subsequently, a stronger robustness guarantee is given by Li et al.
Revised: Subsequently, a stronger robustness guarantee is given by Li et al.

Original: Cohen et al.
Revised: Cohen et al.

Original: (2019) provide a tight robustness bound for isotropic Gaussian noise in l 2 robustness setting.
Revised: (2019) provide a tight robustness bound for isotropic Gaussian noise in l 2 robustness setting.

Original: The theoretical properties of randomized smoothing in various norm and noise distribution settings have been further discussed in the literature (Blum et al., 2020; Kumar et al., 2020;
Revised: The theoretical properties of randomized smoothing in various norm and noise distribution settings have been further discussed in the literature (Blum et al., 2020; Kumar et al., 2020;

Original: Yang et al., 2020; Lee et al., 2019; Teng et al., 2019; Zhang et al., 2020).
Revised: Yang et al., 2020; Lee et al., 2019; Teng et al., 2019; Zhang et al., 2020).

Original: Recently, a series of works (Salman et al., 2019a; Zhai et al., 2020) develop practical algorithms to train a base classiﬁer for randomized smoothing.
Revised: Recently, a series of works (Salman et al., 2019a; Zhai et al., 2020) develop practical algorithms to train a base classiﬁer for randomized smoothing.

Original: Our work improves the performance of smoothed classiﬁers via weighted ensembling of pre-trained base classiﬁers.
Revised: Our work improves the performance of smoothed classiﬁers via weighted ensembling of pre-trained base classiﬁers.

Original: Model ensembling
Revised: Model ensembling

Original: Model ensembling has been widely studied and applied in machine learning as a technique to improve the generalization performance of the model (Hansen & Salamon, 1990;
Revised: Model ensembling has been widely studied and applied in machine learning as a technique to improve the generalization performance of the model (Hansen & Salamon, 1990;

Original: Krogh & Vedelsby, 1994).
Revised: Krogh & Vedelsby, 1994).

Original: Krogh & Vedelsby (1994) show that ensembles constructed from accurate and diverse networks perform better.
Revised: Krogh & Vedelsby (1994) show that ensembles constructed from accurate and diverse networks perform better.

Original: Recently, simple averaging of multiple neural networks has been a success in ILSVRC competitions (He et al., 2016; Krizhevsky et al., 2017; Simonyan & Zisserman, 2015).
Revised: Recently, simple averaging of multiple neural networks has been a success in ILSVRC competitions (He et al., 2016; Krizhevsky et al., 2017; Simonyan & Zisserman, 2015).

Original: Model ensembling has also been used in defenses against adversarial examples (Liu et al., 2018;
Revised: Model ensembling has also been used in defenses against adversarial examples (Liu et al., 2018;

Original: Strauss et al., 2018; Pang et al., 2019; Wang et al., 2019;
Revised: Strauss et al., 2018; Pang et al., 2019; Wang et al., 2019;

Original: Meng et al., 2020; Sen et al., 2020).
Revised: Meng et al., 2020; Sen et al., 2020).

Original: Wang et al.
Revised: Wang et al.

Original: (2019) have shown that a jointly trained ensemble of noise injected ResNets can improve clean and robust accuracies.
Revised: (2019) have shown that a jointly trained ensemble of noise injected ResNets can improve clean and robust accuracies.

Original: Recently, Meng et al.
Revised: Recently, Meng et al.

Original: ﬁnd that ensembling diverse weak models can be quite robust to adversarial attacks.
Revised: ﬁnd that ensembling diverse weak models can be quite robust to adversarial attacks.

Original: Unlike the above works, which are empirical or heuristic, we employ ensembling in randomized smoothing to provide a theoretical robustness certiﬁcation.
Revised: Unlike the above works, which are empirical or heuristic, we employ ensembling in randomized smoothing to provide a theoretical robustness certiﬁcation.

Original:  P RELIMINARIES Notation
Revised:  P RELIMINARIES Notation

Original: Let Y = { 1 , 2 , ..., M } .
Revised: Let Y = { 1 , 2 , ..., M } .

Original: We overload notation slightly, letting k refer the M -dimensional one-hot vector whose k -th entry is 1 for k = 1 , ..., M as well.
Revised: We overload notation slightly, letting k refer the M -dimensional one-hot vector whose k -th entry is 1 for k = 1 , ..., M as well.

Original: The choice should be clear from context.
Revised: The choice should be clear from context.

Original: Let ∆ k
Revised: Let ∆ k

Original: = { ( p 1 , p 2 , ..., p k ) (cid:12)(cid:12) p i ≥ 0 , (cid:80) ki
Revised: = { ( p 1 , p 2 , ..., p k ) (cid:12)(cid:12) p i ≥ 0 , (cid:80) ki

Original: =1 p i = 1 } be the k -dimensional probability simplex for k ∈ N + , and ∆ =
Revised: =1 p i = 1 } be the k -dimensional probability simplex for k ∈ N + , and ∆ =

Original: ∆ M .
Revised: ∆ M .

Original: For an M -dimensional function f , we use f i to refer to its i -th entry, i = 1 , 2 , · · · , M .
Revised: For an M -dimensional function f , we use f i to refer to its i -th entry, i = 1 , 2 , · · · , M .

Original: We use N (0 , σ 2 I ) to denote the d -dimensional Gaussian distribution with mean 0 and variance σ 2 I .
Revised: We use N (0 , σ 2 I ) to denote the d -dimensional Gaussian distribution with mean 0 and variance σ 2 I .

Original: We use Φ − 1 to denote the inverse of the standard Gaussian CDF, and use Γ to denote the gamma function.
Revised: We use Φ − 1 to denote the inverse of the standard Gaussian CDF, and use Γ to denote the gamma function.

Original: We use R ∗ to denote the set of non-negative real numbers.
Revised: We use R ∗ to denote the set of non-negative real numbers.

Original: For x, a, b ∈ R , a ≤ b , we deﬁne clip( x ; a, b )
Revised: For x, a, b ∈ R , a ≤ b , we deﬁne clip( x ; a, b )

Original: =
Revised: =

Original: min { max { x, a } , b } .
Revised: min { max { x, a } , b } .

Original: We use Ω( · ) to denote Big-Omega notation that suppresses multiplicative constants.
Revised: We use Ω( · ) to denote Big-Omega notation that suppresses multiplicative constants.

Original: Neural network and classiﬁer
Revised: Neural network and classiﬁer

Original: Consider a classiﬁcation problem from X ⊆ R d to classes Y .
Revised: Consider a classiﬁcation problem from X ⊆ R d to classes Y .

Original: Assume the input space X has ﬁnite diameter D
Revised: Assume the input space X has ﬁnite diameter D

Original: = sup x 1 ,x 2 ∈X (cid:107) x 1 − x 2 (cid:107) 2 < ∞ .
Revised: = sup x 1 ,x 2 ∈X (cid:107) x 1 − x 2 (cid:107) 2 < ∞ .

Original: The training set { ( x i , y i ) } ni =1 is i.i.d.
Revised: The training set { ( x i , y i ) } ni =1 is i.i.d.

Original: drawn from the data distribution D .
Revised: drawn from the data distribution D .

Original: We call f a probability function or a classiﬁer if it is a mapping from R d to ∆ or Y , respectively.
Revised: We call f a probability function or a classiﬁer if it is a mapping from R d to ∆ or Y , respectively.

Original: For a probability function f , its induced classiﬁer f ∗ is deﬁned such that f ∗ ( x ) =
Revised: For a probability function f , its induced classiﬁer f ∗ is deﬁned such that f ∗ ( x ) =

Original: arg max 1 ≤ i ≤ M f i ( x ) .
Revised: arg max 1 ≤ i ≤ M f i ( x ) .

Original: For simplicity, we will not distinguish between f and f ∗ when there is no ambiguity, and hence all deﬁnitions and properties for classiﬁers automatically apply to probability functions as well.
Revised: For simplicity, we will not distinguish between f and f ∗ when there is no ambiguity, and hence all deﬁnitions and properties for classiﬁers automatically apply to probability functions as well.

Original: f ( · ; θ ) denotes a neural network parameterized by θ ∈ Θ .
Revised: f ( · ; θ ) denotes a neural network parameterized by θ ∈ Θ .

Original: Here Θ can include hyper-parameters, thus the architectures of f ( · ; θ ) ’s do not have to be identical.
Revised: Here Θ can include hyper-parameters, thus the architectures of f ( · ; θ ) ’s do not have to be identical.

Original: Certiﬁed robustness
Revised: Certiﬁed robustness

Original: We call x + δ an adversarial example of a classiﬁer f , if f correctly classiﬁes x but f ( x + δ ) (cid:54) =
Revised: We call x + δ an adversarial example of a classiﬁer f , if f correctly classiﬁes x but f ( x + δ ) (cid:54) =

Original: f ( x ) .
Revised: f ( x ) .

Original: Usually (cid:107) δ (cid:107) 2 is small enough so x + δ and x appear almost identical for the human eye.
Revised: Usually (cid:107) δ (cid:107) 2 is small enough so x + δ and x appear almost identical for the human eye.

Original: The ( l 2 -)robust radius of f is deﬁned as
Revised: The ( l 2 -)robust radius of f is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (1) which is the radius of the largest l 2 ball centered at x within which f consistently predicts the true label y of x .
Revised: (1) which is the radius of the largest l 2 ball centered at x within which f consistently predicts the true label y of x .

Original: Note that r ( x, y ; f )
Revised: Note that r ( x, y ; f )

Original: = 

Revised: =

Original: if f ( x ) (cid:54) =
Revised: if f ( x ) (cid:54) =

Original: y .
Revised: y .

Original: As mentioned before, we can extend the above deﬁnitions to the case when f is a probability function by considering the induced classiﬁer f ∗ .
Revised: As mentioned before, we can extend the above deﬁnitions to the case when f is a probability function by considering the induced classiﬁer f ∗ .

Original: A certiﬁed robustness method tries to ﬁnd some lower bound r c ( x, y ; f ) of r ( x, y ; f ) , and we call r c a certiﬁed radius of f .
Revised: A certiﬁed robustness method tries to ﬁnd some lower bound r c ( x, y ; f ) of r ( x, y ; f ) , and we call r c a certiﬁed radius of f .

Original: Randomized smoothing
Revised: Randomized smoothing

Original: Let f be a probability function or a classiﬁer.
Revised: Let f be a probability function or a classiﬁer.

Original: The (randomized) smoothed function of f is deﬁned as
Revised: The (randomized) smoothed function of f is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The (randomized) smoothed classiﬁer of f is then deﬁned as g ∗ .
Revised: The (randomized) smoothed classiﬁer of f is then deﬁned as g ∗ .

Original: Cohen et al.
Revised: Cohen et al.

Original: ﬁrst provide a tight robustness guarantee for classiﬁer-based smoothed classiﬁers, which is summerized in the following theorem:
Revised: ﬁrst provide a tight robustness guarantee for classiﬁer-based smoothed classiﬁers, which is summerized in the following theorem:

Original: Theorem 1.
Revised: Theorem 1.

Original: (Cohen et al.
Revised: (Cohen et al.

Original: For any classiﬁer f , denote its smoothed function by g .
Revised: For any classiﬁer f , denote its smoothed function by g .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Later on, Salman et al. (2019a); Zhai et al.
Revised: Later on, Salman et al. (2019a); Zhai et al.

Original: (2020) extends Theorem 1 for probability functions.
Revised: (2020) extends Theorem 1 for probability functions.

Original:  SWEEN: S MOOTHED WEIGHTED ENSEMBLING
Revised:  SWEEN: S MOOTHED WEIGHTED ENSEMBLING

Original: In this section, we describe the SWEEN framework we use.
Revised: In this section, we describe the SWEEN framework we use.

Original: We also present some theoretical results for SWEEN models.
Revised: We also present some theoretical results for SWEEN models.

Original: The proofs of the results in this section can be found in Appendix A.
Revised: The proofs of the results in this section can be found in Appendix A.

Original: 4. SWEEN: O VERVIEW
Revised: 4. SWEEN: O VERVIEW

Original: To be speciﬁc, we adopt a data-dependent weighted average of neural networks to serve as the base model for smoothing.
Revised: To be speciﬁc, we adopt a data-dependent weighted average of neural networks to serve as the base model for smoothing.

Original: Suppose we have some pre-trained neural networks f ( · ; θ 1 ) , ..., f ( · ; θ K ) as ensemble candidates.
Revised: Suppose we have some pre-trained neural networks f ( · ; θ 1 ) , ..., f ( · ; θ K ) as ensemble candidates.

Original: A weighted ensemble model is then
Revised: A weighted ensemble model is then

Original: [Equation]
Revised: [Equation]

Original: where θ =
Revised: where θ =

Original: ( θ 1 , ·
Revised: ( θ 1 , ·

Original: · · , θ K ) ∈ Θ K , and w ∈ ∆ K is the ensemble weight.
Revised: · · , θ K ) ∈ Θ K , and w ∈ ∆ K is the ensemble weight.

Original: For a speciﬁc f ens , the corresponding SWEEN model is deﬁned as the smoothed function of f ens , denoted by g ens .
Revised: For a speciﬁc f ens , the corresponding SWEEN model is deﬁned as the smoothed function of f ens , denoted by g ens .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where g ( · ; θ ) is the smoothed function of f ( · ; θ ) .
Revised: where g ( · ; θ ) is the smoothed function of f ( · ; θ ) .

Original: This result means that g ens is the weighted sum of the smoothed functions of the candidate models under the same weight w , or more brieﬂy, randomized smoothing and weighted ensembling are commutative.
Revised: This result means that g ens is the weighted sum of the smoothed functions of the candidate models under the same weight w , or more brieﬂy, randomized smoothing and weighted ensembling are commutative.

Original: Thus, ensembling under the randomized smoothing framework can provides beneﬁts in improving the accuracy and robustness.
Revised: Thus, ensembling under the randomized smoothing framework can provides beneﬁts in improving the accuracy and robustness.

Original: To ﬁnd the optimal SWEEN model, we can minimize a surrogate loss of g ens over the training set to obtain the value of appropriate weights.
Revised: To ﬁnd the optimal SWEEN model, we can minimize a surrogate loss of g ens over the training set to obtain the value of appropriate weights.

Original: These data-dependent weights can make the ensemble model robust to the presence of some biased candidate models, as they will be assigned with small weights.
Revised: These data-dependent weights can make the ensemble model robust to the presence of some biased candidate models, as they will be assigned with small weights.

Original: 4. C ERTIFIED ROBUSTNESS OF SWEEN MODELS
Revised: 4. C ERTIFIED ROBUSTNESS OF SWEEN MODELS

Original: For a smoothed function g , the certiﬁed radius at ( x, y ) provided by Theorem 1 is r c ( x, y ; g ) =
Revised: For a smoothed function g , the certiﬁed radius at ( x, y ) provided by Theorem 1 is r c ( x, y ; g ) =

Original: clip( σ 2 [Φ − 1 ( g y ( x )) − Φ − 1 (max k (cid:54) =
Revised: clip( σ 2 [Φ − 1 ( g y ( x )) − Φ − 1 (max k (cid:54) =

Original: y g k ( x ))]; 0 , D ) .
Revised: y g k ( x ))]; 0 , D ) .

Original: We now formally deﬁne γ -robustness index as a criterion of certiﬁed robustness.
Revised: We now formally deﬁne γ -robustness index as a criterion of certiﬁed robustness.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: ( γ -robustness index).
Revised: ( γ -robustness index).

Original: For γ : R ∗ → R ∗ and a smoothed function g , the γ -robustness index of g is deﬁned as
Revised: For γ : R ∗ → R ∗ and a smoothed function g , the γ -robustness index of g is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: It can be easily observed that γ -robustness index is an extension of many frequently-used criteria of certiﬁed robustness of smoothed classiﬁers.
Revised: It can be easily observed that γ -robustness index is an extension of many frequently-used criteria of certiﬁed robustness of smoothed classiﬁers.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let γ 1 ( r ) = 1 { r ≥ R } , γ 2 ( r ) =
Revised: Let γ 1 ( r ) = 1 { r ≥ R } , γ 2 ( r ) =

Original: r, γ 3 ( r ) =
Revised: r, γ 3 ( r ) =

Original: π d 2 Γ( d 2 +1) r d .
Revised: π d 2 Γ( d 2 +1) r d .

Original: Then, γ 1 -robustness index is the certiﬁed accuracy at radius R (Cohen et al., 2019); γ 2 -robustness index is the average certiﬁed radius (Zhai et al., 2020); γ 3 -robustness index is the average volume of the certiﬁed region.
Revised: Then, γ 1 -robustness index is the certiﬁed accuracy at radius R (Cohen et al., 2019); γ 2 -robustness index is the average certiﬁed radius (Zhai et al., 2020); γ 3 -robustness index is the average volume of the certiﬁed region.

Original: We note that criteria considering the volume of the certiﬁed region are more comprehensive than those only considering the certiﬁed radii in a sense, as they take the input dimension into account.
Revised: We note that criteria considering the volume of the certiﬁed region are more comprehensive than those only considering the certiﬁed radii in a sense, as they take the input dimension into account.

Original: Now consider F = { f ( · ; θ ) :
Revised: Now consider F = { f ( · ; θ ) :

Original: R d → ∆ (cid:12)(cid:12) θ ∈ Θ } , the set of neural networks parametrized over Θ .
Revised: R d → ∆ (cid:12)(cid:12) θ ∈ Θ } , the set of neural networks parametrized over Θ .

Original: The corresponding set of smoothed functions is G = { g ( x ; θ ) =
Revised: The corresponding set of smoothed functions is G = { g ( x ; θ ) =

Original: E δ ∼N (0 ,σ 2 I )
Revised: E δ ∼N (0 ,σ 2 I )

Original: [ f ( x + δ ; θ )] | θ ∈ Θ } .
Revised: [ f ( x + δ ; θ )] | θ ∈ Θ } .

Original: Suppose θ 1 , · · · , θ K are drawn i.i.d.
Revised: Suppose θ 1 , · · · , θ K are drawn i.i.d.

Original: from a ﬁxed probability distribution p on Θ .
Revised: from a ﬁxed probability distribution p on Θ .

Original: The set of SWEEN models is then
Revised: The set of SWEEN models is then

Original: [Equation]
Revised: [Equation]

Original: Similar to Rahimi & Recht (2008), we consider mixtures of the form φ ( x ) =
Revised: Similar to Rahimi & Recht (2008), we consider mixtures of the form φ ( x ) =

Original: (cid:82) Θ w ( θ ) g ( x ; θ ) dθ .
Revised: (cid:82) Θ w ( θ ) g ( x ; θ ) dθ .

Original: For a mixture φ , we deﬁne (cid:107) φ (cid:107) p :=
Revised: For a mixture φ , we deﬁne (cid:107) φ (cid:107) p :=

Original: sup θ | w ( θ ) p ( θ ) | .
Revised: sup θ | w ( θ ) p ( θ ) | .

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Note that for any φ ∈ F p , φ is a smoothed probability function.
Revised: Note that for any φ ∈ F p , φ is a smoothed probability function.

Original: Intuitively, F p is quite a rich set.
Revised: Intuitively, F p is quite a rich set.

Original: The following result shows that with high probability, the best γ -robustness index a SWEEN model can obtain is near the optimal γ -robustness index in the class F p .
Revised: The following result shows that with high probability, the best γ -robustness index a SWEEN model can obtain is near the optimal γ -robustness index in the class F p .

Original: Thus, the ensembling generality also holds for the γ -robustness index we deﬁned for robustness.
Revised: Thus, the ensembling generality also holds for the γ -robustness index we deﬁned for robustness.

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose γ is a Lipschitz function.
Revised: Suppose γ is a Lipschitz function.

Original: Given η > 0 .
Revised: Given η > 0 .

Original: For any ε > 0 , for sufﬁcently large K , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: For any ε > 0 , for sufﬁcently large K , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes
Revised: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Moreover, if there exists φ 0 ∈ F p such that I
Revised: Moreover, if there exists φ 0 ∈ F p such that I

Original: [Equation]
Revised: [Equation]

Original: In practice, the deﬁned robustness index I γ ( · ) may be hard to optimized directly, in which case we choose a surrogate loss function l : R M × Y → R to approximate it.
Revised: In practice, the deﬁned robustness index I γ ( · ) may be hard to optimized directly, in which case we choose a surrogate loss function l : R M × Y → R to approximate it.

Original: Now the optimization for the ensemble weight w of a SWEEN model over a training set { ( x i , y i ) } ni
Revised: Now the optimization for the ensemble weight w of a SWEEN model over a training set { ( x i , y i ) } ni

Original: =1 can be formulated as
Revised: =1 can be formulated as

Original: [Equation]
Revised: [Equation]

Original: However, this process typically invovles Monte Carlo simulation since we only have access to f ( · , θ k ) , k = 1 , · · · , K .
Revised: However, this process typically invovles Monte Carlo simulation since we only have access to f ( · , θ k ) , k = 1 , · · · , K .

Original: We deﬁne the risk and empirical risk w.r.t.
Revised: We deﬁne the risk and empirical risk w.r.t.

Original: the surrogate loss l .
Revised: the surrogate loss l .

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: (Risk and empirical risk).
Revised: (Risk and empirical risk).

Original: For a surrogate loss function l : R M × Y → R , the risk of a probability function φ are deﬁned as
Revised: For a surrogate loss function l : R M × Y → R , the risk of a probability function φ are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: If φ ( x ) = (cid:80) Kk =1 w k g ( x ; θ k ) ∈
Revised: If φ ( x ) = (cid:80) Kk =1 w k g ( x ; θ k ) ∈

Original: ˆ F θ , for training set { ( x i , y i ) } ni =1 and sample size s , the empirical risk of φ is deﬁned as
Revised: ˆ F θ , for training set { ( x i , y i ) } ni =1 and sample size s , the empirical risk of φ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: (12) where δ ijk i.i.d.
Revised: where δ ijk i.i.d.

Original: ∼ N (0 , σ 2 I ) , 1 ≤ i ≤ n, 1 ≤ j ≤ s, 1 ≤ k ≤ K .
Revised: ∼ N (0 , σ 2 I ) , 1 ≤ i ≤ n, 1 ≤ j ≤ s, 1 ≤ k ≤ K .

Original: Now solving for w is reduced to ﬁnding the minimizer of R emp .
Revised: Now solving for w is reduced to ﬁnding the minimizer of R emp .

Original: When the loss function l is convex, this problem is a low-dimensional convex optimization, so we can obtain the global empirical risk minimizer using traditional convex optimization algorithms.
Revised: When the loss function l is convex, this problem is a low-dimensional convex optimization, so we can obtain the global empirical risk minimizer using traditional convex optimization algorithms.

Original: Furthermore, we have:
Revised: Furthermore, we have:

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose for all y ∈ Y , l ( · , y ) is a Lipschitz function with constant L which satisﬁes l (0 , y ) = 0 .
Revised: Suppose for all y ∈ Y , l ( · , y ) is a Lipschitz function with constant L and is uniformly bounded.

Original: Given η > 0 .
Revised: Given η > 0 .

Original: For any ε > 0 , for sufﬁcently large K , if n = Ω( K 2 ε 2 ) , s = Ω( log Knε 2 ) , then with probability at least 1 − η over the training dataset { ( x i , y i ) } ni =1 drawn i.i.d.
Revised: For any ε > 0 , for sufﬁcently large K , if n = Ω( K 2 ε 2 ) , s = Ω( log Knε 2 ) , then with probability at least 1 − η over the training dataset { ( x i , y i ) } ni =1 drawn i.i.d.

Original: from D and the parameters θ 1 , ..., θ K drawn i.i.d.
Revised: from D and the parameters θ 1 , ..., θ K drawn i.i.d.

Original: from p and the noise samples drawn i.i.d.
Revised: from p and the noise samples drawn i.i.d.

Original: from N (0 , σ 2 I ) , the empirical risk minimizer ˆ φ over ˆ F θ satisﬁes
Revised: from N (0 , σ 2 I ) , the empirical risk minimizer ˆ φ over ˆ F θ satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Moreover, if there exists φ 0 ∈ F p such that R [ φ 0 ] = inf φ ∈ F p R [ φ ] , K = Ω( 1 ε 4 ) .
Revised: Moreover, if there exists φ 0 ∈ F p such that R [ φ 0 ] = inf φ ∈ F p R [ φ ] , K = Ω( 1 ε 4 ) .

Original: Theorem 2 gives a guarantee that, for large enough K, n, s , the gap between the risk of the empirical risk minimizer ˆ φ and inf φ ∈ F p R [ φ ] can be arbitrarily small with high probability.
Revised: Theorem 2 gives a guarantee that, for large enough K, n, s , the gap between the risk of the empirical risk minimizer ˆ φ and inf φ ∈ F p R [ φ ] can be arbitrarily small with high probability.

Original: Note that we can solve ˆ φ to any given precision when l is convex.
Revised: Note that we can solve ˆ φ to any given precision when l is convex.

Original: Moreover, Theorem 2 reveals the accessibility of ˆ φ in Lemma 1 when l approximates the γ -robustness index well.
Revised: Moreover, Theorem 2 reveals the accessibility of ˆ φ in Lemma 1 when l approximates the γ -robustness index well.

Revised: While the number of candidate models and the number of training samples need to be large to ensure good theoretical properties, we will show that the performance of SWEEN models of practical settings is good enough in Section 5.

Original: 4. A DAPTIVE PREDICTION ALGORITHM
Revised: 4. A DAPTIVE PREDICTION ALGORITHM

Original: A major drawback of ensembling is the high execution cost during inference, which consists of prediction and certiﬁcation costs for smoothed classiﬁers.
Revised: A major drawback of ensembling is the high execution cost during inference, which consists of prediction and certiﬁcation costs for smoothed classiﬁers.

Original: The evaluation of smoothed classiﬁers relies on Monte Carlo simulation, which is computationally expensive.
Revised: The evaluation of smoothed classiﬁers relies on Monte Carlo simulation, which is computationally expensive.

Original: For instance, Cohen et al.
Revised: For instance, Cohen et al.

Original: (2019) use 100 Monte Carlo samples for prediction and 100,000 samples for certiﬁcation.
Revised: (2019) use 100 Monte Carlo samples for prediction and 100,000 samples for certiﬁcation.

Original: If we use 100 candidate models to construct a SWEEN model, the certiﬁcation of a single data point will require 10,010,000 local evaluations.
Revised: If we use 100 candidate models to construct a SWEEN model, the certiﬁcation of a single data point will require 10,010,000 local evaluations (10,000 for prediction and 10,000,000 for certiﬁcation).

Original: Inoue (2019) observes that ensembling does not make improvements for inputs predicted with high probabilities even when they are mispredicted.
Revised: Inoue (2019) observes that ensembling does not make improvements for inputs predicted with high probabilities even when they are mispredicted.

Original: He proposes an adaptive ensemble prediction algorithm to reduce the execution cost of unweighted ensemble models.
Revised: He proposes an adaptive ensemble prediction algorithm to reduce the execution cost of unweighted ensemble models.

Original: We modify the algorithm to make it applicative to weighted ensemble models, which is detailed in Algorithm 1.
Revised: We modify the algorithm to make it applicative to weighted ensemble models, which is detailed in Appendix B.1.

Original: For a data point, classiﬁers are evaluated in descending order with respect to their weights.
Revised: For a data point, classiﬁers are evaluated in descending order with respect to their weights.

Original: Whenever an early-exit condition is satisﬁed, we stop the evaluation and return the current prediction.
Revised: Whenever an early-exit condition is satisﬁed, we stop the evaluation and return the current prediction.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we design extensive experiments to evaluate the performance of SWEEN models.
Revised: In this section, we design extensive experiments to evaluate the performance of SWEEN models.

Original: 5. S ETUP Model setup
Revised: 5. S ETUP Model setup

Original: We train different network architectures on CIFAR-10 and SVHN to serve as candidates for ensembling, including LeNet (LeCun et al., 1989), AlexNet (Krizhevsky et al., 2017),
Revised: We train different network architectures on CIFAR-10 and SVHN to serve as candidates for ensembling, including LeNet (LeCun et al., 1989), AlexNet (Krizhevsky et al., 2017),

Original: ResNet-20 (He et al., 2016), ResNet-26, ResNet-32, ResNet-110, DenseNet (Huang et al., 2017) (depth=100), VGG-16 (Simonyan & Zisserman, 2015), and VGG-19.
Revised: ResNet-20 (He et al., 2016), ResNet-26, ResNet-32, ResNet-110, DenseNet (Huang et al., 2017) (depth=100), VGG-16 (Simonyan & Zisserman, 2015), and VGG-19.

Original: We particularly evaluate two compositions of SWEEN models.
Revised: We particularly evaluate two compositions of SWEEN models.

Original: The ﬁrst is a relatively rich set of models, including LeNet,
Revised: The ﬁrst is a relatively rich set of models, including LeNet,

Original: AlexNet, ResNet-20, ResNet-110, DenseNet, VGG-16, VGG-19, denoted by the SWEEN-7 model.
Revised: AlexNet, ResNet-20, ResNet-110, DenseNet, VGG-16, VGG-19, denoted by the SWEEN-7 model.

Original: The second is a small set of small models, including ResNet-20, ResNet-26, ResNet-32, denoted by the SWEEN-3 model.
Revised: The second is a small set of small models, including ResNet-20, ResNet-26, ResNet-32, denoted by the SWEEN-3 model.

Original: The SWEEN-7 model and the SWEEN-3 model simulate how much SWEEN can help in scenarios when we have adequate and limited numbers of candidate models, respectively.
Revised: The SWEEN-7 model and the SWEEN-3 model simulate how much SWEEN can help in scenarios when we have adequate and limited numbers of candidate models, respectively.

Original: A SWEEN model and its candidate models are trained and evaluated with the same noise level σ .
Revised: A SWEEN model and its candidate models are trained and evaluated with the same noise level σ .

Revised: The detailed algorithm for obtaining a SWEEN model is presented in Appendix B.2.

Original: Candidate model Training
Revised: Candidate model Training

Original: We train candidate models using two training schemes, including
Revised: We train candidate models using two training schemes, including

Original: Gaussian data augmentation training (Cohen et al., 2019), which is denoted as the standard training for simplicity, and MACER training (Zhai et al., 2020).
Revised: Gaussian data augmentation training (Cohen et al., 2019), which is denoted as the standard training for simplicity, and MACER training (Zhai et al., 2020).

Original: All hyper-parameters used in our experiments are listed in Appendix B.1.
Revised: All hyper-parameters used in our experiments are listed in Appendix C.1.

Original: Solving the ensembling weight
Revised: Solving the ensembling weight

Original: From Section 4 we know that we can obtain the empirical risk minimizer by solving a convex optimization.
Revised: From Section 4 we know that we can obtain the empirical risk minimizer by solving a convex optimization.

Original: However, this requires ﬁrst to approximate the value of smoothed functions of candidate models at every data point, which can be very costly when the number of candidates or training data samples is large.
Revised: However, this requires ﬁrst to approximate the value of smoothed functions of candidate models at every data point, which can be very costly when the number of candidates or training data samples is large.

Original: Hence, we use Gaussian data augmented training to solve the ensembling weight, which is much faster and yields comparable results empirically.
Revised: Hence, we use Gaussian data augmented training to solve the ensembling weight. More precisely, we freeze the parameters of candidate models and minimize the cross-entropy loss of the SWEEN model on Gaussian augmented data from the evaluation set. Empirically we ﬁnd that this approach is much faster and yields comparable results.

Original: Certiﬁcation
Revised: Certiﬁcation

Original: Following previous works, we report the approximated certiﬁed accuracy (ACA), which is the fraction of the test set that can be certiﬁed to be robust at radius r approximately (see
Revised: Following previous works, we report the approximated certiﬁed accuracy (ACA), which is the fraction of the test set that can be certiﬁed to be robust at radius r approximately (see

Original: Cohen et al.
Revised: Cohen et al.

Original: for more details).
Revised: for more details).

Original: We also report the average certiﬁed radius (ACR) following
Revised: We also report the average certiﬁed radius (ACR) following

Original: Zhai et al.
Revised: Zhai et al.

Original: The ACR equals to the area under the radius-accuracy curve (see Figure 1).
Revised: The ACR equals to the area under the radius-accuracy curve (see Figure 1).

Original: All results were certiﬁed using algorithms in Cohen et al.
Revised: All results were certiﬁed using algorithms in Cohen et al.

Original: (2019) with N = 100 , 000 samples and failure probability α = 0 .
Revised: (2019) with N = 100 , 000 samples and failure probability α = 0 .

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Due to space constraints, we only report the main results here.
Revised: Due to space constraints, we only report the main results here.

Original: Results of further experiments can be found in Appendix B.
Revised: Results of further experiments can be found in Appendix C.

Original: Standard training on CIFAR- 
Revised: Standard training on CIFAR-

Original: Table 1 displays the performance of two kinds of SWEEN models under noise levels σ ∈ { 0 . 25 , 0 .
Revised: Table 1 displays the performance of two kinds of SWEEN models under noise levels σ ∈ { 0 . 25 , 0 .

Original: The performance of a single ResNet-110 is included for comparison, and we also report the upper envelopes of the ACA and ACR of their corresponding candidate models as UE.
Revised: The performance of a single ResNet-110 is included for comparison, and we also report the upper envelopes of the ACA and ACR of their corresponding candidate models as UE.

Original: In Figure 1, we display the radius-accuracy curves for the
Revised: In Figure 1, we display the radius-accuracy curves for the

Original: SWEEN models and all their corresponding candidate models under σ = 0 .
Revised: SWEEN models and all their corresponding candidate models under σ = 0 .

Original: 50 on CIFAR-10.
Revised: 50 on CIFAR-10.

Revised: We also include full-size ﬁgures in Appendix D.

Revised: The results show that SWEEN models signiﬁcantly boost the performance compared to their corresponding candidate models.

Revised: According to Figure 1, the SWEEN-7 model consistently outperforms

Original: [Table]
Revised: [Table]

Original: SWEEN71.

Original: ResNet51.
Revised: ResNet51.

Original: UE SWEEN50.
Revised: UE SWEEN50.

Original: SWEEN51.
Revised: SWEEN51.

Original: UE SWEEN52.
Revised: UE SWEEN52.

Original: SWEEN52.
Revised: SWEEN52.

Original: The results show that SWEEN models signiﬁcantly boost the performance compared to their corresponding candidate models. According to Figure 1, the SWEEN-7 model consistently outperforms all its candidates in terms of the ACA at all radii.
Revised: all its candidates in terms of the ACA at all radii.

Original: The ACR of the SWEEN-7 model is 0.678, much higher than that of the upper envelope of the candidates, which is 0.574.
Revised: The ACR of the SWEEN-7 model is 0.678, much higher than that of the upper envelope of the candidates, which is 0.574.

Original: It conﬁrms our theoretical analysis in Section 4 that SWEEN can combine the strength of candidate models and attain superior performance.
Revised: It conﬁrms our theoretical analysis in Section 4 that SWEEN can combine the strength of candidate models and attain superior performance.

Original: Besides, SWEEN is effective when only limited numbers of small candidate models are available.
Revised: Besides, SWEEN is effective when only limited numbers of small candidate models are available.

Original: The SWEEN-3 model using ResNet-20, ResNet-26, and ResNet-32 achieves higher
Revised: The SWEEN-3 model using ResNet-20, ResNet-26, and ResNet-32 achieves higher

Original: ACA than the ResNet-110 at all radii on all noise levels.
Revised: ACA than the ResNet-110 at all radii on all noise levels.

Original: The total training time and the number of parameters of the SWEEN-3 model are 36 % and 30% less than those of ResNet-110, respectively.
Revised: The total training time and the number of parameters of the SWEEN-3 model are 36 % and 30% less than those of ResNet-110, respectively.

Original: The improvements can be further ampliﬁed by increasing the number and size of candidate models.
Revised: The improvements can be further ampliﬁed by increasing the number and size of candidate models.

Original: As an instance, the ACR of the SWEEN-7 model is at least 13% higher than that of the ResNetin Table 1.
Revised: As an instance, the ACR of the SWEEN-7 model is at least 13% higher than that of the ResNetin Table 1.

Original: The above results verify the effectiveness of SWEEN for randomized smoothing.
Revised: The above results verify the effectiveness of SWEEN for randomized smoothing.

Original: MACER training on CIFAR- 
Revised: MACER training on CIFAR-

Original: Since SWEEN is compatible with previous training algorithms, we adopt MACER training for the SWEEN-3 model.
Revised: Since SWEEN is compatible with previous training algorithms, we adopt MACER training for the SWEEN-3 model.

Original: The results are summarized in Table 6.
Revised: The results are summarized in Table 2.

Original: For the
Revised: For the

Original: ACA and ACR of the ResNet-110 model, we use the original numbers from Zhai et al.
Revised: ACA and ACR of the ResNet-110 model, we use the original numbers from Zhai et al.

Original: In the results, the SWEEN-3 model achieves comparable results to the ResNet-110 but is more efﬁcient.
Revised: In the results, the SWEEN-3 model achieves comparable results to the ResNet-110 but is more efﬁcient.

Original: From Table 6 and 3, the SWEEN-3 model takes 33.9 hours to achieve 0.727 in terms of the
Revised: From Table 2 and 3, the SWEEN-3 model takes 33.9 hours to achieve 0.727 in terms of the

Original: ACR for σ = 0 .
Revised: ACR for σ = 0 .

Original: 5 , using three small and easy-to-train candidates models.
Revised: 5 , using three small and easy-to-train candidates models.

Original: Meanwhile, it takes 49.
Revised: Meanwhile, it takes 49.

Original: hours for the ResNet-110 to achieve similar performance on CIFAR-10.
Revised: hours for the ResNet-110 to achieve similar performance on CIFAR-10.

Original: This 32% speed up reveals the efﬁciency of applying SWEEN to previous training methods.[Table] [Table]
Revised: This 32% speed up reveals the efﬁciency of applying SWEEN to previous training methods.

Original: Adaptive prediction ensembling
Revised: Adaptive prediction ensembling

Original: To alleviate the higher execution cost introduced by SWEEN [Table] we apply the previously mentioned adaptive prediction algorithm to speed up the certiﬁcation.
Revised: To alleviate the higher execution cost introduced by SWEEN, we apply the previously mentioned adaptive prediction algorithm to speed up the certiﬁcation.

Original: Experiments are conducted on the SWEEN-7 models via the standard training on CIFAR-10 and the results are summarized in Table 4.
Revised: Experiments are conducted on the SWEEN-7 models via the standard training on CIFAR-10 and the results are summarized in Table 4.

Original: It can be observed that the adaptive prediction successfully reduce the number of evaluations.
Revised: It can be observed that the adaptive prediction successfully [Table] All the experiments are run on a single NVIDIA 1080 Ti GPU. [Table]

Revised: reduce the number of evaluations.

Original: However, the performance of the adaptive prediction models is only slightly worse than their vanilla counterparts
Revised: However, the performance of the adaptive prediction models is only slightly worse than their vanilla counterparts

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work
Revised: In this work

Original: [Table]
Revised: [Table]

Original: we introduced the smoothed weighted ensembling (SWEEN) to improve randomized smoothed classiﬁers in terms of both accuracy and robustness.
Revised: we introduced the smoothed weighted ensembling (SWEEN) to improve randomized smoothed classiﬁers in terms of both accuracy and robustness.

Original: We showed that SWEEN can achieve optimal certiﬁed robustness w.r.t.
Revised: We showed that SWEEN can achieve optimal certiﬁed robustness w.r.t.

Original: our deﬁned γ -robustness index.
Revised: our deﬁned γ -robustness index.

Original: Furthermore, we can obtain the optimal SWEEN model w.r.t.
Revised: Furthermore, we can obtain the optimal SWEEN model w.r.t.

Original: a surrogate loss from training under mild assumptions.
Revised: a surrogate loss from training under mild assumptions.

Original: We also developed an adaptive prediction algorithm to accelerate the prediction and certiﬁcation process.
Revised: We also developed an adaptive prediction algorithm to accelerate the prediction and certiﬁcation process.

Original: Our extensive experiments showed that a properly designed SWEEN model was able to outperform all its candidate models by a signiﬁcant margin consistently.
Revised: Our extensive experiments showed that a properly designed SWEEN model was able to outperform all its candidate models by a signiﬁcant margin consistently.

Original: Moreover, SWEEN models using a few small and easy-to-train candidates could match or exceed a large individual model on performance with a notable reduction in total training time.
Revised: Moreover, SWEEN models using a few small and easy-to-train candidates could match or exceed a large individual model on performance with a notable reduction in total training time.

Original: Our theoretical and empirical results conﬁrmed that SWEEN is a viable tool for improving the performance of randomized smoothing models.
Revised: Our theoretical and empirical results conﬁrmed that SWEEN is a viable tool for improving the performance of randomized smoothing models.

Original: A P ROOFS A. P ROOF OF L EMMA
Revised: A P ROOFS A. P ROOF OF L EMMADeﬁne

Original: Deﬁne

Original: F (cid:48)

Original: [Equation]
Revised: [Equation]

Original: We have F p ⊆ F (cid:48) p , ˆ F θ ⊆
Revised: We have F p ⊆ F (cid:48) p , ˆ F θ ⊆

Original: ˆ F (cid:48) θ .
Revised: ˆ F (cid:48) θ .

Original: Lemma 2.
Revised: Lemma 2.

Original: Let µ be any probability measure on R d .
Revised: Let µ be any probability measure on R d .

Original: For φ : R d → R M , deﬁne the norm (cid:107) φ (cid:107) 2 µ (cid:44) (cid:82)
Revised: For φ : R d → R M , deﬁne the norm (cid:107) φ (cid:107) 2 µ (cid:44) (cid:82)

Original: R d (cid:107) φ ( x ) (cid:107) 22 dµ ( x ) .
Revised: R d (cid:107) φ ( x ) (cid:107) 22 dµ ( x ) .

Original: Fix φ ∈ F (cid:48) p , then for any η > 0 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: Fix φ ∈ F (cid:48) p , then for any η > 0 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˆ φ ( x ) =
Revised: from p , there exists ˆ φ ( x ) =

Original: K (cid:80)
Revised: K (cid:80)

Original: [Equation]
Revised: [Equation]

Original: g

Original: [Equation]
Revised: (cid:48)

Original:  which satisﬁes
Revised: θ which satisﬁes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Sine φ ∈ F (cid:48) p , we can write φ ( x ) =
Revised: Sine φ ∈ F (cid:48) p , we can write φ ( x ) =

Original: (cid:82) Θ w ( θ ) g ( x ; θ ) dθ , where w ( θ ) ≥ 0 .
Revised: (cid:82) Θ w ( θ ) g ( x ; θ ) dθ , where w ( θ ) ≥ 0 .

Original: Construct φ k = β k g ( · ; θ k ) , k = 1 , 2 , · · · , K , where β k = w ( θ k ) p ( θ k ) , then E φ k = φ, (cid:107) φ k (cid:107) µ = (cid:113)(cid:82) R d β 2 k (cid:107) g
Revised: Construct φ k = β k g ( · ; θ k ) , k = 1 , 2 , · · · , K , where β k = w ( θ k ) p ( θ k ) , then E φ k = φ, (cid:107) φ k (cid:107) µ = (cid:113)(cid:82) R d β 2 k (cid:107) g

Original: ( x ; θ k ) (cid:107) 22 dµ ( x ) ≤ | β k | ≤ (cid:107) φ (cid:107) p .
Revised: ( x ; θ k ) (cid:107) 22 dµ ( x ) ≤ | β k | ≤ (cid:107) φ (cid:107) p .

Original: We then deﬁne
Revised: We then deﬁne

Original: [Equation]
Revised: [Equation]

Original: First, by using Jensen’s inequality and the fact that (cid:107) φ k (cid:107) µ ≤ (cid:107) φ (cid:107) p , we have
Revised: First, by using Jensen’s inequality and the fact that (cid:107) φ k (cid:107) µ ≤ (cid:107) φ (cid:107) p , we have

Original: [Equation]
Revised: [Equation]

Original: Next, for θ 1 , · · · , θ M and ˜ θ i , we have
Revised: Next, for θ 1 , · · · , θ M and ˜ θ i , we have

Original: [Equation]
Revised: [Equation]

Original: Now we can use McDiarmid’s inequality to bound u ( θ )
Revised: Now we can use McDiarmid’s inequality to bound u ( θ )

Original: [Equation]
Revised: [Equation]

Original: which gives
Revised: which gives

Original: [Equation]
Revised: [Equation]

Original: The theorem follows by setting δ to the right hand side and solving ε .
Revised: The theorem follows by setting δ to the right hand side and solving ε .

Original: Lemma 3.
Revised: Lemma 3.

Original: Let µ be any probability measure on R d .
Revised: Let µ be any probability measure on R d .

Original: For φ : R d → R M , deﬁne the norm (cid:107) φ (cid:107) 2 µ (cid:44) (cid:82) R d (cid:107) φ ( x ) (cid:107) 22 dµ ( x ) , then for any η > 0 , for K ≥ M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 2 log 1 η ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: For φ : R d → R M , deﬁne the norm (cid:107) φ (cid:107) 2 µ (cid:44) (cid:82) R d (cid:107) φ ( x ) (cid:107) 22 dµ ( x ) , then for any η > 0 , for K ≥ M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 2 log 1 η ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˆ φ ( x ) =
Revised: from p , there exists ˆ φ ( x ) =

Original: K (cid:80)
Revised: K (cid:80)

Original: [Equation]
Revised: [Equation]

Original: which satisﬁes
Revised: which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix φ ∈ F p ⊆ F (cid:48) p , by using Lemma 2, we have that for any δ > 0 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: Fix φ ∈ F p ⊆ F (cid:48) p , by using Lemma 2, we have that for any δ > 0 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˜ φ ( x ) =
Revised: from p , there exists ˜ φ ( x ) =

Original: K (cid:80) k
Revised: K (cid:80) k

Original: [Equation]
Revised: [Equation]

Original: 1 c
Revised: 1 c

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: x ;
Revised: x ;

Original: [Equation]
Revised: [Equation]

Original: which satisﬁes
Revised: which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Denote C =
Revised: Denote C =

Original: K (cid:80) k
Revised: K (cid:80) k

Original: =1 c k , and deﬁne s ( t ) (cid:44)
Revised: =1 c k , and deﬁne s ( t ) (cid:44)

Original: M (cid:80) i =
Revised: M (cid:80) i =

Original: 1 t i as the sum of all elements of t ∈ R M .
Revised: 1 t i as the sum of all elements of t ∈ R M .

Original: Then s ( g ( x ; θ ))
Revised: Then s ( g ( x ; θ ))

Original: = 1 , ∀ x ∈ R d , θ ∈ Θ .
Revised: = 1 , ∀ x ∈ R d , θ ∈ Θ .

Original: Thus, s ( φ ( x )) =
Revised: Thus, s ( φ ( x )) =

Original: M (cid:88) i =
Revised: M (cid:88) i =

Original: φ i ( x ) =
Revised: φ i ( x ) =

Original: M (cid:88) i = (cid:90)
Revised: M (cid:88) i = (cid:90)

Original: Θ w ( θ ) g i ( x ; θ ) dθ =
Revised: Θ w ( θ ) g i ( x ; θ ) dθ =

Original: (cid:90) Θ w ( θ )
Revised: (cid:90) Θ w ( θ )

Original: M (cid:88) i =
Revised: M (cid:88) i =

Original: g i ( x ; θ ) dθ =
Revised: g i ( x ; θ ) dθ =

Original: (cid:90) Θ w ( θ ) dθ = 1 , s (˜ φ ( x )) =
Revised: (cid:90) Θ w ( θ ) dθ = 1 , s (˜ φ ( x )) =

Original: M (cid:88) i =
Revised: M (cid:88) i =

Original: ˜
Revised: ˜

Original: φ i ( x ) =
Revised: φ i ( x ) =

Original: M (cid:88) i =
Revised: M (cid:88) i =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = c
Revised: = c

Original: k g i
Revised: k g i

Original: ( x ; θ k ) =
Revised: ( x ; θ k ) =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = c k
Revised: = c k

Original: M (cid:88) i =
Revised: M (cid:88) i =

Original: g i ( x ; θ k ) =
Revised: g i ( x ; θ k ) =

Original: K (cid:88) k
Revised: K (cid:88) k

Original: = c k = C.
Revised: = c k = C.

Original: Now we have
Revised: Now we have

Original: [Equation]
Revised: [Equation]

Original: which gives 1 − √ MB ( K )
Revised: which gives 1 − √ MB ( K )

Original: <
Revised: <

Original: C < 1 + √ MB
Revised: C < 1 + √ MB

Original: ( K ) .
Revised: ( K ) .

Original: Construct ˆ φ ( x ) =
Revised: Construct ˆ φ ( x ) =

Original: ˜
Revised: ˜

Original: φ ( x ) C , then ˆ φ ∈ ˆ F θ and (cid:107) ˆ φ − φ (cid:107) 2 µ
Revised: φ ( x ) C , then ˆ φ ∈ ˆ F θ and (cid:107) ˆ φ − φ (cid:107) 2 µ

Original: [Equation]
Revised: [Equation]

Original: ) dµ ( x ) .
Revised: ) dµ ( x ) .

Original: Sine we have C 2 M ≤ (cid:107) ˜
Revised: Sine we have C 2 M ≤ (cid:107) ˜

Original: φ ( x ) (cid:107) 22 ≤ C 2 , |(cid:104) φ ( x ) , ˜ φ ( x ) (cid:105)| ≤ (cid:113) (cid:107) φ ( x ) (cid:107) 22 (cid:107) ˜
Revised: φ ( x ) (cid:107) 22 ≤ C 2 , |(cid:104) φ ( x ) , ˜ φ ( x ) (cid:105)| ≤ (cid:113) (cid:107) φ ( x ) (cid:107) 22 (cid:107) ˜

Original: φ ( x ) (cid:107) 22 ≤ C , it holds that
Revised: φ ( x ) (cid:107) 22 ≤ C , it holds that

Original: (i)
Revised: (i)

Original: When 1 < C < 1 + √ MB ( K ) ,
Revised: When 1 < C < 1 + √ MB ( K ) ,

Original: [Equation]
Revised: [Equation]

Original: (ii)
Revised: (ii)

Original: When
Revised: When

Original: 1 − √ MB
Revised: 1 − √ MB

Original: ( K ) <
Revised: ( K ) <

Original: C ≤ 1 ,
Revised: C ≤ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Thus, with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: Thus, with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p ,
Revised: from p ,

Original: [Equation]
Revised: [Equation]

Original: <
Revised: <

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: Suppose l ( · , · ) is L -Lipschitz in its ﬁrst argument.
Revised: Suppose l ( · , · ) is L -Lipschitz in its ﬁrst argument.

Original: Fix φ ∈ F p , then for any η > 0 , for
Revised: Fix φ ∈ F p , then for any η > 0 , for

Original: K ≥ M (cid:107)
Revised: K ≥ M (cid:107)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: there exists ˆ φ ∈ ˆ F θ which satisﬁes
Revised: there exists ˆ φ ∈ ˆ F θ which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: The desired result follows from Lemma 3.
Revised: The desired result follows from Lemma 3.

Original: Lemma 5.
Revised: Lemma 5.

Original: (Corollary of Proposition 1 in Zhai et al.
Revised: (Corollary of Proposition 1 in Zhai et al.

Original: Given any p 1 , p 2 , · · · , p M satisﬁes p 1 ≥ p 2 ≥ · · · ≥ p M ≥ 0 and p 1 + p 2 + · · · + p M = 1 .
Revised: Given any p 1 , p 2 , · · · , p M satisﬁes p 1 ≥ p 2 ≥ · · · ≥ p M ≥ 0 and p 1 + p 2 + · · · + p M = 1 .

Original: The derivative of clip( σ 2 [Φ − 1 ( p 1 ) − Φ − 1 ( p 2 )]; 0 , D ) with respect to p 1 and p 2 is bounded.
Revised: The derivative of clip( σ 2 [Φ − 1 ( p 1 ) − Φ − 1 ( p 2 )]; 0 , D ) with respect to p 1 and p 2 is bounded.

Original: Now we can prove Lemma 1.
Revised: Now we can prove Lemma 1.

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Let φ 0 ∈ F p such that I γ ( φ 0 ) >
Revised: Let φ 0 ∈ F p such that I γ ( φ 0 ) >

Original: sup φ ∈ F p I γ ( φ ) − ε 2 .
Revised: sup φ ∈ F p I γ ( φ ) − ε 2 .

Original: From Lemma 5 we know that q ( p, y ) (cid:44) clip( σ 2 [Φ − 1 ( p y ) − Φ − 1 (max k (cid:54) =
Revised: From Lemma 5 we know that q ( p, y ) (cid:44) clip( σ 2 [Φ − 1 ( p y ) − Φ − 1 (max k (cid:54) =

Original: y p k )]; 0 , D ) is Lipschitz in its ﬁrst argument.
Revised: y p k )]; 0 , D ) is Lipschitz in its ﬁrst argument.

Original: Since m is Lipschitz, c ( p, y ) (cid:44) m ( q ( p, y )) is also Lipschitz in its ﬁrst argument with some constant L .
Revised: Since m is Lipschitz, c ( p, y ) (cid:44) m ( q ( p, y )) is also Lipschitz in its ﬁrst argument with some constant L .

Original: Apply Lemma 4, we have that for K ≥ M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 1 + 2 log 1 δ ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: Apply Lemma 4, we have that for K ≥ M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 1 + 2 log 1 δ ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes
Revised: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: (ˆ φ ) =
Revised: (ˆ φ ) =

Original: E ( x,y ) ∼D
Revised: E ( x,y ) ∼D

Original: [ l ( φ 0 ( x ) , y )]
Revised: [ l ( φ 0 ( x ) , y )]

Original: [Equation]
Revised: [Equation]

Original: E ( x,y ) ∼D [ l
Revised: E ( x,y ) ∼D [ l

Original: [Equation]
Revised: [Equation]

Original: ( x ) , y )]
Revised: ( x ) , y )]

Original: [Equation]
Revised: [Equation]

Original: When K > 256 L 4 (cid:107) φ 0 (cid:107) 2 p M (1+ (cid:113) 2log 1 η )ε, we have
Revised: When K > 256 L 4 (cid:107) φ 0 (cid:107) 2 p M (1+ (cid:113) 2log 1 η )ε, we have

Original: [Equation]
Revised: [Equation]

Original: If I γ ( φ 0 ) = sup φ ∈ F p
Revised: If I γ ( φ 0 ) = sup φ ∈ F p

Original: I γ ( φ ) , which means (cid:107) φ 0 (cid:107) p is independent of ε , K = Ω( 1 ε 4 ) .
Revised: I γ ( φ ) , which means (cid:107) φ 0 (cid:107) p is independent of ε , K = Ω( 1 ε 4 ) .

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: First we introduce some results from statistical learning theory.
Revised: First we introduce some results from statistical learning theory.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: (Gaussian complexity).
Revised: (Gaussian complexity).

Original: Let µ be a probability distribution on a set X and suppose that x 1 , ..., x n are independent samples selected according to µ .
Revised: Let µ be a probability distribution on a set X and suppose that x 1 , ..., x n are independent samples selected according to µ .

Original: Let F be a class of functions mapping from X to R .
Revised: Let F be a class of functions mapping from X to R .

Original: The Gaussian complexity of F is
Revised: The Gaussian complexity of F is

Original: [Equation]
Revised: [Equation]

Original: where ξ 1 , ..., ξ n are independent N (0 , 1) random variables.
Revised: where ξ 1 , ..., ξ n are independent N (0 , 1) random variables.

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: (Rademacher complexity) Let µ be a probability distribution on a set X and suppose that x 1 , ..., x n are independent samples selected according to µ .
Revised: (Rademacher complexity) Let µ be a probability distribution on a set X and suppose that x 1 , ..., x n are independent samples selected according to µ .

Original: Let F be a class of functions mapping from X to R .
Revised: Let F be a class of functions mapping from X to R .

Original: The Rademacher complexity of F is
Revised: The Rademacher complexity of F is

Original: [Equation]
Revised: [Equation]

Original: where σ 1 , ..., σ n are independent uniform {± 1 } -valued random variables.
Revised: where σ 1 , ..., σ n are independent uniform {± 1 } -valued random variables.

Original: Lemma 6.
Revised: Lemma 6.

Original: (Part of Lemma 4 in Bartlett & Mendelson (2001)).
Revised: (Part of Lemma 4 in Bartlett & Mendelson (2001)).

Original: There are absolute constants β such that for every class F and every integer n , R n ( F ) ≤ βG n ( F ) .
Revised: There are absolute constants β such that for every class F and every integer n , R n ( F ) ≤ βG n ( F ) .

Original: Lemma 7.
Revised: Lemma 7.

Original: (Corollary of Theorem 8 in Bartlett & Mendelson (2001)).
Revised: (Corollary of Theorem 8 in Bartlett & Mendelson (2001)).

Original: Consider a loss function c : A × Y → [0 , 1] .
Revised: Consider a loss function c : A × Y → [0 , 1] .

Original: Let F be a class of functions mapping from X to A and let ( x i , y i ) ni
Revised: Let F be a class of functions mapping from X to A and let ( x i , y i ) ni

Original: =1 be independently selected according to the probability measure µ .
Revised: =1 be independently selected according to the probability measure µ .

Original: Then, for any integer n and any 0 < η < 1 , with probability at least 1 − η over samples of length n , every f in F satisﬁes
Revised: Then, for any integer n and any 0 < η < 1 , with probability at least 1 − η over samples of length n , every f in F satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where ˜ c ◦ F = { ( x, y ) (cid:55)→ c ( f ( x ) , y ) − c (0 , y ) (cid:12)(cid:12) f ∈ F }
Revised: where ˜ c ◦ F = { ( x, y ) (cid:55)→ c ( f ( x ) , y ) − c (0 , y ) (cid:12)(cid:12) f ∈ F }

Original: Lemma 8.
Revised: Lemma 8.

Original: (Corollary of Theorem 14 in Bartlett & Mendelson (2001)).
Revised: (Corollary of Theorem 14 in Bartlett & Mendelson (2001)).

Original: Let A = R M and let F be a class of functions mapping from X to A .
Revised: Let A = R M and let F be a class of functions mapping from X to A .

Original: Suppose that there are real-valued classes F 1 , ..., F M such that F is a subset of their Cartesian product.
Revised: Suppose that there are real-valued classes F 1 , ..., F M such that F is a subset of their Cartesian product.

Original: Assume further that c : A × Y → R is such that, for all y ∈ Y , c ( · , y ) is a Lipschitz function with constant L which passes through the origin and is uniformly bounded.
Revised: Assume further that c : A × Y → R is such that, for all y ∈ Y , c ( · , y ) is a Lipschitz function with constant L which passes through the origin and is uniformly bounded.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Now we prove the following lemma:
Revised: Now we prove the following lemma:

Original: Lemma 9.
Revised: Lemma 9.

Original: Let c, F , ( x i , y i )
Revised: Let c, F , ( x i , y i )

Original: ni
Revised: ni

Original: =1 , ˜ c ◦ F be as in Lemma 7.
Revised: =1 , ˜ c ◦ F be as in Lemma 7.

Original: Then, for any integer n and any 0 < η < 1 , with probability at least 1 − η over samples of length n , every f in F satisﬁes
Revised: Then, for any integer n and any 0 < η < 1 , with probability at least 1 − η over samples of length n , every f in F satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: When an ( x i , y i ) pair changes, the random variable sup h ∈ ˜ c ◦ F (ˆ E n h − E h ) can change by no more than 2 n .
Revised: When an ( x i , y i ) pair changes, the random variable sup h ∈ ˜ c ◦ F (ˆ E n h − E h ) can change by no more than 2 n .

Original: McDiarmid’s inequality implies that with probability at least 1 − η 2 ,
Revised: McDiarmid’s inequality implies that with probability at least 1 − η 2 ,

Original: [Equation]
Revised: [Equation]

Original: A similar argument, together with the fact that E ˆ E n c (0 , y )
Revised: A similar argument, together with the fact that E ˆ E n c (0 , y )

Original: =
Revised: =

Original: E c (0 , y ) , shows that with probability at least 1 − η ,
Revised: E c (0 , y ) , shows that with probability at least 1 − η ,

Original: [Equation]
Revised: [Equation]

Original: It’s left to show that E sup h ∈ ˜ c ◦ F (ˆ E n h − E h ) ≤ R n [˜ c ◦ F ] .
Revised: It’s left to show that E sup h ∈ ˜ c ◦ F (ˆ E n h − E h ) ≤ R n [˜ c ◦ F ] .

Original: Let ( x (cid:48) 1 , y (cid:48) 1 ) , ..., ( x (cid:48) n , y
Revised: Let ( x (cid:48) 1 , y (cid:48) 1 ) , ..., ( x (cid:48) n , y

Original: [Equation]
Revised: [Equation]

Original: n ) be drawn i.i.d.
Revised: n ) be drawn i.i.d.

Original: from µ and independent from ( x i , y i ) ni =1 , then
Revised: from µ and independent from ( x i , y i ) ni =1 , then

Original: [Equation]
Revised: [Equation]

Original: EE sup h ∈ ˜ c ◦ F [ ˆ E n h − 1 n n (cid:88) i =
Revised: EE sup h ∈ ˜ c ◦ F [ ˆ E n h − 1 n n (cid:88) i =

Original: h ( x (cid:48)
Revised: h ( x (cid:48)

Original: i , y (cid:48) i )]
Revised: i , y (cid:48) i )]

Original: =
Revised: =

Original: E sup h ∈ ˜
Revised: E sup h ∈ ˜

Original: c
Revised: c

Original: ◦ F n ( n (cid:88) i = h ( x i , y i ) − n (cid:88) i
Revised: ◦ F n ( n (cid:88) i = h ( x i , y i ) − n (cid:88) i

Original: = h ( x (cid:48)
Revised: = h ( x (cid:48)

Original: i , y (cid:48) i ))
Revised: i , y (cid:48) i ))

Original: ≤ 2 E sup h ∈ ˜
Revised: ≤ 2 E sup h ∈ ˜

Original: c ◦ F n n (cid:88) i =
Revised: c ◦ F n n (cid:88) i =

Original: σ i h ( x i , y i )
Revised: σ i h ( x i , y i )

Original: ≤ R n [˜ c ◦ F ] .
Revised: ≤ R n [˜ c ◦ F ] .

Original: We can prove the following result:
Revised: We can prove the following result:

Original: Theorem 3.
Revised: Theorem 3.

Original: Let A = R M and let F be a class of functions mapping from X to A .
Revised: Let A = R M and let F be a class of functions mapping from X to A .

Original: Suppose that there are real-valued classes F 1 , ..., F M such that F is a subset of their Cartesian product.
Revised: Suppose that there are real-valued classes F 1 , ..., F M such that F is a subset of their Cartesian product.

Original: Assume further that the loss function c : A × Y → R is such that, for all y ∈ Y , c ( · , y ) is a Lipschitz function with constant Lwhich passes through the origin and is uniformly bounded.
Revised: Assume further that the loss function c : A × Y → R is such that, for all y ∈ Y , c ( · , y ) is a Lipschitz function with constant L and is uniformly bounded.

Original: Let { ( x i , y i ) } ni
Revised: Let { ( x i , y i ) } ni

Original: =1 be independently selected according to the probability measure µ .
Revised: =1 be independently selected according to the probability measure µ .

Original: Then, for any integer n and any 0 < η < 1 , there is a probability of at least 1 − η that every f ∈ F has
Revised: Then, for any integer n and any 0 < η < 1 , there is a probability of at least 1 − η that every f ∈ F has

Original: [Equation]
Revised: [Equation]

Original: where β is a constant.
Revised: where β is a constant.

Original: Proof.
Revised: Proof.

Original: From Lemma 7 and 9 we have that with probability at least 1 − η over samples of length n , every f in F satisﬁes
Revised: From Lemma 7 and 9 we have that with probability at least 1 − η over samples of length n , every f in F satisﬁes

Original: [Equation]
Revised: [Equation]

Original: it follows by applying Lemma 6 and 8.
Revised: it follows by applying Lemma 6 and 8.

Original: Lemma 10.
Revised: Lemma 10.

Original: Let c ( · , · ) , β be as in Theorem 3.
Revised: Let c ( · , · ) , β be as in Theorem 3.

Original: Let ( x i , y i ) ni
Revised: Let ( x i , y i ) ni

Original: =1 be independently selected according to the probability measure D .
Revised: =1 be independently selected according to the probability measure D .

Original: For any integer n and any 0 < η < 1 , there is a probability of at least 1 − η that every f ∈ ˆ F θ has
Revised: For any integer n and any 0 < η < 1 , there is a probability of at least 1 − η that every f ∈ ˆ F θ has

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: We have that ˆ F θ ⊆
Revised: We have that ˆ F θ ⊆

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: k = ˆ F θ ( i ) , where (cid:78) stands for a Cartesian product operation.
Revised: k = ˆ F θ ( i ) , where (cid:78) stands for a Cartesian product operation.

Original: The Gaussian comlexities of ˆ F θ ( i ) ’s can be bounded as
Revised: The Gaussian comlexities of ˆ F θ ( i ) ’s can be bounded as

Original: [Equation]
Revised: [Equation]

Original: The desired result follows by applying Theorem 3 to ˆ F θ , ˆ F θ (1) , · · · , ˆ F θ ( M ) and D .
Revised: The desired result follows by applying Theorem 3 to ˆ F θ , ˆ F θ (1) , · · · , ˆ F θ ( M ) and D .

Original: Next, we give the deﬁnition of semi-empirical risk.
Revised: Next, we give the deﬁnition of semi-empirical risk.

Original: The term ”semi-” implies that it is empirical with respect to the training set but not the smoothing operation.
Revised: The term ”semi-” implies that it is empirical with respect to the training set but not the smoothing operation.

Original: Deﬁnition 5.
Revised: Deﬁnition 5.

Original: (Semi-empirical risk).
Revised: (Semi-empirical risk).

Original: For a surrogate loss function l ( · , · ) :
Revised: For a surrogate loss function l ( · , · ) :

Original: R M × R M
Revised: R M × R M

Original: → R and training set { ( x i , y i ) } ni =1 , the semi-empirical risk of φ ( x ) =
Revised: → R and training set { ( x i , y i ) } ni =1 , the semi-empirical risk of φ ( x ) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 w k g ( x ; θ k ) ∈
Revised: Kk =1 w k g ( x ; θ k ) ∈

Original: ˆ F θ are deﬁned as
Revised: ˆ F θ are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We can use Lemma 4 and 10 to prove the following result:
Revised: We can use Lemma 4 and 10 to prove the following result:

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose for all y ∈ Y , l ( · , y ) is a Lipschitz function with constant Lwhich passes through the origin and is uniformly bounded.
Revised: Suppose for all y ∈ Y , l ( · , y ) is a Lipschitz function with constant L and is uniformly bounded.

Original: Fix φ ∈ F p , then for any η > 0 , with probability at least 1 − η over the training dataset { ( x i , y i ) } ni =1 drawn i.i.d.
Revised: Fix φ ∈ F p , then for any η > 0 , with probability at least 1 − η over the training dataset { ( x i , y i ) } ni =1 drawn i.i.d.

Original: from D and the parameters θ 1 , ..., θ K drawn i.i.d.
Revised: from D and the parameters θ 1 , ..., θ K drawn i.i.d.

Original: from p , the semi-empirical risk minimizer ˆ φ over ˆ F θ satisﬁes
Revised: from p , the semi-empirical risk minimizer ˆ φ over ˆ F θ satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where β is a constant.
Revised: where β is a constant.

Original: Proof.
Revised: Proof.

Original: Let φ ∗ be the minimizer of R over ˆ F θ .
Revised: Let φ ∗ be the minimizer of R over ˆ F θ .

Original: Combine Lemma 4 and 10, we derive that, with probability at least 1 − 2 δ over the training dataset and the choice of the parameters θ 1 , ..., θ K ,
Revised: Combine Lemma 4 and 10, we derive that, with probability at least 1 − 2 δ over the training dataset and the choice of the parameters θ 1 , ..., θ K ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 11.
Revised: Lemma 11.

Original: Let µ be a probability distribution on ∆ .
Revised: Let µ be a probability distribution on ∆ .

Original: For any η > 0 , with probability at least 1 − η over x 1 , ..., x s drawn i.i.d.
Revised: For any η > 0 , with probability at least 1 − η over x 1 , ..., x s drawn i.i.d.

Original: from µ , it holds that
Revised: from µ , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne u ( x 1 , · · · , x s ) =
Revised: Deﬁne u ( x 1 , · · · , x s ) =

Original: (cid:107) 1 s (cid:80) si =1 x i − E [ x ] (cid:107) 2 .
Revised: (cid:107) 1 s (cid:80) si =1 x i − E [ x ] (cid:107) 2 .

Original: By using Jensen’s inequality, we have
Revised: By using Jensen’s inequality, we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Next, for x 1 , · · · , x M and ˜ x k , we have
Revised: Next, for x 1 , · · · , x M and ˜ x k , we have

Original: [Equation]
Revised: [Equation]

Original: Now we can use McDiarmid’s inequality to bound u ( x ) , which gives
Revised: Now we can use McDiarmid’s inequality to bound u ( x ) , which gives

Original: [Equation]
Revised: [Equation]

Original: The result follows by setting η to the right hand side and solving ε .
Revised: The result follows by setting η to the right hand side and solving ε .

Original: Now we are ready to prove Theorem 2.
Revised: Now we are ready to prove Theorem 2.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Let φ 0 ∈ F p such that R [ φ 0 ] < inf φ ∈ F p R [ φ ] + ε 4 .
Revised: Let φ 0 ∈ F p such that R [ φ 0 ] < inf φ ∈ F p R [ φ ] + ε 4 .

Original: By Lemma 11, with probability at least 1 − η 3 ,
Revised: By Lemma 11, with probability at least 1 − η 3 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: hold simultaneously.
Revised: hold simultaneously.

Original: So with probability at least 1 − η 3 , for every φ =
Revised: So with probability at least 1 − η 3 , for every φ =

Original: [Equation]
Revised: [Equation]

Original:  = 
Revised: Kk = [Equation] x

Original: [Equation]
Revised: [Equation]

Original:  k ) 
Revised: θ k ) ∈

Original: [Equation]
Revised: [Equation]

Original: F θ , it holds that
Revised: F θ , it holds that

Original: [Equation]
Revised: [Equation]

Original: By Lemma 10, with probability at least 1 − η 3 , for every φ ∈ ˆ F θ , it holds that
Revised: By Lemma 10, with probability at least 1 − η 3 , for every φ ∈ ˆ F θ , it holds that

Original: [Equation]
Revised: [Equation]

Original: Let φ ∗ be the minimizer of R over ˆ F θ .
Revised: Let φ ∗ be the minimizer of R over ˆ F θ .

Original: By Lemma 4, with probability at least 1 − δ 3 , for K ≥
Revised: By Lemma 4, with probability at least 1 − δ 3 , for K ≥

Original: M (cid:107) φ
Revised: M (cid:107) φ

Original: [Equation]
Revised: [Equation]

Original: So with probability at least 1 − η , it holds that
Revised: So with probability at least 1 − η , it holds that

Original: [Equation]
Revised: [Equation]

Original: When K > 256 L 4 (cid:107) φ 0 (cid:107) 2 p M (1+ (cid:113)2log 1 η )ε, n > [Equation] [Equation]
Revised: When K > 256 L 4 (cid:107) φ 0 (cid:107) 2 p M (1+ (cid:113)

Original: [Equation]
Revised: [Equation]

Original: ε

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: If R [ φ 0 ] = inf φ ∈ F p R [ φ ] , which means (cid:107) φ 0 (cid:107) p is independent of ε , K = Ω( 1 ε 4 ) .
Revised: If R [ φ 0 ] = inf φ ∈ F p R [ φ ] , which means (cid:107) φ 0 (cid:107) p is independent of ε , K = Ω( 1 ε 4 ) .

Revised: B A LGORITHMS B. A DAPTIVE PREDICTION ALGORITHM

Revised: The entire adaptive prediction algorithm is shown in Algorithm 1.

Revised: It is modiﬁed from Inoue (2019) to generalize to weighted ensembles.

Revised: The exit condition is the weighted version of the conﬁdence-levelbased early-exit condition in Inoue (2019).

Original: B S UPPLEMENTARY MATERIAL FOR EXPERIMENTS B. D ETAILED SETTINGS AND HYPER - PARAMETERS
Revised: The algorithm accelerates the evaluation of the ensemble function f and the smoothed operation g ( x ) = E δ [ f ( x + δ )] remains the same, so it does not affect the Monte Carlo estimation and certiﬁcation procedure of smoothed classiﬁers. .., θ K C S UPPLEMENTARY MATERIAL FOR EXPERIMENTS C. D ETAILED SETTINGS AND HYPER - PARAMETERS

Original: We perform all experiments on a single GeForce GTX 1080 Ti GPU.
Revised: We perform all experiments on a single GeForce GTX 1080 Ti GPU.

Original: For training the SWEEN models, we divide the training set into two parts, one for training candidate models, and the other for solving weights.
Revised: For training the SWEEN models, we divide the training set into two parts, one for training candidate models, and the other for solving weights.

Original: We employ 2,000 images for solving weights on CIFARand 3000 images for solving weights on SVHN.
Revised: We employ 2,000 images for solving weights on CIFARand 3000 images for solving weights on SVHN.

Original: For Gaussian data augmentation training, all the models are trained for 400 epochs using SGD.
Revised: For Gaussian data augmentation training, all the models are trained for 400 epochs using SGD.

Original: The learning rate is initialized set as 0.01, and decayed by 0.1 at the 150th/300th epoch.
Revised: The learning rate is initialized set as 0.01, and decayed by 0.1 at the 150th/300th epoch.

Original: For MACER training, we use the same hyper-parameters as Zhai et al.
Revised: For MACER training, we use the same hyper-parameters as Zhai et al.

Original: (2020), i.e., we use k = 16 , β = 16 . 0 , γ = 8 . 0 , and we use λ = 12 . 

Revised: (2020), i.e., we use k = 16 , β = 16 . 0 , γ = 8 . 0 , and we use λ = 12 .

Original: for σ = 0 .
Revised: for σ = 0 .

Original: 25 and λ = 4 . 

Revised: 25 and λ = 4 .

Original: for σ = 0 .
Revised: for σ = 0 .

Original: We train the models for 440 epochs, the learning rate is initialized set as 0.01, and decayed by 1 at the 200th/400th epoch.
Revised: We train the models for 440 epochs, the learning rate is initialized set as 0.01, and decayed by 0.1 at the 200th/400th epoch.

Original: B. R ESULTS ON SVHN
Revised: C. R ESULTS ON SVHN

Original: To further evaluate our method we also experiment on SVHN.
Revised: To further evaluate our method, we also experiment on SVHN.

Original: The results in Table 5 show that
Revised: The results in Table 5 show that

Original: SWEEN models outperform the upper envelopes of their corresponding candidate models as well.
Revised: SWEEN models outperform the upper envelopes of their corresponding candidate models as well.

Original: Figure 2 plots the results on CIFAR-10 and SVHN for comparison.[Table]
Revised: Figure 2 plots the results on CIFAR-10 and SVHN for comparison.

Original: B. SWEEN MODELS USING CANDIDATES WITH IDENTICAL ARCHITECTURES
Revised: C. SWEEN MODELS USING CANDIDATES WITH IDENTICAL ARCHITECTURES

Original: The SWEEN-3 and SWEEN-7 models are all using candidate models with diverse architectures.
Revised: The SWEEN-3 and SWEEN-7 models are all using candidate models with diverse architectures.

Original: For a more comprehensive result, we also experiment with SWEEN models using candidates with identical  architectures.
Revised: For a more comprehensive result, we also experiment with SWEEN models using candidates with identical [Table] architectures.

Original: For σ ∈ { 0 .
Revised: For σ ∈ { 0 .

Original: , We train 8 ResNet-110 models using different random seeds on CIFAR-10 via the standard training.
Revised: , We train 8 ResNet-110 models using different random seeds on CIFAR-10 via the standard training.

Original: We then use these models to ensemble SWEEN models.
Revised: We then use these models to ensemble SWEEN models.

Original: The results are shown in Table 6 and Figure 3.
Revised: The results are shown in Table 6 and Figure 3.

Original: We can see that SWEEN is still effective in this scenario and signiﬁcantly boosts the performance compared to candidate models.[Table] SWEEN 54.
Revised: We can see that SWEEN is still effective in this scenario and signiﬁcantly boosts the performance compared to candidate models.

Original: B. MACER TRAINING ON CIFAR- 
Revised: C. MACER TRAINING ON CIFAR-

Original: In this section, we plot the radius-accuracy curves for SWEEN models with candidate models trained by MACER on CIFAR-10 in Figure 4.[Figure]
Revised: In this section [Table] we plot the radius-accuracy curves for SWEEN models with candidate models trained by MACER on CIFAR-10 in Figure 4.

Original:  T HE SCATTER PLOT OF THE CERTIFIED ACCURACY
Revised: C. T HE SCATTER PLOT OF THE CERTIFIED ACCURACY

Original: This section plots the scatter diagram of the certiﬁed accuracy of test data points for SWEEN-7 versus ResNet-110.
Revised: This section plots the scatter diagram of the certiﬁed accuracy of test data points for SWEENversus ResNet-110 in Figure 5.

Original: Most of the points lie under the line y
Revised: Most of the points lie under the line y

Original: =
Revised: =

Original: x , implying that SWEEN-7 performs superior to ResNet-110. 
Revised: x , implying that SWEENperforms superior to ResNet-110. [Table]

Revised: C. SWEEN VERSUS ADVERSARIAL ATTACKS

Revised: We further investigate the performance of SWEEN models versus AutoAttack (Croce & Hein, 2020), which is an ensemble of four diverse attacks to reliably evaluate robustness.

Revised: Similar to Salman et al.

Revised: (2019a)

Revised: [Table]

Revised: we used 128 samples to estimate the smoothed classiﬁer.

Revised: We share the results below in Table 7.

Revised: It can be seen that SWEEN can improve the empirical robustness as well.

Revised: [Table]

Revised: D F IGURES