Original: Enhancing Certiﬁed Robustness via Smoothed Weighted Ensembling
Revised: Enhancing Certiﬁed Robustness via Smoothed Weighted Ensembling

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: Randomized smoothing has achieved state-of-theart certiﬁed robustness against l 2 -norm adversarial attacks.
Revised: Randomized smoothing has achieved state-of-theart certiﬁed robustness against l 2 -norm adversarial attacks.

Original: However, it is not wholly resolved on how to ﬁnd the optimal base classiﬁer for randomized smoothing.
Revised: However, it is not wholly resolved on how to ﬁnd the optimal base classiﬁer for randomized smoothing.

Original: In this work, we employ a Smoothed WEighted ENsembling (SWEEN) scheme to improve the performance of randomized smoothed classiﬁers.
Revised: In this work, we employ a Smoothed WEighted ENsembling (SWEEN) scheme to improve the performance of randomized smoothed classiﬁers.

Original: We show the ensembling generality that SWEEN can help achieve optimal certiﬁed robustness.
Revised: We show the ensembling generality that SWEEN can help achieve optimal certiﬁed robustness.

Original: Furthermore, theoretical analysis proves that the optimal SWEEN model can be obtained from training under mild assumptions.
Revised: Furthermore, theoretical analysis proves that the optimal SWEEN model can be obtained from training under mild assumptions.

Original: We also develop an adaptive prediction algorithm to reduce the prediction and certiﬁcation cost of SWEEN models.
Revised: We also develop an adaptive prediction algorithm to reduce the prediction and certiﬁcation cost of SWEEN models.

Original: Extensive experiments show that SWEEN models outperform the upper envelope of their corresponding candidate models by a large margin.
Revised: Extensive experiments show that SWEEN models outperform the upper envelope of their corresponding candidate models by a large margin.

Original: Moreover,
Revised: Moreover,

Original: SWEEN models constructed using a few small models can achieve comparable performance to a single large model with a notable reduction in training time.
Revised: SWEEN models constructed using a few small models can achieve comparable performance to a single large model with a notable reduction in training time.

Original: 1. Introduction
Revised: 1. Introduction

Original: Deep neural networks have achieved great success in image classiﬁcation tasks.
Revised: Deep neural networks have achieved great success in image classiﬁcation tasks.

Original: However, they are vulnerable to adversarial examples , which are small imperceptible perturbations on the original inputs that can cause misclassiﬁcation (Biggio et al., 2013; Szegedy et al., 2014).
Revised: However, they are vulnerable to adversarial examples , which are small imperceptible perturbations on the original inputs that can cause misclassiﬁcation (Biggio et al., 2013; Szegedy et al., 2014).

Original: To tackle this problem, researchers have proposed various defense methods to train classiﬁers robust to adversarial perturbations.
Revised: To tackle this problem, researchers have proposed various defense methods to train classiﬁers robust to adversarial perturbations.

Original: Recently, a new certiﬁed defense technique called randomized smoothing has been proposed (Lecuyer et al., 2019; 
Revised: Recently, a new certiﬁed defense technique called randomized smoothing has been proposed (Lecuyer et al., 2019; 1 Center for Data Science, Peking University 2 School of Mathematical Sciences, Peking University 3 National School of Development, Peking University 4 Beijing International Center for Mathematical Research 5 Institute for Artiﬁcial Inteligence, Peking University.

Original:  Cohen et al., 2019).
Revised: Correspondence to: Bin Dong < dong- bin@math.pku.edu.cn > . Accepted by the ICML 2021 workshop on A Blessing in Disguise: The Prospects and Perils of Cohen et al., 2019).

Original: A (randomized) smoothed classiﬁer is constructed from a base classiﬁer, typically a deep neural network.
Revised: A (randomized) smoothed classiﬁer is constructed from a base classiﬁer, typically a deep neural network.

Original: It outputs the most probable class given by its base classiﬁer under a random noise perturbation of the input.
Revised: It outputs the most probable class given by its base classiﬁer under a random noise perturbation of the input.

Original: Randomized smoothing is scalable due to its independency over architectures and has achieved state-of-the-art certiﬁed l 2 -robustness.
Revised: Randomized smoothing is scalable due to its independency over architectures and has achieved state-of-the-art certiﬁed l 2 -robustness.

Original: In theory, randomized smoothing can apply to any classiﬁers.
Revised: In theory, randomized smoothing can apply to any classiﬁers.

Original: However, naively using randomized smoothing on standard-trained classiﬁers leads to poor robustness results.
Revised: However, naively using randomized smoothing on standard-trained classiﬁers leads to poor robustness results.

Original: It is still not wholly resolved on how a base classiﬁer should be trained so that the corresponding smoothed classiﬁer has good robustness properties.
Revised: It is still not wholly resolved on how a base classiﬁer should be trained so that the corresponding smoothed classiﬁer has good robustness properties.

Original: Recently, Salman et al.
Revised: Recently, Salman et al.

Original: (2019a) employ adversarial training to train base classiﬁers and substantially improve the performance of randomized smoothing, which indicates that techniques originally proposed for empirical defenses can be useful in ﬁnding good base classiﬁers for randomized smoothing.
Revised: (2019a) employ adversarial training to train base classiﬁers and substantially improve the performance of randomized smoothing, which indicates that techniques originally proposed for empirical defenses can be useful in ﬁnding good base classiﬁers for randomized smoothing.

Original: In this paper, we take a step towards ﬁnding suitable base models for randomized smoothing by model ensembling.
Revised: In this paper, we take a step towards ﬁnding suitable base models for randomized smoothing by model ensembling.

Original: The idea of model ensembling has been used in various empirical defenses against adversarial examples and shows promising results for robustness (Liu et al., 2018; Strauss et al., 2018;
Revised: The idea of model ensembling has been used in various empirical defenses against adversarial examples and shows promising results for robustness (Liu et al., 2018; Strauss et al., 2018;

Original: Pang et al., 2019; Wang et al., 2019; Meng et al., 2020; Sen et al., 2020).
Revised: Pang et al., 2019; Wang et al., 2019; Meng et al., 2020; Sen et al., 2020).

Original: Moreover, an ensemble can combine the strengths of candidate models 1 to achieve superior clean accuracy (Hansen & Salamon, 1990; Krogh & Vedelsby, 1994).
Revised: Moreover, an ensemble can combine the strengths of candidate models 1 to achieve superior clean accuracy (Hansen & Salamon, 1990; Krogh & Vedelsby, 1994).

Original: Thus, we believe ensembling several smoothed models can help improve both the robustness and accuracy.
Revised: Thus, we believe ensembling several smoothed models can help improve both the robustness and accuracy.

Original: Speciﬁcally for randomized smoothing, the smoothing operator is commutative with the ensembling operator: ensembling several smoothed models is equivalent to smoothing an ensembled base model.
Revised: Speciﬁcally for randomized smoothing, the smoothing operator is commutative with the ensembling operator: ensembling several smoothed models is equivalent to smoothing an ensembled base model.

Original: This property makes the combination suitable and efﬁcient.
Revised: This property makes the combination suitable and efﬁcient.

Original: Therefore, we directly ensemble a base model by taking some pre-trained models as candidates and optimizing the optimal weights for randomized smoothing.
Revised: Therefore, we directly ensemble a base model by taking some pre-trained models as candidates and optimizing the optimal weights for randomized smoothing.

Original: We refer to the ﬁnal model as a
Revised: We refer to the ﬁnal model as a

Original: Smoothed WEighted ENsembling (SWEEN) model.
Revised: Smoothed WEighted ENsembling (SWEEN) model.

Original: Moreover, SWEEN does not limit how individual candidate classiﬁers are trained, thus is compatible with most previously proposed training algorithms on randomized smoothing.
Revised: Moreover, SWEEN does not limit how individual candidate classiﬁers are trained, thus is compatible with most previously proposed training algorithms on randomized smoothing.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: We propose SWEEN to substantially improve the per- formance of smoothed models.
Revised: We propose SWEEN to substantially improve the performance of smoothed models.

Original: Theoretical analysis shows the ensembling generality and the optimization guarantee of SWEEN.
Revised: Theoretical analysis shows the ensembling generality and the optimization guarantee of SWEEN.

Original: We develop an adaptive prediction algorithm for the weighted ensembling, which effectively reduces the prediction and certiﬁcation cost of the smoothed ensemble classiﬁer.
Revised: We develop an adaptive prediction algorithm for the weighted ensembling, which effectively reduces the prediction and certiﬁcation cost of the smoothed ensemble classiﬁer.

Original: We evaluate our proposed method through extensive experiments.
Revised: We evaluate our proposed method through extensive experiments.

Original: On all tasks, SWEEN models outperform the upper envelopes of their respective candidate models in terms of the approximated certiﬁed accuracy by a large margin.
Revised: On all tasks, SWEEN models outperform the upper envelopes of their respective candidate models in terms of the approximated certiﬁed accuracy by a large margin.

Original: To the best of our knowledge, our best results achieve the state-of-the-art certiﬁed robustness on CIFAR-10 under (cid:96) 2 norm.
Revised: To the best of our knowledge, our best results achieve the state-of-the-art certiﬁed robustness on CIFAR-10 under (cid:96) 2 norm.

Original: In addition, SWEEN models can achieve comparable or superior performance to a large individual model using a few candidates with a notable reduction in total training time.
Revised: In addition, SWEEN models can achieve comparable or superior performance to a large individual model using a few candidates with a notable reduction in total training time.

Original: 2. Preliminaries 
Revised: 2. Preliminaries Notation

Original: Notation

Original: Let Y = { 1 , 2 , ..., M } .
Revised: Let Y = { 1 , 2 , ..., M } .

Original: We overload notation slightly, letting k refer the M -dimensional onehot vector whose k -th entry is 1 for k = 1 , ..., M as well.
Revised: We overload notation slightly, letting k refer the M -dimensional onehot vector whose k -th entry is 1 for k = 1 , ..., M as well.

Original: The choice should be clear from context.
Revised: The choice should be clear from context.

Original: Let ∆ k
Revised: Let ∆ k

Original: = { ( p 1 , p 2 , ..., p k ) (cid:12)(cid:12) p i ≥ 0 , (cid:80) ki
Revised: = { ( p 1 , p 2 , ..., p k ) (cid:12)(cid:12) p i ≥ 0 , (cid:80) ki

Original: =1 p i = 1 } be the kdimensional probability simplex for k ∈ N + , and ∆ =
Revised: =1 p i = 1 } be the kdimensional probability simplex for k ∈ N + , and ∆ =

Original: ∆ M .
Revised: ∆ M .

Original: For an M -dimensional function f , we use f i to refer to its i -th entry, i = 1 , 2 , · · · , M .
Revised: For an M -dimensional function f , we use f i to refer to its i -th entry, i = 1 , 2 , · · · , M .

Original: We use N (0 , σ 2 I ) to denote the d -dimensional Gaussian distribution with mean 0 and variance σ 2 I .
Revised: We use N (0 , σ 2 I ) to denote the d -dimensional Gaussian distribution with mean 0 and variance σ 2 I .

Original: We use Φ − 1 to denote the inverse of the standard Gaussian CDF.
Revised: We use Φ − 1 to denote the inverse of the standard Gaussian CDF.

Original: Neural network and classiﬁer
Revised: Neural network and classiﬁer

Original: Consider a classiﬁcation problem from X ⊆ R d to classes Y .
Revised: Consider a classiﬁcation problem from X ⊆ R d to classes Y .

Original: The training set { ( x i , y i ) } ni =1 is i.i.d.
Revised: The training set { ( x i , y i ) } ni =1 is i.i.d.

Original: drawn from the data distribution D .
Revised: drawn from the data distribution D .

Original: We call f a probability function or a classiﬁer if it is a mapping from R d to ∆ or Y , respectively.
Revised: We call f a probability function or a classiﬁer if it is a mapping from R d to ∆ or Y , respectively.

Original: For a probability function f , its induced classiﬁer f ∗ is deﬁned such that f ∗ ( x ) =
Revised: For a probability function f , its induced classiﬁer f ∗ is deﬁned such that f ∗ ( x ) =

Original: arg max 1 ≤ i ≤ M f i ( x ) .
Revised: arg max 1 ≤ i ≤ M f i ( x ) .

Original: Certiﬁed robustness
Revised: Certiﬁed robustness

Original: We call x + δ an adversarial example of a classiﬁer f , if f correctly classiﬁes x but f ( x + δ ) (cid:54) =
Revised: We call x + δ an adversarial example of a classiﬁer f , if f correctly classiﬁes x but f ( x + δ ) (cid:54) =

Original: f ( x ) .
Revised: f ( x ) .

Original: Usually (cid:107) δ (cid:107) 2 is small enough so x + δ and x appear almost identical for the human eye.
Revised: Usually (cid:107) δ (cid:107) 2 is small enough so x + δ and x appear almost identical for the human eye.

Original: The ( l 2 -)robust radius of f is deﬁned as
Revised: The ( l 2 -)robust radius of f is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: which is the radius of the largest l 2 ball centered at x within which f consistently predicts the true label y of x .
Revised: which is the radius of the largest l 2 ball centered at x within which f consistently predicts the true label y of x .

Original: Note that r ( x, y ; f )
Revised: Note that r ( x, y ; f )

Original: = 

Revised: =

Original: if f ( x ) (cid:54) =
Revised: if f ( x ) (cid:54) =

Original: y .
Revised: y .

Original: As mentioned before, we can extend the above deﬁnitions to the case when f is a probability function by considering the induced classiﬁer f ∗ .
Revised: As mentioned before, we can extend the above deﬁnitions to the case when f is a probability function by considering the induced classiﬁer f ∗ .

Original: A certiﬁed robustness method tries to ﬁnd some lower bound r c ( x, y ; f ) of r ( x, y ; f ) , and we call r c a certiﬁed radius of f .
Revised: A certiﬁed robustness method tries to ﬁnd some lower bound r c ( x, y ; f ) of r ( x, y ; f ) , and we call r c a certiﬁed radius of f .

Original: Randomized smoothing
Revised: Randomized smoothing

Original: Let f be a probability function or a classiﬁer.
Revised: Let f be a probability function or a classiﬁer.

Original: The (randomized) smoothed function of f is deﬁned as
Revised: The (randomized) smoothed function of f is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The (randomized) smoothed classiﬁer of f is then deﬁned as g ∗ .
Revised: The (randomized) smoothed classiﬁer of f is then deﬁned as g ∗ .

Original: Cohen et al.
Revised: Cohen et al.

Original: ﬁrst provide a tight robustness guarantee for classiﬁer-based smoothed classiﬁers, which is summerized in the following theorem:
Revised: ﬁrst provide a tight robustness guarantee for classiﬁer-based smoothed classiﬁers, which is summerized in the following theorem:

Original: Theorem 1.
Revised: Theorem 1.

Original: (Cohen et al.
Revised: (Cohen et al.

Original: For any classiﬁer f , denote its smoothed function by g .
Revised: For any classiﬁer f , denote its smoothed function by g .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Later on, Salman et al. (2019a); Zhai et al.
Revised: Later on, Salman et al. (2019a); Zhai et al.

Original: (2020) extends
Revised: (2020) extends

Original: Theorem 1 for probability functions.
Revised: Theorem 1 for probability functions.

Original: 3. SWEEN: Smoothed weighted ensembling
Revised: 3. SWEEN: Smoothed weighted ensembling

Original: In this section, we describe the SWEEN framework we use.
Revised: In this section, we describe the SWEEN framework we use.

Original: We also present some theoretical results for SWEEN models in Appendix B which show that SWEEN can achieve optimal certiﬁed robustness w.r.t.
Revised: We also present some theoretical results for SWEEN models in Appendix B which show that SWEEN can achieve optimal certiﬁed robustness w.r.t.

Original: the deﬁned γ -robustness index, which is an extension of previously proposed criteria of certiﬁed robustness (Lemma 1), and SWEEN can be easily trained to a near-optimal risk with a surrogate loss (Theorem 2)..
Revised: the deﬁned γ -robustness index, which is an extension of previously proposed criteria of certiﬁed robustness (Lemma 1), and SWEEN can be easily trained to a near-optimal risk with a surrogate loss (Theorem 2)..

Original: To be speciﬁc, we adopt a data-dependent weighted average of neural networks to serve as the base model for smoothing.
Revised: To be speciﬁc, we adopt a data-dependent weighted average of neural networks to serve as the base model for smoothing.

Original: Suppose we have some pre-trained neural networks f ( · ; θ 1 ) , ..., f ( · ; θ K ) as ensemble candidates.
Revised: Suppose we have some pre-trained neural networks f ( · ; θ 1 ) , ..., f ( · ; θ K ) as ensemble candidates.

Original: A weighted ensemble model is then
Revised: A weighted ensemble model is then

Original: [Equation]
Revised: [Equation]

Original: where θ =
Revised: where θ =

Original: ( θ 1 , ·
Revised: ( θ 1 , ·

Original: · · , θ K ) ∈ Θ K , and w ∈ ∆ K is the ensemble weight.
Revised: · · , θ K ) ∈ Θ K , and w ∈ ∆ K is the ensemble weight.

Original: For a speciﬁc f ens , the corresponding SWEEN model is deﬁned as the smoothed function of f ens , denoted by g ens .
Revised: For a speciﬁc f ens , the corresponding SWEEN model is deﬁned as the smoothed function of f ens , denoted by g ens .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: where g ( · ; θ ) is the smoothed function of f ( · ; θ ) .
Revised: where g ( · ; θ ) is the smoothed function of f ( · ; θ ) .

Original: This result means that g ens is the weighted sum of the smoothed

Original: [Table] [Table] [Table] functions of the candidate models under the same weight w , or more brieﬂy, randomized smoothing and weighted ensembling are commutative.
Revised: This result means that g ens is the weighted sum of the smoothed functions of the candidate models under the same weight w , or more brieﬂy, randomized smoothing and weighted ensembling are commutative.

Original: Thus, ensembling under the randomized smoothing framework can provides beneﬁts in improving the accuracy and robustness.
Revised: Thus, ensembling under the randomized smoothing framework can provides beneﬁts in improving the accuracy and robustness.

Original: To ﬁnd the optimal SWEEN model [Table] we can minimize a surrogate loss of g ens over the training set to obtain the value of appropriate weights.
Revised: To ﬁnd the optimal SWEEN model, we can minimize a surrogate loss of g ens over the training set to obtain the value of appropriate weights.

Original: These data-dependent weights can make the ensemble model robust to the presence of some biased candidate models, as they will be assigned with small weights.
Revised: These data-dependent weights can make the ensemble model robust to the presence of some biased candidate models, as they will be assigned with small weights.

Original: 4. Experiments
Revised: 4. Experiments

Original: In this section [Table] we design extensive experiments on CIFAR10 and ImageNet to evaluate the performance of SWEEN models.
Revised: In this section, we design extensive experiments on CIFAR10 and ImageNet to evaluate the performance of SWEEN models.

Original: 4.1. Setup
Revised: 4.1. Setup

Original: Model setup We train different network architectures on
Revised: Model setup We train different network architectures on

Original: CIFAR-10 to serve as candidates for SWEEN, including
Revised: CIFAR-10 to serve as candidates for SWEEN, including

Original: ResNet-20 (He et al., 2016), ResNet-26, ResNet-32, ResNet50, ResNet-80 and ResNet-110.
Revised: ResNet-20 (He et al., 2016), ResNet-26, ResNet-32, ResNet50, ResNet-80 and ResNet-110.

Original: We particularly evaluate two compositions of SWEEN models.
Revised: We particularly evaluate two compositions of SWEEN models.

Original: The ﬁrst is a relatively rich set of models, including all six candidate models, denoted by the SWEEN-6 model.
Revised: The ﬁrst is a relatively rich set of models, including all six candidate models, denoted by the SWEEN-6 model.

Original: The second is a small set of small models, including ResNet-20, ResNet-26, ResNet32, denoted by the SWEEN-3 model.
Revised: The second is a small set of small models, including ResNet-20, ResNet-26, ResNet32, denoted by the SWEEN-3 model.

Original: The SWEEN-6 model and the SWEEN-3 model simulate how much SWEEN can help in scenarios when we have adequate and limited numbers of candidate models, respectively.
Revised: The SWEEN-6 model and the SWEEN-3 model simulate how much SWEEN can help in scenarios when we have adequate and limited numbers of candidate models, respectively.

Original: On ImageNet, we train three candidate models including ResNet-18, ResNet34 and ResNet-50.
Revised: On ImageNet, we train three candidate models including ResNet-18, ResNet34 and ResNet-50.

Original: We evaluate the SWEEN model containing the three models, denoted by the SWEEN-IN model.
Revised: We evaluate the SWEEN model containing the three models, denoted by the SWEEN-IN model.

Original: A SWEEN model and its candidate models are trained and evaluated with the same noise level σ .
Revised: A SWEEN model and its candidate models are trained and evaluated with the same noise level σ .

Original: The detailed algorithm for obtaining a SWEEN model is presented in Appendix D.
Revised: The detailed algorithm for obtaining a SWEEN model is presented in Appendix D.

Original: Candidate model Training We train candidate models using two training schemes, including Gaussian data augmentation training (Cohen et al., 2019), which is denoted as the standard training for simplicity, and MACER training (Zhai et al., 2020).
Revised: Candidate model Training We train candidate models using two training schemes, including Gaussian data augmentation training (Cohen et al., 2019), which is denoted as the standard training for simplicity, and MACER training (Zhai et al., 2020).

Original: All hyper-parameters used in our experiments are listed in Appendix F.1.
Revised: All hyper-parameters used in our experiments are listed in Appendix F.1.

Original: Solving the ensembling weight From Section 3 we know that we can obtain the empirical risk minimizer by solving a convex optimization.
Revised: Solving the ensembling weight From Section 3 we know that we can obtain the empirical risk minimizer by solving a convex optimization.

Original: However, this requires ﬁrst to approximate the value of smoothed functions of candidate models at every data point, which can be very costly when the number of candidates or training data samples is large.
Revised: However, this requires ﬁrst to approximate the value of smoothed functions of candidate models at every data point, which can be very costly when the number of candidates or training data samples is large.

Original: Hence, we use Gaussian data augmented training to solve the ensembling weight.
Revised: Hence, we use Gaussian data augmented training to solve the ensembling weight.

Original: More precisely, we freeze the parameters of candidate models and minimize the cross-entropy loss of the SWEEN model on Gaussian augmented data from the evaluation set.
Revised: More precisely, we freeze the parameters of candidate models and minimize the cross-entropy loss of the SWEEN model on Gaussian augmented data from the evaluation set.

Original: Empirically we ﬁnd that this approach is much faster and yields comparable results.
Revised: Empirically we ﬁnd that this approach is much faster and yields comparable results.

Original: Certiﬁcation Following previous works, we report the approximated certiﬁed accuracy (ACA), which is the fraction of the test set that can be certiﬁed to be robust at radius r approximately (see (Cohen et al., 2019) for more details).
Revised: Certiﬁcation Following previous works, we report the approximated certiﬁed accuracy (ACA), which is the fraction of the test set that can be certiﬁed to be robust at radius r approximately (see (Cohen et al., 2019) for more details).

Original: We also report the average certiﬁed radius (ACR) following Zhai et al.
Revised: We also report the average certiﬁed radius (ACR) following Zhai et al.

Original: The ACR equals to the area under the radius-accuracy curve (see Figure 1).
Revised: The ACR equals to the area under the radius-accuracy curve (see Figure 1).

Original: All results were certiﬁed using algorithms in (Cohen et al., 2019) with
Revised: All results were certiﬁed using algorithms in (Cohen et al., 2019) with

Original: N = 100 , 000 samples and failure probability α = 0 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: NSEMBLE 33. 1.10M 166.44M N = 100 , 000 samples and failure probability α = 0 .

Original: 4.2. Results
Revised: 4.2. Results

Original: Standard training on CIFAR-10 Due to space constraints, we report the detailed table of results in Appendix F.2.
Revised: Standard training on CIFAR-10 Due to space constraints, we report the detailed table of results in Appendix F.2.

Original: In Figure 1, we display the radius-accuracy curves for the SWEEN models and all their corresponding candidate models under σ = 0 .
Revised: In Figure 1, we display the radius-accuracy curves for the SWEEN models and all their corresponding candidate models under σ = 0 .

Original: 50 on CIFAR-10.
Revised: 50 on CIFAR-10.

Original: The results show that SWEEN models signiﬁcantly boost the performance compared to their corresponding candidate models.
Revised: The results show that SWEEN models signiﬁcantly boost the performance compared to their corresponding candidate models.

Original: According to Figure 1, the SWEEN-6 model consistently outperforms all its candidates in terms of the ACA at all radii.
Revised: According to Figure 1, the SWEEN-6 model consistently outperforms all its candidates in terms of the ACA at all radii.

Original: MACER training on CIFAR- 
Revised: MACER training on CIFAR-

Original: Since SWEEN is compatible with previous training algorithms, we adopt MACER training for the SWEEN-3 model.
Revised: Since SWEEN is compatible with previous training algorithms, we adopt MACER training for the SWEEN-3 model.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: For the ACA and ACR of the ResNetmodel, we use the original numbers from Zhai et al.
Revised: For the ACA and ACR of the ResNetmodel, we use the original numbers from Zhai et al.

Original: In the results, the SWEEN-3 model achieves comparable results to the ResNet-110 but is more efﬁcient.
Revised: In the results, the SWEEN-3 model achieves comparable results to the ResNet-110 but is more efﬁcient.

Original: From Table 5 and 1, the SWEEN-3 model takes 33.9 hours to achieve 0.727 in terms of the ACR for σ = 0 .
Revised: From Table 5 and 1, the SWEEN-3 model takes 33.9 hours to achieve 0.727 in terms of the ACR for σ = 0 .

Original: 5 , using three small and easy-to-train candidates models.
Revised: 5 , using three small and easy-to-train candidates models.

Original: Meanwhile, it takes 49.
Revised: Meanwhile, it takes 49.

Original: hours for the ResNet-110 to achieve similar performance on CIFAR-10.
Revised: hours for the ResNet-110 to achieve similar performance on CIFAR-10.

Original: This 32% speed up reveals the efﬁciency of applying SWEEN to previous training methods.
Revised: This 32% speed up reveals the efﬁciency of applying SWEEN to previous training methods.

Original: Moreover, the SWEEN-6 models outperform the ResNet-110 model on
Revised: Moreover, the SWEEN-6 models outperform the ResNet-110 model on

Original: ACA of most radius levels and on ACR by a large margin.
Revised: ACA of most radius levels and on ACR by a large margin.

Original: To our best knowledge, it establishes a new state-of-the-art certiﬁed robustness on CIFAR-10 under (cid:96) 2 norm.
Revised: To our best knowledge, it establishes a new state-of-the-art certiﬁed robustness on CIFAR-10 under (cid:96) 2 norm.

Original: Results on ImageNet
Revised: Results on ImageNet

Original: Table 2 displays the performance of ResNet-50 and SWEEN-IN under the noise levels σ ∈ { 0 . 25 , 0 .
Revised: Table 2 displays the performance of ResNet-50 and SWEEN-IN under the noise levels σ ∈ { 0 . 25 , 0 .

Original: We note that the performance of ResNet50 is also the upper envelope of the three models in SWEENIN.
Revised: We note that the performance of ResNet50 is also the upper envelope of the three models in SWEENIN.

Original: We can see that the SWEEN-IN model signiﬁcantly outperforms its corresponding candidate models, similar to the results on CIFAR-10.
Revised: We can see that the SWEEN-IN model signiﬁcantly outperforms its corresponding candidate models, similar to the results on CIFAR-10.

Original: The results again conﬁrm the effectiveness of our proposed SWEEN framework.
Revised: The results again conﬁrm the effectiveness of our proposed SWEEN framework.

Original: Other experimental results
Revised: Other experimental results

Original: Due to space constraints, we only report the main results here.
Revised: Due to space constraints, we only report the main results here.

Original: The results of further experiments can be found in Appendix F.
Revised: The results of further experiments can be found in Appendix F.

Original: 5. Conclusions
Revised: 5. Conclusions

Original: In this work, we introduced the smoothed weighted ensembling (SWEEN) to improve randomized smoothed classiﬁers in terms of both accuracy and robustness.
Revised: In this work, we introduced the smoothed weighted ensembling (SWEEN) to improve randomized smoothed classiﬁers in terms of both accuracy and robustness.

Original: We showed that
Revised: We showed that

Original: SWEEN can achieve optimal certiﬁed robustness through our theoretical analysis.
Revised: SWEEN can achieve optimal certiﬁed robustness through our theoretical analysis.

Original: Our extensive experiments showed that a properly designed SWEEN model was able to outperform all its candidate models by a signiﬁcant margin consistently.
Revised: Our extensive experiments showed that a properly designed SWEEN model was able to outperform all its candidate models by a signiﬁcant margin consistently.

Original: Moreover, SWEEN models using a few small and easy-to-train candidates could match or exceed a large individual model on performance with a notable reduction in total training time.
Revised: Moreover, SWEEN models using a few small and easy-to-train candidates could match or exceed a large individual model on performance with a notable reduction in total training time.

Original: Our theoretical and empirical results conﬁrmed that SWEEN is a viable tool for improving the performance of randomized smoothing models.
Revised: Our theoretical and empirical results conﬁrmed that SWEEN is a viable tool for improving the performance of randomized smoothing models.

Original: A. Related Work Adversarial Defenses
Revised: A. Related Work Adversarial Defenses

Original: In the past few years, numerous defenses have been proposed to build classiﬁers robust to adversarial examples.
Revised: In the past few years, numerous defenses have been proposed to build classiﬁers robust to adversarial examples.

Original: These defenses can be roughly categorized into empirical defenses and certiﬁed defenses.
Revised: These defenses can be roughly categorized into empirical defenses and certiﬁed defenses.

Original: One of the most successful empirical defenses is adversarial training (Kurakin et al., 2017; Madry et al., 2018), which optimizes the model by minimizing the loss over adversarial examples generated during training.
Revised: One of the most successful empirical defenses is adversarial training (Kurakin et al., 2017; Madry et al., 2018) which optimizes the model by minimizing the loss over adversarial examples generated during training.

Original: Empirical defenses produce models robust to certain attacks without a theoretical guarantee.
Revised: Empirical defenses produce models robust to certain attacks without a theoretical guarantee.

Original: Most of the empirical defenses are heuristic and subsequently broken by more sophisticated adversaries (Carlini & Wagner, 2017; Athalye et al., 2018; Uesato et al., 2018; Tramer et al., 2020).
Revised: Most of the empirical defenses are heuristic and subsequently broken by more sophisticated adversaries (Carlini & Wagner, 2017; Athalye et al., 2018; Uesato et al., 2018; Tramer et al., 2020).

Original: Certiﬁed defenses, either exact or conservative, are introduced to mitigate such deﬁciency in empirical defenses.
Revised: Certiﬁed defenses, either exact or conservative, are introduced to mitigate such deﬁciency in empirical defenses.

Original: In the context of l p norm-bounded perturbations, exact methods report whether an adversarial example exists within an l p ball with radius r centered at a given input x .
Revised: In the context of l p norm-bounded perturbations, exact methods report whether an adversarial example exists within an l p ball with radius r centered at a given input x .

Original: Exact methods are usually based on Satisﬁability Modulo Theories (Katz et al., 2017; Ehlers, 2017) or mixed-integer linear programming (Lomuscio & Maganti, 2017; Fischetti & Jo, 2017), which are computationally inefﬁcient and not scalable (Tjeng et al., 2019).
Revised: Exact methods are usually based on Satisﬁability Modulo Theories (Katz et al., 2017; Ehlers, 2017) or mixed-integer linear programming (Lomuscio & Maganti, 2017; Fischetti & Jo, 2017), which are computationally inefﬁcient and not scalable (Tjeng et al., 2019).

Original: Conservative methods are more computationally efﬁcient, but might mistakenly ﬂag a safe data point as vulnerable to adversarial examples (Raghunathan et al., 2018a; Wong & Kolter, 2018;
Revised: Conservative methods are more computationally efﬁcient, but might mistakenly ﬂag a safe data point as vulnerable to adversarial examples (Raghunathan et al., 2018a; Wong & Kolter, 2018;

Original: Wong et al., 2018; Gehr et al., 2018; Mirman et al., 2018; Weng et al., 2018; Zhang et al., 2018; Raghunathan et al., 2018b; Dvijotham et al., 2018b; Singh et al., 2018;
Revised: Wong et al., 2018; Gehr et al., 2018; Mirman et al., 2018; Weng et al., 2018; Zhang et al., 2018; Raghunathan et al., 2018b; Dvijotham et al., 2018b; Singh et al., 2018;

Original: Wang et al., 2018b;
Revised: Wang et al., 2018b;

Original: Salman et al., 2019b; Croce et al., 2019;
Revised: Salman et al., 2019b; Croce et al., 2019;

Original: Gowal et al., 2018; Dvijotham et al., 2018a; Wang et al., 2018a).
Revised: Gowal et al., 2018; Dvijotham et al., 2018a; Wang et al., 2018a).

Original: However, both types of defenses are not scalable to practical networks that perform well on modern machine learning problems (e.g., the ImageNet (Deng et al., 2009) classiﬁcation task).
Revised: However, both types of defenses are not scalable to practical networks that perform well on modern machine learning problems (e.g., the ImageNet (Deng et al., 2009) classiﬁcation task).

Original: Randomized smoothing
Revised: Randomized smoothing

Original: Randomized smoothing constructs a smoothed classiﬁer from a base classiﬁer via convolution between the input distribution and certain noise distribution.
Revised: Randomized smoothing constructs a smoothed classiﬁer from a base classiﬁer via convolution between the input distribution and certain noise distribution.

Original: It is ﬁrst proposed as a heuristic defense by (Liu et al., 2018; Cao & Gong, 2017).
Revised: It is ﬁrst proposed as a heuristic defense by (Liu et al., 2018; Cao & Gong, 2017).

Original: Lecuyer et al.
Revised: Lecuyer et al.

Original: ﬁrst prove robustness guarantees for randomized smoothing utilizing tools from differential privacy.
Revised: ﬁrst prove robustness guarantees for randomized smoothing utilizing tools from differential privacy.

Original: Subsequently, a stronger robustness guarantee is given by Li et al.
Revised: Subsequently, a stronger robustness guarantee is given by Li et al.

Original: Cohen et al.
Revised: Cohen et al.

Original: (2019) provide a tight robustness bound for isotropic Gaussian noise in l 2 robustness setting.
Revised: (2019) provide a tight robustness bound for isotropic Gaussian noise in l 2 robustness setting.

Original: The theoretical properties of randomized smoothing in various norm and noise distribution settings have been further discussed in the literature (Blum et al., 2020; Kumar et al., 2020;
Revised: The theoretical properties of randomized smoothing in various norm and noise distribution settings have been further discussed in the literature (Blum et al., 2020; Kumar et al., 2020;

Original: Yang et al., 2020; Lee et al., 2019; Teng et al., 2019; Zhang et al., 2020).
Revised: Yang et al., 2020; Lee et al., 2019; Teng et al., 2019; Zhang et al., 2020).

Original: Recently, a series of works (Salman et al., 2019a;
Revised: Recently, a series of works (Salman et al., 2019a;

Original: Zhai et al., 2020) develop practical algorithms to train a base classiﬁer for randomized smoothing.
Revised: Zhai et al., 2020) develop practical algorithms to train a base classiﬁer for randomized smoothing.

Original: Our work improves the performance of smoothed classiﬁers via weighted en- sembling of pre-trained base classiﬁers
Revised: Our work improves the performance of smoothed classiﬁers via weighted en- sembling of pre-trained base classiﬁers

Original: . Model ensembling
Revised: . Model ensembling

Original: Model ensembling has been widely studied and applied in machine learning as a technique to improve the generalization performance of the model (Hansen & Salamon, 1990; Krogh & Vedelsby, 1994).
Revised: Model ensembling has been widely studied and applied in machine learning as a technique to improve the generalization performance of the model (Hansen & Salamon, 1990; Krogh & Vedelsby, 1994).

Original: Krogh &
Revised: Krogh &

Original: Vedelsby (1994) show that ensembles constructed from accurate and diverse networks perform better.
Revised: Vedelsby (1994) show that ensembles constructed from accurate and diverse networks perform better.

Original: Recently, simple averaging of multiple neural networks has been a success in ILSVRC competitions (He et al., 2016; Krizhevsky et al., 2017; Simonyan & Zisserman, 2015).
Revised: Recently, simple averaging of multiple neural networks has been a success in ILSVRC competitions (He et al., 2016; Krizhevsky et al., 2017; Simonyan & Zisserman, 2015).

Original: Model ensembling has also been used in defenses against adversarial examples (Liu et al., 2018;
Revised: Model ensembling has also been used in defenses against adversarial examples (Liu et al., 2018;

Original: Strauss et al., 2018; Pang et al., 2019; Wang et al., 2019; Meng et al., 2020; Sen et al., 2020).
Revised: Strauss et al., 2018; Pang et al., 2019; Wang et al., 2019; Meng et al., 2020; Sen et al., 2020).

Original: Wang et al.
Revised: Wang et al.

Original: (2019) have shown that a jointly trained ensemble of noise injected ResNets can improve clean and robust accuracies.
Revised: (2019) have shown that a jointly trained ensemble of noise injected ResNets can improve clean and robust accuracies.

Original: Recently, Meng et al.
Revised: Recently, Meng et al.

Original: ﬁnd that ensembling diverse weak models can be quite robust to adversarial attacks.
Revised: ﬁnd that ensembling diverse weak models can be quite robust to adversarial attacks.

Original: Unlike the above works, which are empirical or heuristic, we employ ensembling in randomized smoothing to provide a theoretical robustness certiﬁcation.
Revised: Unlike the above works, which are empirical or heuristic, we employ ensembling in randomized smoothing to provide a theoretical robustness certiﬁcation.

Original: B. Certiﬁed robustness of SWEEN models
Revised: B. Certiﬁed robustness of SWEEN models

Original: The proofs of the results in this section can be found in
Revised: The proofs of the results in this section can be found in

Original: Appendix C.
Revised: Appendix C.

Original: Additional Notation
Revised: Additional Notation

Original: We use Γ to denote the gamma function.
Revised: We use Γ to denote the gamma function.

Original: We use R ∗ to denote the set of non-negative real numbers.
Revised: We use R ∗ to denote the set of non-negative real numbers.

Original: For x, a, b ∈ R , a ≤ b , we deﬁne clip( x ; a, b )
Revised: For x, a, b ∈ R , a ≤ b , we deﬁne clip( x ; a, b )

Original: =
Revised: =

Original: min { max { x, a } , b } .
Revised: min { max { x, a } , b } .

Original: We use Ω( · ) to denote Big-Omega notation that suppresses multiplicative constants.
Revised: We use Ω( · ) to denote Big-Omega notation that suppresses multiplicative constants.

Original: Assume the input space X has ﬁnite diameter D
Revised: Assume the input space X has ﬁnite diameter D

Original: = sup x 1 ,x 2 ∈X (cid:107) x 1 − x 2 (cid:107)< ∞ .
Revised: = sup x 1 ,x 2 ∈X (cid:107) x 1 − x 2 (cid:107)< ∞ .

Original: For a smoothed function g , the certiﬁed radius at ( x, y ) provided by
Revised: For a smoothed function g , the certiﬁed radius at ( x, y ) provided by

Original: Theorem 1 is r c ( x, y ; g )
Revised: Theorem 1 is r c ( x, y ; g )

Original: =
Revised: =

Original: clip( σ 2 [Φ − 1 ( g y ( x )) − Φ − 1 (max k (cid:54) =
Revised: clip( σ 2 [Φ − 1 ( g y ( x )) − Φ − 1 (max k (cid:54) =

Original: y g k ( x ))]; 0 , D ) .
Revised: y g k ( x ))]; 0 , D ) .

Original: We now formally deﬁne γrobustness index as a criterion of certiﬁed robustness.
Revised: We now formally deﬁne γrobustness index as a criterion of certiﬁed robustness.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: ( γ -robustness index).
Revised: ( γ -robustness index).

Original: For γ : R ∗ → R ∗ and a smoothed function g , the γ -robustness index of g is deﬁned as
Revised: For γ : R ∗ → R ∗ and a smoothed function g , the γ -robustness index of g is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: It can be easily observed that γ -robustness index is an extension of many frequently-used criteria of certiﬁed robustness of smoothed classiﬁers.
Revised: It can be easily observed that γ -robustness index is an extension of many frequently-used criteria of certiﬁed robustness of smoothed classiﬁers.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let γ 1 ( r ) = 1 { r ≥ R } , γ 2 ( r ) =
Revised: Let γ 1 ( r ) = 1 { r ≥ R } , γ 2 ( r ) =

Original: r, γ 3 ( r ) =
Revised: r, γ 3 ( r ) =

Original: π d 2 Γ( d 2 +1) r d .
Revised: π d 2 Γ( d 2 +1) r d .

Original: Then, γ 1 -robustness index is the certiﬁed accuracy at radius R (Cohen et al., 2019); γ 2robustness index is the average certiﬁed radius (Zhai et al., 2020); γ 3 -robustness index is the average volume of the certiﬁed region.
Revised: Then, γ 1 -robustness index is the certiﬁed accuracy at radius R (Cohen et al., 2019); γ 2robustness index is the average certiﬁed radius (Zhai et al., 2020); γ 3 -robustness index is the average volume of the certiﬁed region.

Original: We note that criteria considering the volume of the certiﬁed region are more comprehensive than those only considering the certiﬁed radii in a sense, as they take the input dimension into account.
Revised: We note that criteria considering the volume of the certiﬁed region are more comprehensive than those only considering the certiﬁed radii in a sense, as they take the input dimension into account.

Original: Now consider F = { f ( · ; θ ) :
Revised: Now consider F = { f ( · ; θ ) :

Original: R d → ∆ (cid:12)(cid:12) θ ∈ [Equation] [Equation] the set of neural networks parametrized over Θ .
Revised: R d → ∆ (cid:12)(cid:12) θ ∈ Θ } , the set of neural networks parametrized over Θ .

Original: The corresponding set of smoothed functions is G = { g ( x ; θ ) =
Revised: The corresponding set of smoothed functions is G = { g ( x ; θ ) =

Original: E δ ∼N [Equation] ,σ [Equation]
Revised: E δ ∼N (0 ,σ 2 I )

Original: [ f ( x + δ ; θ )] | θ ∈ Θ } .
Revised: [ f ( x + δ ; θ )] | θ ∈ Θ } .

Original: Suppose θ 1 , · · · , θ K are drawn i.i.d.
Revised: Suppose θ 1 , · · · , θ K are drawn i.i.d.

Original: from a ﬁxed probability distribution p on Θ .
Revised: from a ﬁxed probability distribution p on Θ .

Original: The set of SWEEN models is then
Revised: The set of SWEEN models is then

Original: [Equation]
Revised: [Equation]

Original: Similar to Rahimi & Recht (2008), we consider mixtures of the form φ ( x ) =
Revised: Similar to Rahimi & Recht (2008), we consider mixtures of the form φ ( x ) =

Original: (cid:82) Θ w ( θ ) g ( x ; θ ) dθ .
Revised: (cid:82) Θ w ( θ ) g ( x ; θ ) dθ .

Original: For a mixture φ , we deﬁne (cid:107) 
Revised: For a mixture φ , we deﬁne (cid:107) φ

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: sup 
Revised: sup θ

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: w [Equation] θ ) p

Original: [Equation]
Revised: [Equation]

Original: Note that for any φ ∈ F p , φ is a smoothed probability function.
Revised: θ [Equation] | . Deﬁne

Revised: [Equation]

Revised: Note that for any φ ∈ F p , φ is a smoothed probability function.

Original: Intuitively, F p is quite a rich set.
Revised: Intuitively, F p is quite a rich set.

Original: The following result shows that with high probability, the best γ -robustness index a SWEEN model can obtain is near the optimal γrobustness index in the class F p .
Revised: The following result shows that with high probability, the best γ -robustness index a SWEEN model can obtain is near the optimal γrobustness index in the class F p .

Original: Thus, the ensembling generality also holds for the γ -robustness index we deﬁned for robustness.
Revised: Thus, the ensembling generality also holds for the γ -robustness index we deﬁned for robustness.

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose γ is a Lipschitz function.
Revised: Suppose γ is a Lipschitz function.

Original: Given η > 0 .
Revised: Given η > 0 .

Original: For any ε > 0 , for sufﬁcently large K , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: For any ε > 0 , for sufﬁcently large K , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes
Revised: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Moreover, if there exists φ 0 ∈ F p such that Iγ ( φ 0 ) = sup φ ∈ F p I γ ( φ ) , K = O
Revised: Moreover, if there exists φ 0 ∈ F p such that I

Original: ( 1 ε 4 ) .
Revised: [Equation]

Original: In practice, the deﬁned robustness index I γ ( · ) may be hard to optimized directly, in which case we choose a surrogate loss function l : R M × Y → R to approximate it.
Revised: In practice, the deﬁned robustness index I γ ( · ) may be hard to optimized directly, in which case we choose a surrogate loss function l : R M × Y → R to approximate it.

Original: Now the optimization for the ensemble weight w of a SWEEN model over a training set { ( x i , y i ) } ni
Revised: Now the optimization for the ensemble weight w of a SWEEN model over a training set { ( x i , y i ) } ni

Original: =1 can be formulated as
Revised: =1 can be formulated as

Original: [Equation]
Revised: [Equation]

Original: However, this process typically invovles Monte Carlo simulation since we only have access to f ( · , θ k ) , k = 1 , · · · , K .
Revised: However, this process typically invovles Monte Carlo simulation since we only have access to f ( · , θ k ) , k = 1 , · · · , K .

Original: We deﬁne the risk and empirical risk w.r.t.
Revised: We deﬁne the risk and empirical risk w.r.t.

Original: the surrogate loss 
Revised: the surrogate loss l .

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: (Risk and empirical risk).
Revised: (Risk and empirical risk).

Original: For a surrogate loss function l : R M × Y → R , the risk of a probability function φ are deﬁned as
Revised: For a surrogate loss function l : R M × Y → R , the risk of a probability function φ are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: If φ ( x ) = (cid:80)
Revised: If φ ( x ) = (cid:80)

Revised: Kk =1 w k g ( x

Original: [Equation]
Revised: [Equation]

Original: ∈
Revised: θ

Original: ˆ F

Original: [Equation]
Revised: [Equation]

Original:  , for training set { ( x i , y i ) } ni =1 and sample size s , the empirical risk of φ is deﬁned as
Revised: ) ∈ ˆ F θ , for training set { ( x i , y i ) } ni =1 and sample size s , the empirical risk of φ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where δ ijk i.i.d.
Revised: where δ ijk i.i.d.

Original: ∼ N (0 , σ 2 I [Equation] 1 ≤ [Equation]
Revised: ∼ N (0 , σ 2 I ) , 1 ≤ i ≤ n, 1 ≤ j ≤ s, 1 ≤ k ≤

Original: ≤

Original: [Equation]

Original: ≤

Original: K .
Revised: K .

Original: Now solving for w is reduced to ﬁnding the minimizer of
Revised: Now solving for w is reduced to ﬁnding the minimizer of

Original: R emp .
Revised: R emp .

Original: When the loss function l is convex, this problem is a low-dimensional convex optimization, so we can obtain the global empirical risk minimizer using traditional convex optimization algorithms.
Revised: When the loss function l is convex, this problem is a low-dimensional convex optimization, so we can obtain the global empirical risk minimizer using traditional convex optimization algorithms.

Original: Furthermore, we have:
Revised: Furthermore, we have:

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose for all y ∈ Y , l ( · , y ) is a Lipschitz function with constant L and is uniformly bounded.
Revised: Suppose for all y ∈ Y , l ( · , y ) is a Lipschitz function with constant L and is uniformly bounded.

Original: Given η > 0 .
Revised: Given η > 0 .

Original: For any ε > 0 , for sufﬁcently large K , if n = Ω( K 2 ε 2 ) , s = Ω( log Knε 2 ) , then with probability at least 1 − η over the training dataset { ( x i , y i ) } ni =1 drawn i.i.d.
Revised: For any ε > 0 , for sufﬁcently large K , if n = Ω( K 2 ε 2 ) , s = Ω( log Knε 2 ) , then with probability at least 1 − η over the training dataset { ( x i , y i ) } ni =1 drawn i.i.d.

Original: from D and the parameters θ 1 , ..., θ K drawn i.i.d.
Revised: from D and the parameters θ 1 , ..., θ K drawn i.i.d.

Original: from p and the noise samples drawn i.i.d.
Revised: from p and the noise samples drawn i.i.d.

Original: from N (0 , σ 2 I ) , the empirical risk minimizer ˆ φ over ˆ F θ satisﬁes
Revised: from N (0 , σ 2 I ) , the empirical risk minimizer ˆ φ over ˆ F θ satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Moreover, if there exists φ 0 ∈ F p such that R [ φ 0 ] = inf φ ∈ F p R [ φ ] , K
Revised: Moreover, if there exists φ 0 ∈ F p such that R [ φ 0 ] = inf φ ∈ F p R [ φ ] , K

Original: = O ( 1 ε 4 ) .
Revised: = O ( 1 ε 4 ) .

Original: Theorem 2 gives a guarantee that, for large enough K, n, s , the gap between the risk of the empirical risk minimizer ˆ φ and inf φ ∈ F p R [ φ ] can be arbitrarily small with high probability.
Revised: Theorem 2 gives a guarantee that, for large enough K, n, s , the gap between the risk of the empirical risk minimizer ˆ φ and inf φ ∈ F p R [ φ ] can be arbitrarily small with high probability.

Original: Note that we can solve ˆ φ to any given precision when l is convex.
Revised: Note that we can solve ˆ φ to any given precision when l is convex.

Original: Moreover, Theorem 2 reveals the accessibility of ˆ φ in Lemma 1 when l approximates the γ -robustness index well.
Revised: Moreover, Theorem 2 reveals the accessibility of ˆ φ in Lemma 1 when l approximates the γ -robustness index well.

Original: While the number of candidate models and the number of training samples need to be large to ensure good theoretical properties, we will show that the performance of SWEEN models of practical settings is good enough in
Revised: While the number of candidate models and the number of training samples need to be large to ensure good theoretical properties, we will show that the performance of SWEEN models of practical settings is good enough in

Original: Section 4.
Revised: Section 4.

Original: C. Proofs C.1. Proof of Lemma
Revised: C. Proofs C.1. Proof of Lemma

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: We have F p ⊆F (cid:48)
Revised: We have F p ⊆

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Let µ be any probability measure on R d .
Revised: Let µ be any probability measure on R d .

Original: For φ : R d → R M , deﬁne the norm (cid:107) φ [Equation]
Revised: For φ : R d → R M , deﬁne the norm (cid:107) φ (cid:107) 2 µ (cid:44) (cid:82) R d (cid:107) φ ( x ) (cid:107) 22 dµ ( x ) .

Original: µ
Revised: Fix φ ∈ F (cid:48) p , then for any η > 0 , with probability at least

Original: [Equation]
Revised: [Equation]

Original: x ) . Fix φ ∈ F (cid:48) p , then for any η > 0 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˆ φ ( x ) =
Revised: from p , there exists ˆ φ ( x ) =

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: which satisﬁes
Revised: which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Sine φ ∈
Revised: Sine φ ∈

Original: F (cid:48) p , we can write φ ( x ) =
Revised: F (cid:48) p , we can write φ ( x ) =

Original: (cid:82) Θ w ( θ ) g ( x ; θ ) dθ , where w ( θ ) ≥ 0 .
Revised: [Equation] Θ w ( θ ) g ( x ; θ ) dθ , where w ( θ ) ≥ 0 .

Original: Construct φ k = β k g ( · ; θ k ) , k = 1 , 2 , · · · , K , where β k = w ( θ k ) p ( θ k ) , thenE
Revised: Construct φ k = β k g ( · ; θ k ) , k = 1 , 2 , · · · , K , where β k = w ( θ k ) p ( θ k ) , then

Original: [Equation]
Revised: [Equation]

Original: (cid:107) φ (cid:107) p .
Revised: p .

Original: We then deﬁne
Revised: We then deﬁne

Original: [Equation]
Revised: [Equation]

Original: First, by using Jensen’s inequality and the fact that (cid:107) φ k (cid:107) µ ≤ (cid:107) φ (cid:107) p , we have
Revised: First, by using Jensen’s inequality and the fact that (cid:107) φ k (cid:107) [Equation] ≤ (cid:107) φ (cid:107) p , we have

Original: [Equation]
Revised: [Equation]

Revised: ) ≤ (cid:107) φ

Revised: [Equation]

Revised: p √ K .

Original: Next, for θ 1 , · · · , θ M and ˜ θ i , we have
Revised: Next, for θ 1 , · · · , θ M and ˜ θ i , we have

Original: [Equation]
Revised: [Equation]

Original: Now we can use McDiarmid’s inequality to bound u ( θ ) , which gives
Revised: Now we can use McDiarmid’s inequality to bound u ( θ ) , which gives

Original: [Equation]
Revised: [Equation]

Original: The theorem follows by setting δ to the right hand side and solving ε .
Revised: The theorem follows by setting δ to the right hand side and solving ε .

Original: Lemma 3.
Revised: Lemma 3.

Original: Let µ be any probability measure on R d .
Revised: Let µ be any probability measure on R d .

Original: For φ : R d → R M , deﬁne the norm (cid:107) φ (cid:107) 2 µ (cid:44) (cid:82) R d (cid:107) φ ( x ) (cid:107) 22 dµ ( x ) , then for any η > 0 , for K ≥ M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 2 log 1 η ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: For φ : R d → R M , deﬁne the norm (cid:107) φ (cid:107) 2 µ (cid:44) (cid:82) R d (cid:107) φ ( x ) (cid:107) 22 dµ ( x ) , then for any η > 0 , for K ≥ M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 2 log 1 η ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˆ φ ( x ) =
Revised: from p , there exists ˆ φ ( x ) =

Original: K (cid:80) 
Revised: K (cid:80) k

Revised: =1 c

Revised: [Equation]

Revised: g

Original: [Equation]
Revised: [Equation]

Revised: x ; θ

Revised: [Equation]

Revised: [Equation]

Original:  F θ which satisﬁes
Revised: ˆ F θ which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Fix φ ∈ F p ⊆ F (cid:48) p , by using Lemma 2, we have that for any δ > 0 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: Fix φ ∈ F p ⊆ F (cid:48) p , by using Lemma 2, we have that for any δ > 0 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˜ φ ( x ) =
Revised: from p , there exists ˜ φ ( x ) =

Original: K (cid:80)
Revised: K (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ˆ F (cid:48) θ which satisﬁes
Revised: ˆ F (cid:48) θ which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Denote C =
Revised: Denote C =

Original: K (cid:80) k
Revised: K (cid:80) k

Original: =1 c k , and deﬁne s ( t ) (cid:44)
Revised: =1 c k , and deﬁne s ( t ) (cid:44)

Original: M (cid:80) i =
Revised: M (cid:80) i =

Original: 1 t i as the sum of all elements of t ∈ R M .
Revised: 1 t i as the sum of all elements of t ∈ R M .

Original: Then s ( g ( x ; θ ))
Revised: Then s ( g ( x ; θ ))

Original: = 1 , ∀ x ∈
Revised: = 1 , ∀ x ∈

Original: R d , θ ∈ Θ .
Revised: R d , θ ∈ Θ .

Original: Thus,s ( φ ( x [Equation] = M (cid:88)
Revised: Thus,

Original: [Equation]

Original: M

Original: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: w
Revised: k

Original: [Equation]
Revised: [Equation]

Original: Now we have
Revised: Now we have

Original: [Equation]
Revised: [Equation]

Original: dµ [Equation] C [Equation]
Revised: which gives 1 − √ MB ( K )

Original: [Equation] M dµ ( x ) = ( C − 1) M , which gives 1 − √ MB ( K ) <
Revised: <

Original: C < 1 + √ MB
Revised: C < 1 + √ MB

Original: ( K ) .
Revised: ( K ) .

Original: Construct ˆ φ ( x ) =
Revised: Construct ˆ φ ( x ) =

Original: ˜
Revised: ˜

Original: φ ( x ) C , then ˆ φ ∈ ˆ F θ and 
Revised: φ ( x ) C , then ˆ φ ∈ ˆ F θ and (cid:107) ˆ φ − φ (cid:107) 2 µ

Original: [Equation]
Revised: [Equation]

Revised: (cid:90)

Original: [Equation]
Revised: [Equation]

Original: x
Revised: = [Equation] ˜

Original: [Equation]
Revised: [Equation]

Revised: ( x

Original: Since we have C
Revised: [Equation] [Equation] x ) = (cid:90) R d ( (cid:107) ˜

Original: [Equation]
Revised: [Equation]

Revised: ( x ) −

Original: [Equation]
Revised: [Equation]

Original: {+( x ) (cid:107) 
 ++} (cid:107)  φ ( x ) (cid:107)22 (cid:107) ˜
Revised: {+( x ) (cid:107) 
 ++} (cid:107) ( C − 1 − 1)˜ φ ( x ) (cid:107)

Original: φ
Revised: [Equation] ( C [Equation]

Original:  ( x )(cid:107) 22 ≤ C , it holds that
Revised: 1) (cid:104) ˜ φ ( x ) − φ ( x ) , ˜ φ ( x ) (cid:105) )

Original: (i){+dµ ( x ) = (cid:90) R d ( (cid:107) ˆ φ ( x ) − φ ( x ) (cid:107) 
 + ( C − 2 − 1) (cid:107) ˜+}
Revised: {+dµ ( x ) = (cid:90) R d ( (cid:107) ˆ φ ( x ) − φ ( x ) (cid:107) 
 + ( C − 2 − 1) (cid:107) ˜+}

Original: When
Revised: φ ( x ) (cid:107)+2(1 − C − 1 ) (cid:104) φ ( x ) , ˜ φ ( x ) (cid:105) ) dµ ( x ) . Since we have C 2 M ≤ (cid:107) ˜ φ ( x ) (cid:107) 22 ≤ C 2 , |(cid:104) φ ( x ) , ˜ φ ( x ) (cid:105)| ≤ (cid:113) (cid:107) φ ( x ) (cid:107) 22 (cid:107) ˜

Original: 1 < C <
Revised: φ ( x ) (cid:107) 22 ≤ C , it holds that (i) [Equation]

Original: [Equation]
Revised: [Equation]

Original: K
Revised: C

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (ii When 1 − √ MB
Revised: (ii) When 1 − √ MB

Original: ( K ) <
Revised: ( K ) <

Original: C ≤ 
Revised: C ≤ 1 ,

Original: [Equation]
Revised: [Equation]

Original: < [Equation]
Revised: < 4 √ MB ( K ) .

Original: Thus, with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: Thus, with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p ,
Revised: from p ,

Original: [Equation]
Revised: [Equation]

Original: ˆ

Original: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: Suppose l ( · , · ) is L -Lipschitz in its ﬁrst argument.
Revised: Suppose l ( · , · ) is L -Lipschitz in its ﬁrst argument.

Original: Fix φ ∈ F p , then for any η > 0 , for K ≥
Revised: Fix φ ∈ F p , then for any η > 0 , for K ≥

Original: M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 2 log 1 η ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 2 log 1 η ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes
Revised: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: | E [ l (ˆ

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The desired result follows from Lemma 3.
Revised: The desired result follows from Lemma 3.

Original: Lemma 5.
Revised: Lemma 5.

Original: (Corollary of Proposition 1 in Zhai et al.
Revised: (Corollary of Proposition 1 in Zhai et al.

Original: Given any p 1 , p 2 , · · · , p M satisﬁes p 1 ≥ p 2 ≥ · · · ≥ p M ≥ 0 and p 1 + p 2 + · · · + p M = 1 .
Revised: Given any p 1 , p 2 , · · · , p M satisﬁes p 1 ≥ p 2 ≥ · · · ≥ p M ≥ 0 and p 1 + p 2 + · · · + p M = 1 .

Original: The derivative of clip( σ 2 [Φ − 1 ( p 1 ) − Φ − 1 ( p 2 )]; 0 , D ) with respect to p 1 and p 2 is bounded.
Revised: The derivative of clip( σ 2 [Φ − 1 ( p 1 ) − Φ − 1 ( p 2 )]; 0 , D ) with respect to p 1 and p 2 is bounded.

Original: Now we can prove Lemma 1.
Revised: Now we can prove Lemma 1.

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Let φ 0 ∈ F p such that I γ ( φ 0 ) >
Revised: Let φ 0 ∈ F p such that I γ ( φ 0 ) >

Original: [Equation]
Revised: [Equation]

Original: From Lemma 5 we know that
Revised: From Lemma 5 we know that

Original: [Equation]
Revised: [Equation]

Original: clip( σ 2 [Φ
Revised: [Equation] 2 [Φ

Original: [Equation]
Revised: [Equation]

Original:  p y ) − Φ
Revised: 1 ( p y ) − Φ

Original: [Equation]
Revised: [Equation]

Original:  (max k (cid:54) =
Revised: 1 (max k (cid:54) =

Original: y p k )]; 0 , D ) is
Revised: y p k )]; 0 , D ) is

Original: Lipschitz in its ﬁrst argument.
Revised: Lipschitz in its ﬁrst argument.

Original: Since m is Lipschitz, c ( p, y ) (cid:44) m ( q ( p, y )) is also Lipschitz in its ﬁrst argument with some constant L .
Revised: Since m is Lipschitz, c ( p, y ) (cid:44) m ( q ( p, y )) is also Lipschitz in its ﬁrst argument with some constant L .

Original: Apply Lemma 4, we have that for
Revised: Apply Lemma 4, we have that for

Original: K ≥ M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 1 + 2 log 1 δ ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.
Revised: K ≥ M (cid:107) φ (cid:107) 2 p (1 + (cid:113) 1 + 2 log 1 δ ) 2 , with probability at least 1 − η over θ 1 , ..., θ K drawn i.i.d.

Original: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes
Revised: from p , there exists ˆ φ ∈ ˆ F θ which satisﬁes

Original: [Equation]
Revised: [Equation]

Original: When K > 
Revised: When K > 256 L 4 (cid:107) [Equation] 1+

Original: [Equation]
Revised: [Equation]

Original: , we have sup
Revised: ε, we have sup

Original: [Equation]
Revised: [Equation]

Original: If I γ ( φ 0 ) = sup φ ∈ F p
Revised: If I γ ( φ 0 ) = sup φ ∈ F p

Original: I γ ( φ ) , which means (cid:107) φ 0 (cid:107) p is independent of ε , K
Revised: I γ ( φ ) , which means (cid:107) φ 0 (cid:107) p is independent of ε , K

Original: = O ( 1 ε 4 ) .
Revised: = O ( 1 ε 4 ) .

Original: C.2. Proof of Theorem
Revised: C.2. Proof of Theorem

Original: First we introduce some results from statistical learning theory.
Revised: First we introduce some results from statistical learning theory.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: (Gaussian complexity).
Revised: (Gaussian complexity).

Original: Let µ be a probability distribution on a set X and suppose that x 1 , ..., x n are independent samples selected according to µ .
Revised: Let µ be a probability distribution on a set X and suppose that x 1 , ..., x n are independent samples selected according to µ .

Original: Let F be a class of functions mapping from X to R .
Revised: Let F be a class of functions mapping from X to R .

Original: The Gaussian complexity of F is
Revised: The Gaussian complexity of F is

Original: [Equation]
Revised: [Equation]

Original: where ξ 1 , ..., ξ n are independent N (0 , 1) random variables.
Revised: where ξ 1 , ..., ξ n are independent N (0 , 1) random variables.

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: (Rademacher complexity) Let µ be a probability distribution on a set X and suppose that x 1 , ..., x n are independent samples selected according to µ .
Revised: (Rademacher complexity) Let µ be a probability distribution on a set X and suppose that x 1 , ..., x n are independent samples selected according to µ .

Original: Let F be a class of functions mapping from X to R .
Revised: Let F be a class of functions mapping from X to R .

Original: The Rademacher complexity of F is
Revised: The Rademacher complexity of F is

Original: [Equation]
Revised: [Equation]

Original: where σ 1 , ..., σ n are independent uniform {± 1 } -valued random variables.
Revised: where σ 1 , ..., σ n are independent uniform {± 1 } -valued random variables.

Original: Lemma 6.
Revised: Lemma 6.

Original: (Part of Lemma 4 in Bartlett & Mendelson (2001)).
Revised: (Part of Lemma 4 in Bartlett & Mendelson (2001)).

Original: There are absolute constants β such that for every class F and every integer n , R n ( F ) ≤ βG n ( F ) .
Revised: There are absolute constants β such that for every class F and every integer n , R n ( F ) ≤ βG n ( F ) .

Original: Lemma 7.
Revised: Lemma 7.

Original: (Corollary of Theorem 8 in Bartlett & Mendelson (2001)).
Revised: (Corollary of Theorem 8 in Bartlett & Mendelson (2001)).

Original: Consider a loss function c : A × Y → [0 , 1] .
Revised: Consider a loss function c : A × Y → [0 , 1] .

Original: Let F be a class of functions mapping from X to A and let ( x i , y i ) ni
Revised: Let F be a class of functions mapping from X to A and let ( x i , y i ) ni

Original: =1 be independently selected according to the probability measure µ .
Revised: =1 be independently selected according to the probability measure µ .

Original: Then, for any integer n and any 0 < η < 1 , with probability at least 1 − η over samples of length n , every f in F satisﬁes
Revised: Then, for any integer n and any 0 < η < 1 , with probability at least 1 − η over samples of length n , every f in F satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where ˜ c ◦ F = { ( x, y ) (cid:55)→ c ( f ( x ) , y ) − c (0 , y ) (cid:12)(cid:12) f ∈ F }
Revised: where ˜ c ◦ F = { ( x, y ) (cid:55)→ c ( f ( x ) , y ) − c (0 , y ) (cid:12)(cid:12) f ∈ F }

Original: Lemma 8.
Revised: Lemma 8.

Original: (Corollary of Theorem 14 in Bartlett & Mendelson (2001)).
Revised: (Corollary of Theorem 14 in Bartlett & Mendelson (2001)).

Original: Let A = R M and let F be a class of functions mapping from X to A .
Revised: Let A = R M and let F be a class of functions mapping from X to A .

Original: Suppose that there are real-valued classes F 1 , ..., F M such that F is a subset of their Cartesian product.
Revised: Suppose that there are real-valued classes F 1 , ..., F M such that F is a subset of their Cartesian product.

Original: Assume further that c : A × Y → R is such that, for all y ∈ Y , c ( · , y ) is a Lipschitz function with constant L which passes through the origin and is uniformly bounded.
Revised: Assume further that c : A × Y → R is such that, for all y ∈ Y , c ( · , y ) is a Lipschitz function with constant L which passes through the origin and is uniformly bounded.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Now we prove the following lemma:
Revised: Now we prove the following lemma:

Original: Lemma 9.
Revised: Lemma 9.

Original: Let c, F , ( x i , y i )
Revised: Let c, F , ( x i , y i )

Original: ni
Revised: ni

Original: =1 , ˜ c ◦ F be as in Lemma 7.
Revised: =1 , ˜ c ◦ F be as in Lemma 7.

Original: Then, for any integer n and any 0 < η < 1 , with probability at least 1 − η over samples of length n , every f in F satisﬁes
Revised: Then, for any integer n and any 0 < η < 1 , with probability at least 1 − η over samples of length n , every f in F satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: n n

Original: [Equation]
Revised: [Equation]

Original: = [Equation] When an ( x i , y i ) pair changes, the random variable sup h ∈ ˜ c ◦ F (ˆ E n h − E h ) can change by no more than 2 n .
Revised: When an ( x i , y i ) pair changes, the random variable sup h ∈ ˜ c ◦ F (ˆ E n h − E h ) can change by no more than 2 n .

Original: McDiarmid’s inequality implies that with probability at least1 − η 2 ,
Revised: McDiarmid’s inequality implies that with probability at least

Original: [Equation]
Revised: [Equation]

Original: A similar argument, together with the fact that E ˆ E n c (0 , y ) 
Revised: A similar argument, together with the fact that E ˆ [Equation] n c (0 , y ) = [Equation]

Original: =
Revised: c

Original: E c (0 , y ) , shows that with probability at least 
Revised: [Equation] y ) , shows that with probability at least [Equation] − η

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: It’s left to show that E sup h ∈ ˜ c ◦ F (ˆ E n h − E h ) ≤ R n [˜ c ◦ F ] .
Revised: It’s left to show that E sup h ∈ ˜ [Equation] ˆ [Equation]

Original:  Let ( x (cid:48) 1 , y (cid:48) 1 ) , ..., ( x (cid:48) n , y (cid:48) n ) be drawn i.i.d.
Revised: E h [Equation] ≤ R n [˜ c ◦ F ] . Let ( x (cid:48) 1 , y (cid:48) 1 ) , ..., ( x (cid:48) n , y (cid:48) n ) be drawn i.i.d.

Original: from µ and independent from ( x i , y i ) ni =1 , then
Revised: from µ and independent from ( x i , y i ) ni =1 , then

Original: E sup
Revised: E sup

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]

Original: F ]

Original: [Equation]

Original: We can prove the following result:
Revised: We can prove the following result:

Original: Theorem 3.
Revised: Theorem 3.

Original: Let A = R M and let F be a class of functions mapping from X to A .
Revised: Let A = R M and let F be a class of functions mapping from X to A .

Original: Suppose that there are realvalued classes F 1 , ..., F M such that F is a subset of their
Revised: Suppose that there are realvalued classes F 1 , ..., F M such that F is a subset of their

Original: Cartesian product.
Revised: Cartesian product.

Original: Assume further that the loss function c : A × Y → R is such that, for all y ∈ Y , c ( · , y ) is a Lipschitz function with constant L and is uniformly bounded.
Revised: Assume further that the loss function c : A × Y → R is such that, for all y ∈ Y , c ( · , y ) is a Lipschitz function with constant L and is uniformly bounded.

Original: Let { ( x i , y i ) } ni
Revised: Let { ( x i , y i ) } ni

Original: =1 be independently selected according to the probability measure µ .
Revised: =1 be independently selected according to the probability measure µ .

Original: Then, for any integer n and any 0 < η < 1 , there is a probability of at least 1 − η that every f ∈ F has
Revised: Then, for any integer n and any 0 < η < 1 , there is a probability of at least 1 − η that every f ∈ F has

Original: [Equation]
Revised: [Equation]

Original:  where β is a constant.
Revised: η n , where β is a constant.

Original: Proof.
Revised: Proof.

Original: From Lemma 7 and 9 we have that with probability at least 1 − η over samples of length n , every f in F satisﬁes[Equation] n
Revised: From Lemma 7 and 9 we have that with probability at least 1 − η over samples of length n , every f in F satisﬁes

Original: [Equation]
Revised: [Equation]

Original: it follows by applying Lemma 6 and 8.
Revised: it follows by applying Lemma 6 and 8.

Original: Lemma 10.
Revised: Lemma 10.

Original: Let c ( · , · ) , β be as in Theorem 3.
Revised: Let c ( · , · ) , β be as in Theorem 3.

Original: Let ( x i , y i ) ni
Revised: Let ( x i , y i ) ni

Original: =1 be independently selected according to the probability measure D .
Revised: =1 be independently selected according to the probability measure D .

Original: For any integer n and any 0 < η < 1 , there is a probability of at least 1 − η that every f ∈ ˆ F θ has
Revised: For any integer n and any 0 < η < 1 , there is a probability of at least 1 − η that every f ∈ ˆ F θ has

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: Denote 
Revised: Denote ˆ

Original: [Equation]
Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: ≤ M

Revised: [Equation]

Original: We have that ˆ F θ ⊆
Revised: We have that ˆ F θ ⊆

Original: M (cid:78) k
Revised: M (cid:78) k

Original: = ˆ F θ ( i ) , where (cid:78) stands for a
Revised: = ˆ F θ ( i ) , where (cid:78) stands for a

Original: Cartesian product operation.
Revised: Cartesian product operation.

Original: The Gaussian comlexities of ˆ F θ ( i ) ’s can be bounded as
Revised: The Gaussian comlexities of ˆ F [Equation] i ) ’s can be bounded as

Original: [Equation]
Revised: [Equation]

Revised: | 2 n n (cid:88) i = ξ

Revised: i

Revised: K (cid:88) k

Revised: = w k g j ( x i ; w k ) | ] =

Revised: E x,ξ [sup w |

Revised: K (cid:88) k

Revised: = w k 2 n n (cid:88) i =

Revised: ξ i g j ( x i ; w k ) | ] ≤ E x,ξ [

Revised: K (cid:88) k

Revised: = |n n (cid:88) i =

Revised: ξ i g j ( x i ; w k ) | ] ≤ E x

Revised: K (cid:88) k

Revised: = (cid:118)(cid:117)(cid:117)(cid:116)

Revised: E g ( 1 n n (cid:88) i =

Revised: ξ i g j ( x i ; w k )) 2 ] =

Revised: E x

Revised: K (cid:88) k

Revised: = (cid:118)(cid:117) (cid:117)(cid:116)nn (cid:88) i =

Revised: g j ( x i ; w k ) 2 ] ≤ E x [

Revised: K (cid:88) k

Original:  The desired result follows by applying Theorem 3 to ˆ F θ , ˆ F θ (1) , · · · , ˆ F θ ( M ) and D .
Revised: = (cid:114) [Equation] The desired result follows by applying Theorem 3 to ˆ F θ , ˆ F θ (1) , · · · , ˆ F θ ( M ) and D .

Original: Next, we give the deﬁnition of semi-empirical risk.
Revised: Next, we give the deﬁnition of semi-empirical risk.

Original: The term ”semi-” implies that it is empirical with respect to the training set but not the smoothing operation.
Revised: The term ”semi-” implies that it is empirical with respect to the training set but not the smoothing operation.

Original: Deﬁnition 5.
Revised: Deﬁnition 5.

Original: (Semi-empirical risk).
Revised: (Semi-empirical risk).

Original: For a surrogate loss function l ( · , · ) :
Revised: For a surrogate loss function l ( · , · ) :

Original: R M × R M
Revised: R M × R M

Original: → R and training set { ( x i , y i ) } ni =1 , the semi-empirical risk of φ ( x ) =
Revised: → R and training set { ( x i , y i ) } ni =1 , the semi-empirical risk of φ ( x ) =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =

Original: [Equation]

Original:  w k g [Equation] x ; θ k ) ∈
Revised: Kk =1 w k g ( x ; θ k ) ∈

Original: ˆ F θ are deﬁned as
Revised: ˆ F θ are deﬁned as

Revised: R se [ φ ] = n n (cid:88) i =

Revised: l (

Original: [Equation]
Revised: [Equation]

Original: We can use Lemma 4 and 10 to prove the following result:
Revised: We can use Lemma 4 and 10 to prove the following result:

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose for all y ∈ Y , l ( · , y ) is a Lipschitz function with constant L and is uniformly bounded.
Revised: Suppose for all y ∈ Y , l ( · , y ) is a Lipschitz function with constant L and is uniformly bounded.

Original: Fix φ ∈ F p , then for any η > 0 , with probability at least 1 − η over the training dataset { ( x i , y i ) } ni =1 drawn i.i.d.
Revised: Fix φ ∈ F p , then for any η > 0 , with probability at least 1 − η over the training dataset { ( x i , y i ) } ni =1 drawn i.i.d.

Original: from
Revised: from

Original: D and the parameters θ 1 , ..., θ K drawn i.i.d.
Revised: D and the parameters θ 1 , ..., θ K drawn i.i.d.

Original: from p , the semi-empirical risk minimizer ˆ φ over ˆ F θ satisﬁes
Revised: from p , the semi-empirical risk minimizer ˆ φ over ˆ F θ satisﬁes

Original: [Equation]
Revised: [Equation]

Original: where β is a constant.
Revised: where β is a constant.

Original: Proof.
Revised: Proof.

Original: Let φ ∗ be the minimizer of R over ˆ F θ .
Revised: Let φ ∗ be the minimizer of R over ˆ F θ .

Original: Combine
Revised: Combine

Original: Lemma 4 and 10, we derive that, with probability at least 1 − 2 δ over the training dataset and the choice of the parameters θ 1 , ..., θ K ,
Revised: Lemma 4 and 10, we derive that, with probability at least 1 − 2 δ over the training dataset and the choice of the parameters θ 1 , ..., θ K ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 11.
Revised: Lemma 11.

Original: Let µ be a probability distribution on ∆ .
Revised: Let µ be a probability distribution on ∆ .

Original: For any η > 0 , with probability at least 1 − η over x 1 , ..., x s drawn i.i.d.
Revised: For any η > 0 , with probability at least 1 − η over x 1 , ..., x s drawn i.i.d.

Original: from µ , it holds that
Revised: from µ , it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne u ( x 1 , · · · , x s ) =
Revised: Deﬁne u ( x 1 , · · · , x s ) =

Original: (cid:107) 1 s (cid:80) si =1 x i − E [ x ] (cid:107) 2 .
Revised: [Equation] si =1 x i − E [ x ] (cid:107) 2 .

Original: By using Jensen’s inequality, we have
Revised: By using Jensen’s inequality, we have

Original: [Equation]
Revised: [Equation]

Original:  Next, for x 1 , · · · , x M and ˜ x k , we have
Revised: ≤ 1 √ s. Next, for x 1 , · · · , x M and ˜ x k , we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: k

Revised: [Equation]

Revised: [Equation]

Original:  Now we can use McDiarmid’s inequality to bound u ( x ) , which gives
Revised: = [Equation] s ≤ s. Now we can use McDiarmid’s inequality to bound u ( x ) , which gives

Original: [Equation]
Revised: [Equation]

Original: The result follows by setting η to the right hand side and solving ε .
Revised: The result follows by setting η to the right hand side and solving ε .

Original: Now we are ready to prove Theorem 2.
Revised: Now we are ready to prove Theorem 2.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Let φ 0 ∈ F p such that R [ φ 0 ] < inf φ ∈ F p R [ φ ] + ε 4 .
Revised: Let φ 0 ∈ F p such that R [ φ 0 ] < inf φ ∈ F p R [ φ ] + ε 4 .

Original: By Lemma 11, with probability at least
Revised: By Lemma 11, with probability at least

Original: [Equation]
Revised: [Equation]

Original: η 3 ,

Original: [Equation]

Original: hold simultaneously.
Revised: hold simultaneously.

Original: So with probability at least 1 − η 3 , for every φ = 
Revised: So with probability at least 1 − η 3 , for every φ = (cid:80) Kk =1 w k g ( x ; θ k ) ∈ ˆ F θ , it holds that

Original: [Equation]
Revised: [Equation]

Original: F θ , it holds that
Revised: By Lemma

Original: [Equation]
Revised: [Equation]

Original: By Lemma 10, with probability at least 1 − η 3 , for every φ ∈ ˆ F θ , it holds that
Revised: with probability at least 1 − η 3 , for every φ ∈ ˆ F θ [Equation] it holds that

Original: [Equation]
Revised: [Equation]

Original: Let φ ∗ be the minimizer of R over ˆ F θ .
Revised: Let φ ∗ be the minimizer of R over ˆ F θ .

Original: By Lemma 4, with probability at least 1 − δ 3 , for K ≥ M (cid:107) φ 
Revised: By Lemma 4, with probability at least 1 − δ 3 , for K ≥ M (cid:107) φ 0 (cid:107) 2 p

Original: [Equation]
Revised: [Equation]

Original: p
Revised: + (cid:113) 2 log 3 η ) 2 ,

Original: [Equation]
Revised: [Equation]

Original: So with probability at least 1 − η , it holds that
Revised: So with probability at least 1 − η , it holds that

Original: [Equation]
Revised: [Equation]

Original: When
Revised: When

Revised: K

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: If R [ φ 0 ] = inf φ ∈ F p R [ φ ] , which means (cid:107) φ 0 (cid:107) p is independent of ε , K
Revised: If R [ φ 0 ] = inf φ ∈ F p R [ φ ] , which means (cid:107) φ 0 (cid:107) p is independent of ε , K

Original: = O ( 
Revised: = O ( 1 ε 4 ) .

Original: D. Detailed SWEEN algorithm
Revised: D. Detailed SWEEN algorithm

Revised: Algorithm

Revised: = { θ g sween ( · ; θ, w ) = (cid:80)

Original: E. Adaptive prediction algorithm
Revised: E. Adaptive prediction algorithm

Original: A major drawback of ensembling is the high execution cost during inference, which consists of prediction and certiﬁcation costs for smoothed classiﬁers.
Revised: A major drawback of ensembling is the high execution cost during inference, which consists of prediction and certiﬁcation costs for smoothed classiﬁers.

Original: The evaluation of smoothed classiﬁers relies on Monte Carlo simulation, which is computationally expensive.
Revised: The evaluation of smoothed classiﬁers relies on Monte Carlo simulation, which is computationally expensive.

Original: For instance, Cohen et al.
Revised: For instance, Cohen et al.

Original: (2019) use 100 Monte Carlo samples for prediction and 100,000 samples for certiﬁcation.
Revised: (2019) use 100 Monte Carlo samples for prediction and 100,000 samples for certiﬁcation.

Original: If we use 100 candidate models to construct a SWEEN model, the certiﬁcation of a single data point will require 10,010,000 local evaluations (10,000 for prediction and 10,000,000 for certiﬁcation).
Revised: If we use 100 candidate models to construct a SWEEN model, the certiﬁcation of a single data point will require 10,010,000 local evaluations (10,000 for prediction and 10,000,000 for certiﬁcation).

Original: Inoue (2019) observes that ensembling does not make improvements for inputs predicted with high probabilities even when they are mispredicted.
Revised: Inoue (2019) observes that ensembling does not make improvements for inputs predicted with high probabilities even when they are mispredicted.

Original: He proposes an adaptive ensemble prediction algorithm to reduce the execution cost of unweighted ensemble models.
Revised: He proposes an adaptive ensemble prediction algorithm to reduce the execution cost of unweighted ensemble models.

Original: We modify the algorithm to make it applicative to weighted ensemble models, which is detailed in Algorithm 2.
Revised: We modify the algorithm to make it applicative to weighted ensemble models, which is detailed in Algorithm 2.

Original: For a data point, classiﬁers are evaluated in descending order with respect to their weights.
Revised: For a data point, classiﬁers are evaluated in descending order with respect to their weights.

Original: Whenever an early-exit condition is satisﬁed, we stop the evaluation and return the current prediction.
Revised: Whenever an early-exit condition is satisﬁed, we stop the evaluation and return the current prediction.

Original: level α threshold T data point − 1 (1 − α 2 ) 3: indices descending order Set i ← i +6:
Revised: [Equation]

Original: Compute the w π i -th local prediction p π i ← ( p π i , 1 , · · · , p π i ,M ) ∈ ∆ 7:
Revised: [Equation]

Original: Compute ˆ p
Revised: [Equation]

Original: i,k ← (cid:80) ij =1 w πj p πj,k (cid:80) ij =1 w πj for k = 1 , 2 , · · · M 8:
Revised: [Equation] [Equation] [Equation] for k = 1 , 2 , · · · M 8:

Original: Compute k
Revised: Compute k

Original: i ← arg max k
Revised: i ← arg max k

Original: ˆ p
Revised: ˆ p

Original: i,k 9: until ˆ p 1 ,k 1 > T or for some 1 < i < K , ˆ p
Revised: i,k 9: until ˆ p 1 ,k 1 > T or for some 1 < i < K , ˆ p

Original: i,k i > 12 + z (cid:113)(cid:80) ij = w 2 πj (cid:80) ij =1 w πj (cid:114) (cid:80) ij =1 w πj πj −  ij or i
Revised: i,k i > 12 + z (cid:113)(cid:80) ij =1 w 2 πj (cid:80) ij =1 w πj (cid:114) (cid:80) ij =1 w πj ( p πj,ki − ˆ p i,ki ) 2 (cid:80) ij =1 w πj

Original:  K 10: return k i  pi k k =
Revised: or i = K 10: return k i and ˆ p

Original:  2 , · · · M
Revised: i,k , k = 1 , 2 , · · · M

Original: F. Supplementary material for experiments F.1. Detailed settings and hyper-parameters
Revised: F. Supplementary material for experiments F.1. Detailed settings and hyper-parameters

Original: We perform all experiments on CIFAR-10 with a single
Revised: We perform all experiments on CIFAR-10 with a single

Original: GeForce GTX 1080 Ti GPU.
Revised: GeForce GTX 1080 Ti GPU.

Original: For the experiments on ImageNet, we use eight V100 GPUs.
Revised: For the experiments on ImageNet, we use eight V100 GPUs.

Original: For training the SWEEN models on CIFAR-10, we divide the training set into two parts, one for training candidate models, and the other for solving weights.
Revised: For training the SWEEN models on CIFAR-10, we divide the training set into two parts, one for training candidate models, and the other for solving weights.

Original: We employ 2, images for solving weights on CIFAR-10.
Revised: We employ 2, images for solving weights on CIFAR-10.

Original: For ImageNet, we use the whole training set to train candidate models and 1/1000 of the training set to solve weights.
Revised: For ImageNet, we use the whole training set to train candidate models and 1/1000 of the training set to solve weights.

Original: For Gaussian data augmentation training, all the models are trained for 400 epochs using SGD on CIFAR-10.
Revised: For Gaussian data augmentation training, all the models are trained for 400 epochs using SGD on CIFAR-10.

Original: The models on ImageNet are trained for 90 epochs.
Revised: The models on ImageNet are trained for 90 epochs.

Original: The learning rate is initialized set as 0.01, and decayed by 0.1 at the 150th/300th epoch.
Revised: The learning rate is initialized set as 0.01, and decayed by 0.1 at the 150th/300th epoch.

Original: For MACER training, we use the same hyper-parameters as Zhai et al.
Revised: For MACER training, we use the same hyper-parameters as Zhai et al.

Original: (2020), i.e., we use k = 16 , β = 16 . 0 , γ = 8 . 0 , and we use λ = 12 . 

Revised: (2020), i.e., we use k = 16 , β = 16 . 0 , γ = 8 . 0 , and we use λ = 12 .

Original: for σ = 0 .
Revised: for σ = 0 .

Original: 25 and λ = 4 . 

Revised: 25 and λ = 4 .

Original: for σ = 0 .
Revised: for σ = 0 .

Original: We train the models for 440 epochs, the learning rate is initialized set as 0.01, and decayed by 0.1 at the 200th/400th epoch.[Table]
Revised: We train the models for 440 epochs, the learning rate is initialized set as 0.01, and decayed by 0.1 at the 200th/400th epoch.

Original: [Table]
Revised: [Table]

Original: F.2. Standard training on CIFAR- 
Revised: F.2. Standard training on CIFAR-

Original: Table 4 displays the performance of two kinds of SWEEN models under noise levels σ ∈ { 0 . 25 , 0 .
Revised: Table 4 displays the performance of two kinds of SWEEN models under noise levels σ ∈ { 0 . 25 , 0 .

Original: The performance of a single ResNet-110 is included for comparison, and we also report the upper envelopes of the ACA and ACR of their corresponding candidate models as UE.
Revised: The performance of a single ResNet-110 is included for comparison, and we also report the upper envelopes of the ACA and ACR of their corresponding candidate models as UE.

Original: The ACR of the SWEEN-6 model is 0.680, much higher than that of the upper envelope of the candidates, which is 0.574.
Revised: The ACR of the SWEEN-6 model is 0.680, much higher than that of the upper envelope of the candidates, which is 0.574.

Original: It conﬁrms our theoretical analysis in Section 3 that SWEEN can combine the strength of candidate models and attain superior performance.
Revised: It conﬁrms our theoretical analysis in Section 3 that SWEEN can combine the strength of candidate models and attain superior performance.

Original: Besides, SWEEN is effective when only limited numbers of small candidate models are available.
Revised: Besides, SWEEN is effective when only limited numbers of small candidate models are available.

Original: The SWEEN-3 model using ResNet-20, ResNet-26, and ResNet-32 achieves higher ACA than the ResNetat all radii on all noise levels.
Revised: The SWEEN-3 model using ResNet-20, ResNet-26, and ResNet-32 achieves higher ACA than the ResNetat all radii on all noise levels.

Original: The total training time and the number of parameters of the SWEEN-3 model are 36 % and 30% less than those of ResNet-110, respectively.
Revised: The total training time and the number of parameters of the SWEEN-3 model are 36 % and 30% less than those of ResNet-110, respectively.

Original: The improvements can be further ampliﬁed by increasing the number and size of candidate models.
Revised: The improvements can be further ampliﬁed by increasing the number and size of candidate models.

Original: As an instance, the
Revised: As an instance, the

Original: ACR of the SWEEN-6 model is at least 13% higher than that of the ResNet-110 in Table 4.
Revised: ACR of the SWEEN-6 model is at least 13% higher than that of the ResNet-110 in Table 4.

Original: The above results verify the effectiveness of SWEEN for randomized smoothing.
Revised: The above results verify the effectiveness of SWEEN for randomized smoothing.

Original: F.3. SWEEN models using candidates with identical architectures
Revised: F.3. SWEEN models using candidates with identical architectures

Original: The SWEEN-3 and SWEEN-6 models are all using candidate models with diverse architectures.
Revised: The SWEEN-3 and SWEEN-6 models are all using candidate models with diverse architectures.

Original: For a more comprehensive result, we also experiment with SWEEN models using candidates with identical architectures.
Revised: For a more comprehensive result, we also experiment with SWEEN models using candidates with identical architectures.

Original: For σ ∈ { 0 .
Revised: For σ ∈ { 0 .

Original: , We train 8 ResNet-110 models using different random seeds on CIFAR-10 via the standard training.
Revised: , We train 8 ResNet-110 models using different random seeds on CIFAR-10 via the standard training.

Original: We then use these models to ensemble SWEEN models.
Revised: We then use these models to ensemble SWEEN models.

Original: [Table]
Revised: [Table]

Original: The results are shown in Table 6 and Figure 2.
Revised: The results are shown in Table 6 and Figure 2.

Original: We can see that SWEEN is still effective in this scenario and signiﬁcantly boosts the performance compared to candidate models.
Revised: We can see that SWEEN is still effective in this scenario and signiﬁcantly boosts the performance compared to candidate models.

Original: We also run experiments on ImageNet using models with identical structure but with different random initialization.
Revised: We also run experiments on ImageNet using models with identical structure but with different random initialization.

Original: We train 3 ResNet-50 on ImageNet via the standard training to ensemble the SWEEN models.
Revised: We train 3 ResNet-50 on ImageNet via the standard training to ensemble the SWEEN models.

Original: Table 7 shows the results.
Revised: Table 7 shows the results.

Original: The improvement of SWEEN is substantial compared with the AVG and UE results.
Revised: The improvement of SWEEN is substantial compared with the AVG and UE results.

Original: F.4. Adaptive prediction ensembling
Revised: F.4. Adaptive prediction ensembling

Original: To alleviate the higher execution cost introduced by
Revised: To alleviate the higher execution cost introduced by

Original: SWEEN, we apply the previously mentioned adaptive prediction algorithm to speed up the certiﬁcation.
Revised: SWEEN, we apply the previously mentioned adaptive prediction algorithm to speed up the certiﬁcation.

Original: Experiments are conducted on the SWEEN-6 models via the standard training on CIFAR-10 and the results are summarized in
Revised: Experiments are conducted on the SWEEN-6 models via the standard training on CIFAR-10 and the results are summarized in

Original: Table 8.
Revised: Table 8.

Original: It can be observed that the adaptive prediction successfully reduce the number of evaluations.
Revised: It can be observed that the adaptive prediction successfully reduce the number of evaluations.

Original: However,
Revised: However,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: the performance of the adaptive prediction models is only slightly worse than their vanilla counterparts.
Revised: the performance of the adaptive prediction models is only slightly worse than their vanilla counterparts.

Original: F.5. SWEEN versus adversarial attacks
Revised: F.5. SWEEN versus adversarial attacks

Original: We further investigate the performance of SWEEN models versus AutoAttack (Croce & Hein, 2020), which is an ensemble of four diverse attacks to reliably evaluate robustness.
Revised: We further investigate the performance of SWEEN models versus AutoAttack (Croce & Hein, 2020), which is an ensemble of four diverse attacks to reliably evaluate robustness.

Original: Similar to Salman et al.
Revised: Similar to Salman et al.

Original: (2019a)
Revised: (2019a)

Original: , we used 128 samples to estimate the smoothed classiﬁer.
Revised: , we used 128 samples to estimate the smoothed classiﬁer.

Original: We share the results below in Table 9.
Revised: We share the results below in Table 9.

Original: It can be seen that SWEEN can improve the empirical robustness as well.
Revised: It can be seen that SWEEN can improve the empirical robustness as well.

Original: [Table]
Revised: [Table]