Original: L OCALIZED R ANDOMIZED S MOOTHING FOR C OLLECTIVE R OBUSTNESS C ERTIFICATION
Revised: L OCALIZED R ANDOMIZED S MOOTHING FOR C OLLECTIVE R OBUSTNESS C ERTIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Models for image segmentation, node classiﬁcation and many other tasks map a single input to multiple labels.
Revised: Models for image segmentation, node classiﬁcation and many other tasks map a single input to multiple labels.

Original: By perturbing this single shared input (e.g. the image) an adversary can manipulate several predictions (e.g. misclassify several pixels).
Revised: By perturbing this single shared input (e.g. the image) an adversary can manipulate several predictions (e.g. misclassify several pixels).

Original: A recent collective robustness certiﬁcate provides strong guarantees on the number of predictions that are simultaneously robust.
Revised: A recent collective robustness certiﬁcate provides strong guarantees on the number of predictions that are simultaneously robust.

Original: This method is however limited to strictly local models, where each prediction is associated with a small receptive ﬁeld.
Revised: This method is however limited to strictly local models, where each prediction is associated with a small receptive ﬁeld.

Original: We propose a more general collective certiﬁcate for the larger class of softly local models, where each output is dependent on the entire input but assigns different levels of importance to different input regions (e.g. based on their proximity in the image).
Revised: We propose a more general collective certiﬁcate for the larger class of softly local models, where each output is dependent on the entire input but assigns different levels of importance to different input regions (e.g. based on their proximity in the image).

Original: The certiﬁcate is based on our novel localized randomized smoothing approach, where the random perturbation strength for different input regions is proportional to their importance for the outputs.
Revised: The certiﬁcate is based on our novel localized randomized smoothing approach, where the random perturbation strength for different input regions is proportional to their importance for the outputs.

Original: The resulting locally smoothed model yields strong collective guarantees while maintaining high prediction quality on both image segmentation and node classiﬁcation tasks.
Revised: The resulting locally smoothed model yields strong collective guarantees while maintaining high prediction quality on both image segmentation and node classiﬁcation tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: There is a wide range of tasks that require models making multiple predictions based on a single input.
Revised: There is a wide range of tasks that require models making multiple predictions based on a single input.

Original: For example, semantic segmentation requires assigning a label to each pixel in an image.
Revised: For example, semantic segmentation requires assigning a label to each pixel in an image.

Original: When deploying such multi-output classiﬁers in practice, their robustness should be a key concern.
Revised: When deploying such multi-output classiﬁers in practice, their robustness should be a key concern.

Original: After all – just like simple classiﬁers (Szegedy et al., 2014) – they can fall victim to adversarial attacks (Xie et al., 2017; Z ¨ ugner & G ¨ unnemann, 2019; Belinkov & Bisk, 2018).
Revised: After all – just like simple classiﬁers (Szegedy et al., 2014) – they can fall victim to adversarial attacks (Xie et al., 2017; Z ¨ ugner & G ¨ unnemann, 2019; Belinkov & Bisk, 2018).

Original: Even without an adversary, random noise or measuring errors could cause one or multiple predictions to unexpectedly change.
Revised: Even without an adversary, random noise or measuring errors could cause one or multiple predictions to unexpectedly change.

Original: In the following, we derive a method that provides provable guarantees on how many predictions can be changed by an adversary.
Revised: In the following, we derive a method that provides provable guarantees on how many predictions can be changed by an adversary.

Original: Since all outputs operate on the same input, they also have to be attacked simultaneously by choosing a single perturbed input.
Revised: Since all outputs operate on the same input, they also have to be attacked simultaneously by choosing a single perturbed input.

Original: While attacks on a single prediction may be easy, attacks on different predictions may be mutually exclusive.
Revised: While attacks on a single prediction may be easy, attacks on different predictions may be mutually exclusive.

Original: We have to explicitly account for this fact to obtain a proper collective robustness certiﬁcate that provides tight bounds.
Revised: We have to explicitly account for this fact to obtain a proper collective robustness certiﬁcate that provides tight bounds.

Original: There already exists a dedicated collective robustness certiﬁcate for multi-output classiﬁers (Schuchardt et al., 2021), but it is only beneﬁcal for models we call strictly local , where each output depends only on a small, well-deﬁned subset of the input.
Revised: There already exists a dedicated collective robustness certiﬁcate for multi-output classiﬁers (Schuchardt et al., 2021), but it is only beneﬁcal for models we call strictly local , where each output depends only on a small, well-deﬁned subset of the input.

Original: One example are graph neural networks that classify each node in a graph based only on its neighborhood.
Revised: One example are graph neural networks that classify each node in a graph based only on its neighborhood.

Original: Multi-output classiﬁers used in practice, however, are often only softly local .
Revised: Multi-output classiﬁers used in practice, however, are often only softly local .

Original: While – unlike strictly local models – all of their predictions are in principle dependent on the entire input, each output may assign different importance to different components.
Revised: While – unlike strictly local models – all of their predictions are in principle dependent on the entire input, each output may assign different importance to different components.

Original: For example, deep convolutional networks used for image segmentation can have very small effective receptive ﬁelds (Luo et al., 2016; Liu et al., 2018b), i.e. primarily use a small region of the input in labeling each pixel.
Revised: For example, deep convolutional networks used for image segmentation can have very small effective receptive ﬁelds (Luo et al., 2016; Liu et al., 2018b), i.e. primarily use a small region of the input in labeling each pixel.

Original: Many models used in node classiﬁcation are based on the homophily assumption that connected nodes are mostly of the same class.
Revised: Many models used in node classiﬁcation are based on the homophily assumption that connected nodes are mostly of the same class.

Original: Thus, they primarily use features from neighboring nodes to classify each node.
Revised: Thus, they primarily use features from neighboring nodes to classify each node.

Original: Even if an architecture is not inherently softly local, a model may learn a softly local mapping through training.
Revised: Even if an architecture is not inherently softly local, a model may learn a softly local mapping through training.

Original: For example, a transformer (Vaswani et al., 2017) can in principle attend to any part of an input sequence.
Revised: For example, a transformer (Vaswani et al., 2017) can in principle attend to any part of an input sequence.

Original: However, in practice the learned attention maps may be ”sparse”, with the prediction for each token being determined primarily by a few (not necessarily nearby)
Revised: However, in practice the learned attention maps may be ”sparse”, with the prediction for each token being determined primarily by a few (not necessarily nearby)

Original: tokens (Shi et al., 2021).
Revised: tokens (Shi et al., 2021).

Original: While an adversarial attack on a single prediction of a softly local model is conceptually no different from that on a single-output classiﬁer, attacking multiple predictions simultaneously can be much more challenging.
Revised: While an adversarial attack on a single prediction of a softly local model is conceptually no different from that on a single-output classiﬁer, attacking multiple predictions simultaneously can be much more challenging.

Original: By deﬁnition, adversarial attacks have to be unnoticeable, meaning the adversary only has a limited budget for perturbing the input.
Revised: By deﬁnition, adversarial attacks have to be unnoticeable, meaning the adversary only has a limited budget for perturbing the input.

Original: When each output is focused on a different part of the input, the adversary has to decide on where to allocate their adversarial budget and may be unable to attack all outputs at once.
Revised: When each output is focused on a different part of the input, the adversary has to decide on where to allocate their adversarial budget and may be unable to attack all outputs at once.

Original: Our collective robustness certiﬁcate explicitly accounts for this budget allocation problem faced by the adversary and can thus provide stronger robustness guarantees.
Revised: Our collective robustness certiﬁcate explicitly accounts for this budget allocation problem faced by the adversary and can thus provide stronger robustness guarantees.

Original: Our certiﬁcate is based on randomized smoothing (Liu et al., 2018a; L´ecuyer et al., 2019; Cohen et al., 2019).
Revised: Our certiﬁcate is based on randomized smoothing (Liu et al., 2018a; L´ecuyer et al., 2019; Cohen et al., 2019).

Original: Randomized smoothing is a versatile black-box certiﬁcation method that has originally been proposed for single-output classiﬁers.
Revised: Randomized smoothing is a versatile black-box certiﬁcation method that has originally been proposed for single-output classiﬁers.

Original: Instead of directly analysing a model, it constructs a smoothed classiﬁer that returns the most likely prediction of the model under random perturbations of its input.
Revised: Instead of directly analysing a model, it constructs a smoothed classiﬁer that returns the most likely prediction of the model under random perturbations of its input.

Original: One can then use statistical methods to certify the robustness of this smoothed classiﬁer.
Revised: One can then use statistical methods to certify the robustness of this smoothed classiﬁer.

Original: We discuss more details in Section 2.
Revised: We discuss more details in Section 2.

Original: Randomized smoothing is typically used with i.i.d.
Revised: Randomized smoothing is typically used with i.i.d.

Original: noise:
Revised: noise:

Original: Each part of the input (e.g. each pixel) independently undergoes random perturbations sampled from the same noise distribution.
Revised: Each part of the input (e.g. each pixel) independently undergoes random perturbations sampled from the same noise distribution.

Original: One can however also use non-i.i.d.
Revised: One can however also use non-i.i.d.

Original: noise (Eiras et al., 2021).
Revised: noise (Eiras et al., 2021).

Original: This results in a smoothed classiﬁer that is certiﬁably more robust to parts of the input that are smoothed with higher noise levels (e.g. larger standard deviation).
Revised: This results in a smoothed classiﬁer that is certiﬁably more robust to parts of the input that are smoothed with higher noise levels (e.g. larger standard deviation).

Original: We apply randomized smoothing to softly-local multi-output classiﬁers in a scheme we call localized randomized smoothing : Instead of using the same smoothing distribution for all outputs, we randomly smooth each output (or set of outputs) using a different non-i.i.d.
Revised: We apply randomized smoothing to softly-local multi-output classiﬁers in a scheme we call localized randomized smoothing : Instead of using the same smoothing distribution for all outputs, we randomly smooth each output (or set of outputs) using a different non-i.i.d.

Original: distribution that matches its inherent soft locality.
Revised: distribution that matches its inherent soft locality.

Original: Using a low noise level for the most relevant parts of the input allows us to retain a high prediction quality (e.g. accuracy).
Revised: Using a low noise level for the most relevant parts of the input allows us to retain a high prediction quality (e.g. accuracy).

Original: Less relevant parts of the input can be smoothed with a higher noise level.
Revised: Less relevant parts of the input can be smoothed with a higher noise level.

Original: The resulting certiﬁcates (one per output) explicitly quantify how robust each prediction is to perturbations of which section of the input – they are certiﬁcates of soft locality.
Revised: The resulting certiﬁcates (one per output) explicitly quantify how robust each prediction is to perturbations of which section of the input – they are certiﬁcates of soft locality.

Original: After certifying each prediction independently using localized randomized smoothing, we construct a (mixed-integer) linear program that combines these per-prediction base certiﬁcates into a collective certiﬁcate that provably bounds the number of simultaneously attackable predictions.
Revised: After certifying each prediction independently using localized randomized smoothing, we construct a (mixed-integer) linear program that combines these per-prediction base certiﬁcates into a collective certiﬁcate that provably bounds the number of simultaneously attackable predictions.

Original: This linear program explicitly accounts for soft locality and the budget allocation problem it causes for the adversary.
Revised: This linear program explicitly accounts for soft locality and the budget allocation problem it causes for the adversary.

Original: This allows us to prove much stronger guarantees of collective robustness than simply certifying each prediction independently.
Revised: This allows us to prove much stronger guarantees of collective robustness than simply certifying each prediction independently.

Original: Our core contributions are:
Revised: Our core contributions are:

Original: • Localized randomized smoothing , a novel smoothing scheme for multi-output classiﬁers.
Revised: • Localized randomized smoothing , a novel smoothing scheme for multi-output classiﬁers.

Original: • A variance smoothing method for efﬁciently certifying smoothed models on discrete data.
Revised: • A variance smoothing method for efﬁciently certifying smoothed models on discrete data.

Original: • A collective certiﬁcate that leverages our identiﬁed common interface for base certiﬁcates.
Revised: • A collective certiﬁcate that leverages our identiﬁed common interface for base certiﬁcates.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: Randomized smoothing.
Revised: Randomized smoothing.

Original: Randomized smoothing is a ﬂexible certiﬁcation technique that can be used for various data types, perturbation models and tasks.
Revised: Randomized smoothing is a ﬂexible certiﬁcation technique that can be used for various data types, perturbation models and tasks.

Original: For simplicity, we focus on a classiﬁcation certiﬁcate for l 2 perturbations (Cohen et al., 2019).
Revised: For simplicity, we focus on a classiﬁcation certiﬁcate for l 2 perturbations (Cohen et al., 2019).

Original: Assume we have a continuous D -dimensional input space R D , a label set Y and a classiﬁer g : R D
Revised: Assume we have a continuous D -dimensional input space R D , a label set Y and a classiﬁer g : R D

Original: → Y .
Revised: → Y .

Original: We can use isotropic Gaussian noise with standard deviation σ ∈ R + to construct the smoothed classiﬁer f = argmax
Revised: We can use isotropic Gaussian noise with standard deviation σ ∈ R + to construct the smoothed classiﬁer f = argmax

Original: y ∈
Revised: y ∈

Original: Y Pr z ∼N ( x ,σ ) [ g ( z ) =
Revised: Y Pr z ∼N ( x ,σ ) [ g ( z ) =

Original: y ] that returns the most likely prediction of base classiﬁer g under the input distribution 1 .
Revised: y ] that returns the most likely prediction of base classiﬁer g under the input distribution 1 .

Original: Given an input x ∈ R D and the smoothed prediction y =
Revised: Given an input x ∈ R D and the smoothed prediction y =

Original: f ( x ) , we want to determine whether the prediction is robust to all l 2 perturbations of magnitude (cid:15) , i.e. whether ∀ x (cid:48) :
Revised: f ( x ) , we want to determine whether the prediction is robust to all l 2 perturbations of magnitude (cid:15) , i.e. whether ∀ x (cid:48) :

Original: || x (cid:48) − x || 2 ≤ (cid:15) : f ( x (cid:48) )
Revised: || x (cid:48) − x || 2 ≤ (cid:15) : f ( x (cid:48) )

Original: =
Revised: =

Original: y .
Revised: y .

Original: Let q
Revised: Let q

Original: =
Revised: =

Original: Pr z ∼N ( x ,σ ) [ g ( x ) =
Revised: Pr z ∼N ( x ,σ ) [ g ( x ) =

Original: y ] be the probability of g predicting label y .
Revised: y ] be the probability of g predicting label y .

Original: The prediction of our smoothed classiﬁer is robust if (cid:15) <
Revised: The prediction of our smoothed classiﬁer is robust if (cid:15) <

Original: σ Φ − 1 ( q ) (Cohen et al., 2019).
Revised: σ Φ − 1 ( q ) (Cohen et al., 2019).

Original: This result showcases a trade-off we alluded to in the previous section: The certiﬁcate can become stronger if the noise-level (here σ ) is increased.
Revised: This result showcases a trade-off we alluded to in the previous section: The certiﬁcate can become stronger if the noise-level (here σ ) is increased.

Original: But doing so could also lower the accuracy of the smoothed classiﬁer or reduce q and thus weaken the certiﬁcate.
Revised: But doing so could also lower the accuracy of the smoothed classiﬁer or reduce q and thus weaken the certiﬁcate.

Original: White-box certiﬁcates for multi-output classiﬁers.
Revised: White-box certiﬁcates for multi-output classiﬁers.

Original: There are multiple recent methods for certifying the robustness of speciﬁc multi-output models (see, for example, (Tran et al., 2021; Z¨ugner &
Revised: There are multiple recent methods for certifying the robustness of speciﬁc multi-output models (see, for example, (Tran et al., 2021; Z¨ugner &

Original: G¨unnemann, 2019; Bojchevski & G¨unnemann, 2019; Z¨ugner & G¨unnemann, 2020;
Revised: G¨unnemann, 2019; Bojchevski & G¨unnemann, 2019; Z¨ugner & G¨unnemann, 2020;

Original: Ko et al., 2019;
Revised: Ko et al., 2019;

Original: Ryou et al., 2021; Shi et al., 2020; Bonaert et al., 2021)) by analyzing their speciﬁc architecture and weights.
Revised: Ryou et al., 2021; Shi et al., 2020; Bonaert et al., 2021)) by analyzing their speciﬁc architecture and weights.

Original: They are however not designed to certify collective robustness.
Revised: They are however not designed to certify collective robustness.

Original: They can only determine independently for each prediction whether or not it can be adversarially attacked.
Revised: They can only determine independently for each prediction whether or not it can be adversarially attacked.

Original: Collective robustness certiﬁcates.
Revised: Collective robustness certiﬁcates.

Original: Most directly related to our work is the certiﬁcate of Schuchardt et al.
Revised: Most directly related to our work is the certiﬁcate of Schuchardt et al.

Original: Like ours, it combines many per-prediction certiﬁcates into a collective certiﬁcate.
Revised: Like ours, it combines many per-prediction certiﬁcates into a collective certiﬁcate.

Original: But, unlike our novel localized smoothing approach, their certiﬁcation procedure is only beneﬁcial for strictly local models, i.e. models whose outputs operate on small subsets of the input.
Revised: But, unlike our novel localized smoothing approach, their certiﬁcation procedure is only beneﬁcial for strictly local models, i.e. models whose outputs operate on small subsets of the input.

Original: Furthermore, their certiﬁcate assumes binary data, while our certiﬁcate deﬁnes a common interface for various data types and perturbation models.
Revised: Furthermore, their certiﬁcate assumes binary data, while our certiﬁcate deﬁnes a common interface for various data types and perturbation models.

Original: A more detailed comparison can be found in Section D. Recently, Fischer et al.
Revised: A more detailed comparison can be found in Section D. Recently, Fischer et al.

Original: (2021) proposed arandomized smoothing certiﬁcate for semantic segmentation.
Revised: (2021) proposed a certiﬁcate for semantic segmentation.

Original: They consider a different notion of collective robustness: Instead of determining how many outputs are robust to simultaneous attacks, they are interested in determining whether all outputs are robust.
Revised: They consider a different notion of collective robustness: They are interested in determining whether all predictions are robust.

Original: For this, they iterate over all predictions and check whether they can be certiﬁed using the  method from (Cohen et al., 2019). Furthermore, they propose to use Holm correction (Holm, 1979) to deal with the multiple comparisons problem (see Section C.4) and to abstain at a higher rate, i.e. only make predictions guaranteed to be very robust.
Revised: In Section C.4 we discuss their method in detail and show that, when used for certifying our notion of collective robustness (i.e. the number of robust predictions), their method is no better than certifying each output independently using the certiﬁcate of Cohen et al. Furthermore, our certiﬁcate can be used to provide equally strong guarantees for their notion of collective robustness by checking whether the number of certiﬁed predictions equals the overall number of predictions.

Original: Another method that– although it is not explicitly designed for it – can be used for certifying collective robustness is center smoothing (Kumar &
Revised: Another method that can be used for certifying collective robustness is center smoothing (Kumar &

Original: Goldstein, 2021).
Revised: Goldstein, 2021).

Original: Center smoothing bounds how much a vector-valued prediction changes w.r.t to a distance function under adversarial perturbations.
Revised: Center smoothing bounds how much a vector-valued prediction changes w.r.t to a distance function under adversarial perturbations.

Original: With the l 0 pseudo-norm as the distance function, center smoothing bounds how many predictions of a classiﬁer can be simultaneously changed.
Revised: With the l 0 pseudo-norm as the distance function, center smoothing bounds how many predictions of a classiﬁer can be simultaneously changed.

Original: Randomized smoothing with non-i.i.d.
Revised: Randomized smoothing with non-i.i.d.

Original: noise.
Revised: noise.

Original: While not designed for certifying collective robustness, two recent certiﬁcates for non-i.i.d.
Revised: While not designed for certifying collective robustness, two recent certiﬁcates for non-i.i.d.

Original: Gaussian (Fischer et al., 2020) and uniform smoothing (Eiras et al., 2021) can be used as a component of our collective certiﬁcation approach: They can serve as per-prediction base certiﬁcates, which can then be combined into our stronger collective certiﬁcate (more details in Section 4) .
Revised: Gaussian (Fischer et al., 2020) and uniform smoothing (Eiras et al., 2021) can be used as a component of our collective certiﬁcation approach: They can serve as per-prediction base certiﬁcates, which can then be combined into our stronger collective certiﬁcate (more details in Section 4) .

Original: Note that we do not use the procedure for optimizing the smoothing distribution proposed by Eiras et al.
Revised: Note that we do not use the procedure for optimizing the smoothing distribution proposed by Eiras et al.

Original: (2021), as this would enable adversarial attacks on the smoothing distribution itself and invalidate the certiﬁcate (see discussion by Wang et al.
Revised: (2021), as this would enable adversarial attacks on the smoothing distribution itself and invalidate the certiﬁcate (see discussion by Wang et al.

Original:  C OLLECTIVE THREAT MODEL
Revised:  C OLLECTIVE THREAT MODEL

Original: Before certifying robustness, we have to deﬁne a threat model, which speciﬁes the type of model that is attacked, the objective of the adversary and which perturbations they are allowed to use.
Revised: Before certifying robustness, we have to deﬁne a threat model, which speciﬁes the type of model that is attacked, the objective of the adversary and which perturbations they are allowed to use.

Original: We assume that we have a multi-output classiﬁer f : X D in → Y D out , that maps from a D in -dimensional vector space to D out labels from label set Y .
Revised: We assume that we have a multi-output classiﬁer f : X D in → Y D out , that maps from a D in -dimensional vector space to D out labels from label set Y .

Original: We further assume that this classiﬁer f is the result of randomly smoothing a base classiﬁer g , as discussed in Section 2.
Revised: We further assume that this classiﬁer f is the result of randomly smoothing a base classiﬁer g , as discussed in Section 2.

Original: To simplify our notation, we write f n to refer to the function x
Revised: To simplify our notation, we write f n to refer to the function x

Original: (cid:55)→ f ( x ) n that outputs the n -th label.
Revised: (cid:55)→ f ( x ) n that outputs the n -th label.

Original: Given this multi-output classiﬁer f , an input x ∈ X D in and the resulting vector of predictions y = f ( x ) , the objective of the adversary is to cause as many predictions from a set of targeted indices T ⊆ { 1 , . . .
Revised: Given this multi-output classiﬁer f , an input x ∈ X D in and the resulting vector of predictions y = f ( x ) , the objective of the adversary is to cause as many predictions from a set of targeted indices T ⊆ { 1 , . . .

Original: , D out } to change.
Revised: , D out } to change.

Original: That is, their objective is min x (cid:48)
Revised: That is, their objective is min x (cid:48)

Original: ∈ B x (cid:80) n ∈ T I [ f n ( x (cid:48) )
Revised: ∈ B x (cid:80) n ∈ T I [ f n ( x (cid:48) )

Original: =
Revised: =

Original: y n ] , where B x ⊆ X D in is the perturbation model.
Revised: y n ] , where B x ⊆ X D in is the perturbation model.

Original: Importantly, note that the minimization operator is outside the sum meaning the predictions have to be attacked using a single input.
Revised: Importantly, note that the minimization operator is outside the sum, meaning the predictions have to be attacked using a single input.

Original: As is common in robustness certiﬁcation, we assume a norm-bound perturbation model.
Revised: As is common in robustness certiﬁcation, we assume a norm-bound perturbation model.

Original: That is, given an input x ∈ X D in , the adversary is only allowed to use perturbed inputs from the set B x = (cid:8) x (cid:48)
Revised: That is, given an input x ∈ X D in , the adversary is only allowed to use perturbed inputs from the set B x = (cid:8) x (cid:48)

Original: ∈
Revised: ∈

Original: X D in | || x (cid:48)
Revised: X D in | || x (cid:48)

Original: − x || p ≤ (cid:15) (cid:9) with p, (cid:15) ≥ 0 .
Revised: − x || p ≤ (cid:15) (cid:9) with p, (cid:15) ≥ 0 .

Original:  A RECIPE FOR COLLECTIVE CERTIFICATES
Revised:  A RECIPE FOR COLLECTIVE CERTIFICATES

Original: Before discussing technical details, we provide a high-level overview of our method.
Revised: Before discussing technical details, we provide a high-level overview of our method.

Original: In localized randomized smoothing, we assign each output g n of a base classiﬁer g its own smoothing distribution Ψ ( n ) that matches our assumptions or knowledge about the base classiﬁer’s soft locality, i.e. for each n ∈ { 1 , . . .
Revised: In localized randomized smoothing, we assign each output g n of a base classiﬁer g its own smoothing distribution Ψ ( n ) that matches our assumptions or knowledge about the base classiﬁer’s soft locality, i.e. for each n ∈ { 1 , . . .

Original: , D out } choose a Ψ ( n ) that induces more noise in input components that are less relevant for g n .
Revised: , D out } choose a Ψ ( n ) that induces more noise in input components that are less relevant for g n .

Original: For example, in Fig. 1, we assume that far-away regions of the image are less relevant and thus perturb pixels in the bottom left with more noise when classifying pixels in the top-right corner.
Revised: For example, in Fig. 1, we assume that far-away regions of the image are less relevant and thus perturb pixels in the bottom left with more noise when classifying pixels in the top-right corner.

Original: The chosen smoothing distributions can then be used to construct the smoothed classiﬁer f .
Revised: The chosen smoothing distributions can then be used to construct the smoothed classiﬁer f .

Original: Given an input x ∈ X D in and the corresponding smoothed prediction y =
Revised: Given an input x ∈ X D in and the corresponding smoothed prediction y =

Original: f ( x ) , randomized smoothing makes it possible to compute per-prediction base certiﬁcates .
Revised: f ( x ) , randomized smoothing makes it possible to compute per-prediction base certiﬁcates .

Original: That is, for each y n , one can compute a set H ( n ) ⊆
Revised: That is, for each y n , one can compute a set H ( n ) ⊆

Original: X D in of perturbed inputs that the prediction is robust to, i.e. ∀ x
Revised: X D in of perturbed inputs that the prediction is robust to, i.e. ∀ x

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: H n : f n ( x (cid:48) )
Revised: H n : f n ( x (cid:48) )

Original: =
Revised: =

Original: y n .
Revised: y n .

Original: Our motivation for using non-i.i.d.
Revised: Our motivation for using non-i.i.d.

Original: distributions is that the H ( n ) will guarantee more robustness for input dimensions smoothed with more noise, i.e. quantify model locality.
Revised: distributions is that the H ( n ) will guarantee more robustness for input dimensions smoothed with more noise, i.e. quantify model locality.

Original: The objective of our adversary is min x (cid:48)
Revised: The objective of our adversary is min x (cid:48)

Original: ∈ B x (cid:80) n ∈ T I [ f n ( x (cid:48) )
Revised: ∈ B x (cid:80) n ∈ T I [ f n ( x (cid:48) )

Original: =
Revised: =

Original: y n ] with collective perturbation model B x ⊆ X D in .
Revised: y n ] with collective perturbation model B x ⊆ X D in .

Original: That is, they want to change as many predictions from the targeted set T as possible.
Revised: That is, they want to change as many predictions from the targeted set T as possible.

Original: A trivial lower bound can be obtained by counting how many predictions are – according to the base certiﬁcates – provably robust to the collective threat model.
Revised: A trivial lower bound can be obtained by counting how many predictions are – according to the base certiﬁcates – provably robust to the collective threat model.

Original: This can be expressed as
Revised: This can be expressed as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the following, we refer to this as the na¨ıve collective certiﬁcate .
Revised: In the following, we refer to this as the na¨ıve collective certiﬁcate .

Original: Thanks to our proposed localized smoothing scheme, we can use the following, tighter bound:
Revised: Thanks to our proposed localized smoothing scheme, we can use the following, tighter bound:

Original: [Equation]
Revised: [Equation]

Original: which preserves the fact that the adversary has to choose a single perturbed input.
Revised: which preserves the fact that the adversary has to choose a single perturbed input.

Original: Because we use different non-i.i.d.
Revised: Because we use different non-i.i.d.

Original: smoothing distributions for different outputs, we provably know that each f n has varying levels of robustness for different parts of the input and that these robustness levels differ among outputs.
Revised: smoothing distributions for different outputs, we provably know that each f n has varying levels of robustness for different parts of the input and that these robustness levels differ among outputs.

Original: Thus, in the r.h.s.
Revised: Thus, in the r.h.s.

Original: problem the adversary has to allocate their limited budget across various input dimensions and may be unable to attack all predictions at once, just like when attacking the classiﬁer in the l.h.s.
Revised: problem the adversary has to allocate their limited budget across various input dimensions and may be unable to attack all predictions at once, just like when attacking the classiﬁer in the l.h.s.

Original: objective (recall Section 1).
Revised: objective (recall Section 1).

Original: This makes our collective certiﬁcate stronger than the na ¨ ıve collective certiﬁcate, which allows each prediction to be attacked independently.
Revised: This makes our collective certiﬁcate stronger than the na ¨ ıve collective certiﬁcate, which allows each prediction to be attacked independently.

Original: As stated in Section 1, the idea of combining base certiﬁcates into stronger collective certiﬁcates has already been explored by Schuchardt et al.
Revised: As stated in Section 1, the idea of combining base certiﬁcates into stronger collective certiﬁcates has already been explored by Schuchardt et al.

Original: But instead of using localized smoothing to capture the (soft) locality of a model, their approach leverages the fact that perturbations outside an output’s receptive ﬁeld can be ignored.
Revised: But instead of using localized smoothing to capture the (soft) locality of a model, their approach leverages the fact that perturbations outside an output’s receptive ﬁeld can be ignored.

Original: For softly local models, which have receptive ﬁelds covering the entire input, their certiﬁcate is no better than the na¨ıve certiﬁcate.
Revised: For softly local models, which have receptive ﬁelds covering the entire input, their certiﬁcate is no better than the na¨ıve certiﬁcate.

Original: Another novel insight underlying our approach is that various non-i.i.d.
Revised: Another novel insight underlying our approach is that various non-i.i.d.

Original: randomized smoothing certiﬁcates share a common interface, which makes our method applicable to diverse data types and perturbation models.
Revised: randomized smoothing certiﬁcates share a common interface, which makes our method applicable to diverse data types and perturbation models.

Original: In the next section, we formalize this common interface.
Revised: In the next section, we formalize this common interface.

Original: We then discuss how it allows us to compute the collective certiﬁcate from Eq.
Revised: We then discuss how it allows us to compute the collective certiﬁcate from Eq.

Original: 1 using (mixed-integer) linear programming.
Revised: 1 using (mixed-integer) linear programming.

Original:  C OMMON INTERFACE FOR BASE CERTIFICATES
Revised:  C OMMON INTERFACE FOR BASE CERTIFICATES

Original: A base certiﬁcate for a prediction y n = f n ( x ) is a set H n ⊆
Revised: A base certiﬁcate for a prediction y n = f n ( x ) is a set H n ⊆

Original: X D in of perturbed inputs that y n is provably robust to, i.e ∀ x (cid:48) ∈ H n : f n ( x (cid:48) ) =
Revised: X D in of perturbed inputs that y n is provably robust to, i.e ∀ x (cid:48) ∈ H n : f n ( x (cid:48) ) =

Original: y n .
Revised: y n .

Original: Note that base certiﬁcates do not have to be exact, but have to be sound, i.e. they do not have to specify all inputs to which the f n are robust but they must not contain any adversarial examples.
Revised: Note that base certiﬁcates do not have to be exact, but have to be sound, i.e. they do not have to specify all inputs to which the f n are robust but they must not contain any adversarial examples.

Original: As a common interface for base certiﬁcates, we propose that the sets H n are parameterized by a weight vector w ( n ) ∈ R D in and a scalar η ( n ) that deﬁne a linear constraint on the element-wise distance between perturbed inputs and the clean input:
Revised: As a common interface for base certiﬁcates, we propose that the sets H n are parameterized by a weight vector w ( n ) ∈ R D in and a scalar η ( n ) that deﬁne a linear constraint on the element-wise distance between perturbed inputs and the clean input:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The
Revised: The

Original: weight vector encodes
Revised: weight vector encodes

Original: how robust y
Revised: how robust y

Original: n
Revised: n

Original: is to perturbations of
Revised: is to perturbations of

Original: different
Revised: different

Original: components of the
Revised: components of the

Original: input.
Revised: input.

Original: The scalar κ is important for collective robustness certiﬁcation, because it encodes which collective perturbation model the base certiﬁcate is compatible with.
Revised: The scalar κ is important for collective robustness certiﬁcation, because it encodes which collective perturbation model the base certiﬁcate is compatible with.

Original: For example, κ = 2 means that the base certiﬁcate can be used for certifying collective robustness to l 2 perturbations.
Revised: For example, κ = 2 means that the base certiﬁcate can be used for certifying collective robustness to l 2 perturbations.

Original: In the following, we present two base certiﬁcates implementing our interface: One for l 2 perturbations of continuous data and one for perturbations of binary data.
Revised: In the following, we present two base certiﬁcates implementing our interface: One for l 2 perturbations of continuous data and one for perturbations of binary data.

Original: In Section B, we further present a certiﬁcate for binary data that can distinguish between adding and deleting bits and a certiﬁcate for l 1 perturbations of continuous data.
Revised: In Section B, we further present a certiﬁcate for binary data that can distinguish between adding and deleting bits and a certiﬁcate for l 1 perturbations of continuous data.

Original: All base certiﬁcates guarantee more robustness for parts of the input smoothed with a higher noise level.
Revised: All base certiﬁcates guarantee more robustness for parts of the input smoothed with a higher noise level.

Original: The certiﬁcates for continuous data are based on known results (Fischer et al., 2020; Eiras et al., 2021) and merely reformulated to match our proposed interface, so that they can be used as part of our collective certiﬁcation procedure.
Revised: The certiﬁcates for continuous data are based on known results (Fischer et al., 2020; Eiras et al., 2021) and merely reformulated to match our proposed interface, so that they can be used as part of our collective certiﬁcation procedure.

Original: The certiﬁcates for discrete data however are original and based on the novel concept of variance smoothing .
Revised: The certiﬁcates for discrete data however are original and based on the novel concept of variance smoothing .

Original: Gaussian smoothing for l 2 perturbations of continuous data The ﬁrst base certiﬁcate is a generalization of Gaussian smoothing to anisotropic noise, a corollary of Theorem A.1 from (Fischer et al., 2020).
Revised: Gaussian smoothing for l 2 perturbations of continuous data The ﬁrst base certiﬁcate is a generalization of Gaussian smoothing to anisotropic noise, a corollary of Theorem A.1 from (Fischer et al., 2020).

Original: In the following, diag( z ) refers to a diagonal matrix with diagonal entries z and Φ − 1 : [0 , 1] → R refers to the the standard normal inverse cumulative distribution function.
Revised: In the following, diag( z ) refers to a diagonal matrix with diagonal entries z and Φ − 1 : [0 , 1] → R refers to the the standard normal inverse cumulative distribution function.

Original: Proposition 1.
Revised: Proposition 1.

Original: Given an output g n : R D in → Y , let f n ( x ) =
Revised: Given an output g n : R D in → Y , let f n ( x ) =

Original: argmax y ∈ Y Pr z ∼N ( x , Σ ) [ g n ( z ) =
Revised: argmax y ∈ Y Pr z ∼N ( x , Σ ) [ g n ( z ) =

Original: y ] be the corresponding smoothed output with Σ = diag ( σ ) 2 and σ ∈ R D in + .
Revised: y ] be the corresponding smoothed output with Σ = diag ( σ ) 2 and σ ∈ R D in + .

Original: Given an input x ∈ R D in and smoothed prediction y n = f n ( x ) , let q =
Revised: Given an input x ∈ R D in and smoothed prediction y n = f n ( x ) , let q =

Original: Pr z ∼N ( x , Σ )
Revised: Pr z ∼N ( x , Σ )

Original: [ g n ( z ) =
Revised: [ g n ( z ) =

Original: y n ] .
Revised: y n ] .

Original: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =
Revised: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =

Original: y n with H ( n ) deﬁned as in Eq. 2, w d = 1 σ d 2 , η = (cid:0) Φ ( − 1) ( q ) (cid:1) 2 and κ = 2 .
Revised: y n with H ( n ) deﬁned as in Eq. 2, w d = 1 σ d 2 , η = (cid:0) Φ ( − 1) ( q ) (cid:1) 2 and κ = 2 .

Original: Bernoulli variance smoothing for perturbations of binary data For binary data, we use a smoothing distribution F ( x , θ ) with θ ∈ [0 , 1] D in that independently ﬂips the d ’th bit with probability θ d , i.e. for x , z ∈ { 0 , 1 } D in and z ∼ F ( x , θ ) we have Pr[ z d (cid:54) =
Revised: Bernoulli variance smoothing for perturbations of binary data For binary data, we use a smoothing distribution F ( x , θ ) with θ ∈ [0 , 1] D in that independently ﬂips the d ’th bit with probability θ d , i.e. for x , z ∈ { 0 , 1 } D in and z ∼ F ( x , θ ) we have Pr[ z d (cid:54) =

Original: x d ] = θ d .
Revised: x d ] = θ d .

Original: A corresponding certiﬁcate could be derived by generalizing (Lee et al., 2019), which considers a single shared θ ∈ [0 , 1] with ∀ d : θ d = θ .
Revised: A corresponding certiﬁcate could be derived by generalizing (Lee et al., 2019), which considers a single shared θ ∈ [0 , 1] with ∀ d : θ d = θ .

Original: However, the cost for computing this certiﬁcate would be exponential in the number of unique values in θ .
Revised: However, the cost for computing this certiﬁcate would be exponential in the number of unique values in θ .

Original: We therefore propose a more efﬁcient alternative.
Revised: We therefore propose a more efﬁcient alternative.

Original: Instead of constructing a smoothed classiﬁer that returns the most likely labels of the base classiﬁer (as discussed in Section 2), we construct a smoothed classiﬁer that returns the labels with the highest expected softmax scores (similar to CDF-smoothing (Kumar et al., 2020)).
Revised: Instead of constructing a smoothed classiﬁer that returns the most likely labels of the base classiﬁer (as discussed in Section 2), we construct a smoothed classiﬁer that returns the labels with the highest expected softmax scores (similar to CDF-smoothing (Kumar et al., 2020)).

Original: For this smoothed model, we can compute a robustness certiﬁcate in constant time.
Revised: For this smoothed model, we can compute a robustness certiﬁcate in constant time.

Original: The certiﬁcate requires determining both the expected value and variance of softmax scores.
Revised: The certiﬁcate requires determining both the expected value and variance of softmax scores.

Original: We therefore call this method variance smoothing .
Revised: We therefore call this method variance smoothing .

Original: While we use it for binary data, it is a general-purpose technique that can be applied to arbitrary domains and smoothing distributions (see discussion in Section B.2).
Revised: While we use it for binary data, it is a general-purpose technique that can be applied to arbitrary domains and smoothing distributions (see discussion in Section B.2).

Original: In the following, we assume the label set Y to consist of numerical labels { 1 , . .
Revised: In the following, we assume the label set Y to consist of numerical labels { 1 , . .

Original: , | Y |} , which simpliﬁes our notation.
Revised: , | Y |} , which simpliﬁes our notation.

Original: Theorem 1.
Revised: Theorem 1.

Original: Given an output g n : { 0 , 1 } D in → ∆ | Y | mapping to scores from the | Y | -dimensional probability simplex, let f n ( x ) = argmax
Revised: Given an output g n : { 0 , 1 } D in → ∆ | Y | mapping to scores from the | Y | -dimensional probability simplex, let f n ( x ) = argmax

Original: y ∈ Y E
Revised: y ∈ Y E

Original: z ∼F ( x , θ ) [ g n ( z ) y ] be the corresponding smoothed classiﬁer with θ ∈ [0 , 1] D in .
Revised: z ∼F ( x , θ ) [ g n ( z ) y ] be the corresponding smoothed classiﬁer with θ ∈ [0 , 1] D in .

Original: Given an input x ∈ { 0 , 1 } D in and smoothed prediction y n = f n ( x ) , let µ = E z ∼F ( x , θ )
Revised: Given an input x ∈ { 0 , 1 } D in and smoothed prediction y n = f n ( x ) , let µ = E z ∼F ( x , θ )

Original: [ g n ( z ) y ] and σ 2 =
Revised: [ g n ( z ) y ] and σ 2 =

Original: Var z ∼F ( x , θ ) [ g n ( z ) y ] .
Revised: Var z ∼F ( x , θ ) [ g n ( z ) y ] .

Original: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =
Revised: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =

Original: y n with H ( n ) deﬁned as in Eq. 2, w d = ln (cid:16) (1 − θ d ) 2 θ d
Revised: y n with H ( n ) deﬁned as in Eq. 2, w d = ln (cid:16) (1 − θ d ) 2 θ d

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  C OMPUTING THE COLLECTIVE ROBUSTNESS CERTIFICATE
Revised:  C OMPUTING THE COLLECTIVE ROBUSTNESS CERTIFICATE

Original: With our common interface for base certiﬁcates in place, we can discuss how to compute the collective robustness certiﬁcate min x (cid:48)
Revised: With our common interface for base certiﬁcates in place, we can discuss how to compute the collective robustness certiﬁcate min x (cid:48)

Original: ∈ B x (cid:80) n ∈ T I (cid:2) x (cid:48)
Revised: ∈ B x (cid:80) n ∈ T I (cid:2) x (cid:48)

Original: ∈ H ( n ) (cid:3) from Eq.
Revised: ∈ H ( n ) (cid:3) from Eq.

Original: The result bounds the number of predictions y n with n ∈ { 1 , . . .
Revised: The result bounds the number of predictions y n with n ∈ { 1 , . . .

Original: , D out } that can be simultaneously attacked by the adversary.
Revised: , D out } that can be simultaneously attacked by the adversary.

Original: In the following, we assume that the base certiﬁcates were obtained by using a smoothing distribution that is compatible with our l p collective perturbation model (i.e. κ = p ), for example by using Gaussian noise for p
Revised: In the following, we assume that the base certiﬁcates were obtained by using a smoothing distribution that is compatible with our l p collective perturbation model (i.e. κ = p ), for example by using Gaussian noise for p

Original: = 2 or Bernoulli noise for p = 0 .
Revised: = 2 or Bernoulli noise for p = 0 .

Original: Inserting the deﬁnition of our base certiﬁcate interface from Eq. 2 and rewriting our perturbation model B x = (cid:8) x (cid:48)
Revised: Inserting the deﬁnition of our base certiﬁcate interface from Eq. 2 and rewriting our perturbation model B x = (cid:8) x (cid:48)

Original: ∈
Revised: ∈

Original: X D in | || x (cid:48)
Revised: X D in | || x (cid:48)

Original: − x || p ≤ (cid:15) (cid:9) as (cid:110) x (cid:48) ∈
Revised: − x || p ≤ (cid:15) (cid:9) as (cid:110) x (cid:48) ∈

Original: X D in | (cid:80) D in
Revised: X D in | (cid:80) D in

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: | x
Revised: | x

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: x d | p ≤ (cid:15) p (cid:111) , our objective from Eq.
Revised: x d | p ≤ (cid:15) p (cid:111) , our objective from Eq.

Original: 1 can be expressed as
Revised: 1 can be expressed as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We can see that the perturbed input x (cid:48) only affects the element-wise distances | x (cid:48)
Revised: We can see that the perturbed input x (cid:48) only affects the element-wise distances | x (cid:48)

Original: d − x d | p .
Revised: d − x d | p .

Original: Rather than optimizing x (cid:48) , we can instead directly optimize these distances, i.e. determine how much adversarial budget is allocated to each input dimension.
Revised: Rather than optimizing x (cid:48) , we can instead directly optimize these distances, i.e. determine how much adversarial budget is allocated to each input dimension.

Original: For this, we deﬁne a vector of variables b ∈ R D in + (or b ∈ { 0 , 1 } D in for binary data).
Revised: For this, we deﬁne a vector of variables b ∈ R D in + (or b ∈ { 0 , 1 } D in for binary data).

Original: Replacing sums with inner products, we can restate Eq.
Revised: Replacing sums with inner products, we can restate Eq.

Original: 3 as
Revised: 3 as

Original: [Equation]
Revised: [Equation]

Original: R D in
Revised: R D in

Original: [Equation]
Revised: [Equation]

Original: In a ﬁnal step, we replace the indicator functions in Eq.
Revised: In a ﬁnal step, we replace the indicator functions in Eq.

Original: 4 with a vector of boolean variables t ∈ { 0 , 1 } D out .
Revised: 4 with a vector of boolean variables t ∈ { 0 , 1 } D out .

Original: Deﬁne the constants η ( n ) =
Revised: Deﬁne the constants η ( n ) =

Original: (cid:15) p · min (cid:16) 0 , min d w ( n ) d (cid:17) .
Revised: (cid:15) p · min (cid:16) 0 , min d w ( n ) d (cid:17) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: is equivalent to Eq.
Revised: is equivalent to Eq.

Original: The ﬁrst constraint guarantees that t n can only be set to 0 if the l.h.s.
Revised: The ﬁrst constraint guarantees that t n can only be set to 0 if the l.h.s.

Original: is greater or equal η ( n ) , i.e. only when the base certiﬁcate can no longer guarantee robustness.
Revised: is greater or equal η ( n ) , i.e. only when the base certiﬁcate can no longer guarantee robustness.

Original: The term involving η ( n ) ensures that for t n = 1 the problem is always feasible 2 .
Revised: The term involving η ( n ) ensures that for t n = 1 the problem is always feasible 2 .

Original: Eq.
Revised: Eq.

Original: 5 can be solved using any mixed-integer linear programming solver.
Revised: 5 can be solved using any mixed-integer linear programming solver.

Original: While the resulting MILP bears some semblance to that of Schuchardt et al.
Revised: While the resulting MILP bears some semblance to that of Schuchardt et al.

Original: (2021), it is conceptually different.
Revised: (2021), it is conceptually different.

Original: When evaluating their base certiﬁcates, they mask out parts of the budget vector b based on a model’s strict locality, while we weigh the budget vector based on the soft locality guaranteed by the base certiﬁcates.
Revised: When evaluating their base certiﬁcates, they mask out parts of the budget vector b based on a model’s strict locality, while we weigh the budget vector based on the soft locality guaranteed by the base certiﬁcates.

Original: In addition, thanks to the interface speciﬁed in Section 5, our problem only involves a single linear constraint per prediction, making it much smaller and more efﬁcient to solve.
Revised: In addition, thanks to the interface speciﬁed in Section 5, our problem only involves a single linear constraint per prediction, making it much smaller and more efﬁcient to solve.

Original: Interestingly, when using randomized smoothing base certiﬁcates for binary data, our certiﬁcate subsumes theirs, i.e. can provide the same robustness guarantees (see Section D.2).
Revised: Interestingly, when using randomized smoothing base certiﬁcates for binary data, our certiﬁcate subsumes theirs, i.e. can provide the same robustness guarantees (see Section D.2).

Original: Improving efﬁciency.
Revised: Improving efﬁciency.

Original: Still, the efﬁciency of our certiﬁcate in Eq.
Revised: Still, the efﬁciency of our certiﬁcate in Eq.

Original: certiﬁcate can be further improved.
Revised: certiﬁcate can be further improved.

Original: In Section A, we show that partitioning the outputs into N out subsets sharing the same smoothing distribution and the the inputs into N in subsets sharing the same noise level (for example like in Fig. 1), as well as quantizing the base certiﬁcate parameters η ( n ) into N bin bins reduces the number of variables and constraints from D in + D out
Revised: In Section A, we show that partitioning the outputs into N out subsets sharing the same smoothing distribution and the the inputs into N in subsets sharing the same noise level (for example like in Fig. 1), as well as quantizing the base certiﬁcate parameters η ( n ) into N bin bins reduces the number of variables and constraints from D in + D out

Original: and D out + 1 to N in + N out · N bins and N out · N bins + 1 , respectively.
Revised: and D out + 1 to N in + N out · N bins and N out · N bins + 1 , respectively.

Original: We can thus control the problem size independent of the data’s dimensionality.
Revised: We can thus control the problem size independent of the data’s dimensionality.

Original: We further derive a linear relaxation of the mixed-integer problem, which can be more efﬁciently solved while preserving the soundness of the certiﬁcate.
Revised: We further derive a linear relaxation of the mixed-integer problem, which can be more efﬁciently solved while preserving the soundness of the certiﬁcate.

Original:  L IMITATIONS
Revised:  L IMITATIONS

Original: The main limitation of our approach is that it assumes softly local models.
Revised: The main limitation of our approach is that it assumes softly local models.

Original: While it can be applied to arbitrary multi-output classiﬁers, it may not necessarily result in better certiﬁcates than randomized smoothing with i.i.d.
Revised: While it can be applied to arbitrary multi-output classiﬁers, it may not necessarily result in better certiﬁcates than randomized smoothing with i.i.d.

Original: distributions.
Revised: distributions.

Original: Furthermore, choosing the smoothing distributions requires some a-priori knowledge or assumptions about which parts of the input are how relevant to making a prediction.
Revised: Furthermore, choosing the smoothing distributions requires some a-priori knowledge or assumptions about which parts of the input are how relevant to making a prediction.

Original: Our experiments show that natural assumptions like homophily can be sufﬁcient for choosing effective smoothing distributions.
Revised: Our experiments show that natural assumptions like homophily can be sufﬁcient for choosing effective smoothing distributions.

Original: But doing so in other tasks may be more challenging.
Revised: But doing so in other tasks may be more challenging.

Original: A limitation of (most) randomized smoothing certiﬁcates is that they use sampling to approximate the smoothed classiﬁer.
Revised: A limitation of (most) randomized smoothing certiﬁcates is that they use sampling to approximate the smoothed classiﬁer.

Original: Because we use different smoothing distributions for different outputs, we can only use a fraction of the samples for each output.
Revised: Because we use different smoothing distributions for different outputs, we can only use a fraction of the samples for each output.

Original: As discussed in Section A.1, we can alleviate this problem by sharing smoothing distributions among multiple outputs.
Revised: As discussed in Section A.1, we can alleviate this problem by sharing smoothing distributions among multiple outputs.

Original: Our experiments show that despite this issue, our method outperforms certiﬁcates that use a single smoothing distribution.
Revised: Our experiments show that despite this issue, our method outperforms certiﬁcates that use a single smoothing distribution.

Original: Still, future work should try to improve the sample efﬁciency of randomized smoothing (for example by developing more methods for de-randomized smoothing (Levine & Feizi, 2020)).Any such advance could then be incorporated into our localized smoothing framework.
Revised: Still, future work should try to improve the sample efﬁciency of randomized smoothing (for example by developing more methods for de-randomized smoothing (Levine & Feizi, 2020)).Any such advance could then be incorporated into our localized smoothing framework.

Original:  E XPERIMENTAL EVALUATION
Revised:  E XPERIMENTAL EVALUATION

Original: Our experimental evaluation has three objectives 1.)
Revised: Our experimental evaluation has three objectives 1.)

Original: Verifying our main claim that localized randomized smoothing offers a better trade-off between accuracy and certiﬁable robustness than smoothing with i.i.d.
Revised: Verifying our main claim that localized randomized smoothing offers a better trade-off between accuracy and certiﬁable robustness than smoothing with i.i.d.

Original: distributions.
Revised: distributions.

Original: Determining to what extend the linear program underlying the proposed collective certiﬁcate strengthens our robustness guarantees.
Revised: Determining to what extend the linear program underlying the proposed collective certiﬁcate strengthens our robustness guarantees.

Original: Assessing the efﬁcacy of our novel variance smoothing certiﬁcate for binary data.
Revised: Assessing the efﬁcacy of our novel variance smoothing certiﬁcate for binary data.

Original: Any of the used datasets and classiﬁers only serve as a means of comparing certiﬁcates.
Revised: Any of the used datasets and classiﬁers only serve as a means of comparing certiﬁcates.

Original: We thus use well-known and well-established architectures instead of overly focusing on maximizing prediction accuracy by using the latest SOTA models.
Revised: We thus use well-known and well-established architectures instead of overly focusing on maximizing prediction accuracy by using the latest SOTA models.

Original: We use two metrics to quantify certiﬁcate strength: Certiﬁed accuracy (i.e. the percentage of correct and certiﬁably robust predictions) and certiﬁed ratio (i.e. the percentage of certiﬁably robust predictions, regardless of correctness) 3 .
Revised: We use two metrics to quantify certiﬁcate strength: Certiﬁed accuracy (i.e. the percentage of correct and certiﬁably robust predictions) and certiﬁed ratio (i.e. the percentage of certiﬁably robust predictions, regardless of correctness) 3 .

Original: As single-number metrics, we report the AUC of the certiﬁed accuracy/ratio functions w.r.t.
Revised: As single-number metrics, we report the AUC of the certiﬁed accuracy/ratio functions w.r.t.

Original: adversarial budget (cid:15) (not to be confused with certifying some AUC metric).
Revised: adversarial budget (cid:15) (not to be confused with certifying some AUC metric).

Original: For localized smoothing, we evaluate both the na¨ıve collective certiﬁcate, i.e. certifying predictions independently (see Section 4), and the proposed LP-based certiﬁcate (using the linearly relaxed version from Appendix A.4).
Revised: For localized smoothing, we evaluate both the na¨ıve collective certiﬁcate, i.e. certifying predictions independently (see Section 4), and the proposed LP-based certiﬁcate (using the linearly relaxed version from Appendix A.4).

Original: We compare our method to two baselines using i.i.d.
Revised: We compare our method to two baselines using i.i.d.

Original: randomized smoothing: The na¨ıve collective certiﬁcate and center smoothing (Kumar & Goldstein, 2021).
Revised: randomized smoothing: The na¨ıve collective certiﬁcate and center smoothing (Kumar & Goldstein, 2021).

Original: For softly local models, the certiﬁcate of Schuchardt et al.
Revised: For softly local models, the certiﬁcate of Schuchardt et al.

Original: (2021) is equivalent to the na¨ıve baseline.
Revised: (2021) is equivalent to the na¨ıve baseline.

Original: When used to certify the number of robust predictions, the segmentation certiﬁcate of Fischer et al.
Revised: When used to certify the number of robust predictions, the segmentation certiﬁcate of Fischer et al.

Original: (2021) is at most as strong as the na¨ıve baseline (see Section C.4).
Revised: (2021) is at most as strong as the na¨ıve baseline (see Section C.4).

Original: Thus, our method is compared to all existing collective certiﬁcates listed in Section 2.
Revised: Thus, our method is compared to all existing collective certiﬁcates listed in Section 2.

Original: In all experiments, we use Monte Carlo randomized smoothing.
Revised: In all experiments, we use Monte Carlo randomized smoothing.

Original: More details on the experimental setup can be found in Section E.
Revised: More details on the experimental setup can be found in Section E.

Original: 8. S EMANTIC SEGMENTATION
Revised: 8. S EMANTIC SEGMENTATION

Original: [Table]
Revised: [Table]

Original: Adversarial budget (cid:15) 0 .0 .0 .0 .0 .1 .
Revised: Adversarial budget (cid:15) 0 .0 .0 .0 .0 .1 .

Original: C e r ti ﬁ e d acc u r ac y
Revised: C e r ti ﬁ e d acc u r ac y

Original: LPlocalized
Revised: LPlocalized

Original: Na¨ıvelocalized
Revised: Na¨ıvelocalized

Original: Na¨ıve i.i.d.
Revised: Na¨ıve i.i.d.

Original: cate for continuous data and l perturbations on the Pascal-VOC 2012 segmentation validation set
Revised: cate for continuous data and l perturbations on the Pascal-VOC 2012 segmentation validation set

Original: Training is performed on 10582 pairs of training samples extracted from SBD 4 (Hariharan et al., 2011), To increase batch sizes and thus allow a more thorough investigation of different smoothing parameters, all images are downscaled to 50% of their original size.
Revised: Training is performed on 10582 pairs of training samples extracted from SBD 4 (Hariharan et al., 2011), To increase batch sizes and thus allow a more thorough investigation of different smoothing parameters, all images are downscaled to 50% of their original size.

Original: Our base model is a U-Net segmentation model with a ResNet-18 backbone.
Revised: Our base model is a U-Net segmentation model with a ResNet-18 backbone.

Original: To obtain accurate and robust smoothed classiﬁers, base models should be trained on the smoothing distribution.
Revised: To obtain accurate and robust smoothed classiﬁers, base models should be trained on the smoothing distribution.

Original: We thus train 51 different instances of our base model, augmenting the training data with a different σ train ∈ { 0 , 0 .
Revised: We thus train 51 different instances of our base model, augmenting the training data with a different σ train ∈ { 0 , 0 .

Original: At test time, when evaluating a baseline i.i.d.
Revised: At test time, when evaluating a baseline i.i.d.

Original: certiﬁcate with smoothing distribution N (0 , σ ) , we load the model trained with σ train =
Revised: certiﬁcate with smoothing distribution N (0 , σ ) , we load the model trained with σ train =

Original: σ .
Revised: σ .

Original: To perform localized randomized smoothing, we choose parameters σ min , σ max ∈ R + and partition all images into regular grids of size 4 × 6 (similar to example Fig. 1).
Revised: To perform localized randomized smoothing, we choose parameters σ min , σ max ∈ R + and partition all images into regular grids of size 4 × 6 (similar to example Fig. 1).

Original: To classify pixels in grid cell ( i, j ) , we sample noise for grid cell ( k, l ) using
Revised: To classify pixels in grid cell ( i, j ) , we sample noise for grid cell ( k, l ) using

Original: N (0 , σ (cid:48) ) , with σ (cid:48) ∈ [ σ min , σ max ] chosen proportional to the distance of ( i, j ) and ( k, l ) (more details in Section E.2.1).
Revised: N (0 , σ (cid:48) ) , with σ (cid:48) ∈ [ σ min , σ max ] chosen proportional to the distance of ( i, j ) and ( k, l ) (more details in Section E.2.1).

Original: As the base model, we load the one trained with σ train = σ min .
Revised: As the base model, we load the one trained with σ train = σ min .

Original: Using the same distribution at train and test time for the i.i.d.
Revised: Using the same distribution at train and test time for the i.i.d.

Original: baselines but not for localized smoothing is meant to skew the results in the baseline’s favor.
Revised: baselines but not for localized smoothing is meant to skew the results in the baseline’s favor.

Original: But, in Section E.2.3, we also repeat our experiments using the same base model for i.i.d.
Revised: But, in Section E.2.3, we also repeat our experiments using the same base model for i.i.d.

Original: and localized smoothing.
Revised: and localized smoothing.

Original: Evaluation.
Revised: Evaluation.

Original: The main goal of our experiments on segmentation is to verify that localized smoothing can offer a better trade-off between accuracy and certiﬁable robustness.
Revised: The main goal of our experiments on segmentation is to verify that localized smoothing can offer a better trade-off between accuracy and certiﬁable robustness.

Original: That is, for all or most σ , there are σ min , σ max such that the locally smoothed model has higher accuracy and certiﬁable collective robustness than i.i.d.
Revised: That is, for all or most σ , there are σ min , σ max such that the locally smoothed model has higher accuracy and certiﬁable collective robustness than i.i.d.

Original: smoothing baselines using N (0 , σ ) .
Revised: smoothing baselines using N (0 , σ ) .

Original: Because σ, σ min , σ max ∈ R + , we can not evaluate all possible combinations.
Revised: Because σ, σ min , σ max ∈ R + , we can not evaluate all possible combinations.

Original: We therefore use the following scheme: We focus on the case σ ∈ [0 , 0 . 5] , which covers all distributions used in (Kumar & Goldstein, 2021) and
Revised: We therefore use the following scheme: We focus on the case σ ∈ [0 , 0 . 5] , which covers all distributions used in (Kumar & Goldstein, 2021) and

Original: (Fischer et al., 2021).
Revised: (Fischer et al., 2021).

Original: First, we evaluate our two baselines for ﬁve σ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .
Revised: First, we evaluate our two baselines for ﬁve σ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } .

Original: This results in baseline models with diverse levels of accuracy and robustness (e.g. the accuracy of the na¨ıve baseline shrinks from 87 .
Revised: This results in baseline models with diverse levels of accuracy and robustness (e.g. the accuracy of the na¨ıve baseline shrinks from 87 .

Original: 7% to 64 .
Revised: 7% to 64 .

Original: 9% and the AUC of its certiﬁed accuracy grows from 0 .
Revised: 9% and the AUC of its certiﬁed accuracy grows from 0 .

Original: 17 to 0 . 644 ).
Revised: 17 to 0 . 644 ).

Original: We then test whether, for each of the σ , we can ﬁnd σ min , σ max such that the locally smoothed models attains higher accuracy and is certiﬁably more robust.
Revised: We then test whether, for each of the σ , we can ﬁnd σ min , σ max such that the locally smoothed models attains higher accuracy and is certiﬁably more robust.

Original: Finally, to verify that { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } were not just a particularly poor choice of baseline parameters, we ﬁx the chosen σ min , σ max .
Revised: Finally, to verify that { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } were not just a particularly poor choice of baseline parameters, we ﬁx the chosen σ min , σ max .

Original: We then perform a ﬁne-grained search over σ ∈ [0 , 0 . 5] with resolution 0 .to
Revised: We then perform a ﬁne-grained search over σ ∈ [0 , 0 . 5] with resolution 0 .to

Original: ﬁnd a baseline model that has at least the same accuracy and certiﬁable robustness (as measured by certiﬁcate AUC) as any of the ﬁxed locally smoothed models.
Revised: ﬁnd a baseline model that has at least the same accuracy and certiﬁable robustness (as measured by certiﬁcate AUC) as any of the ﬁxed locally smoothed models.

Original: If this is not possible, this provides strong evidence that the proposed smoothing scheme and certiﬁcate indeed offer a better trade-off.
Revised: If this is not possible, this provides strong evidence that the proposed smoothing scheme and certiﬁcate indeed offer a better trade-off.

Original: Fig.
Revised: Fig.

Original: 2 shows one example.
Revised: 2 shows one example.

Original: For σ = 0 .
Revised: For σ = 0 .

Original: 4 , the na¨ıve i.i.d.
Revised: 4 , the na¨ıve i.i.d.

Original: baseline has an accuracy of 72 . 5% .
Revised: baseline has an accuracy of 72 . 5% .

Original: With σ min = 0 .
Revised: With σ min = 0 .

Original: 25 , σ max = 1 . 5 , the proposed localized smoothing certiﬁcate yields both a higher accuracy of 76 . 4% and a higher certiﬁed accuracy for all (cid:15) .
Revised: 25 , σ max = 1 . 5 , the proposed localized smoothing certiﬁcate yields both a higher accuracy of 76 . 4% and a higher certiﬁed accuracy for all (cid:15) .

Original: It can certify robustness for (cid:15) up to 1 . 825 , compared to 1 .
Revised: It can certify robustness for (cid:15) up to 1 . 825 , compared to 1 .

Original: 45 of the baseline and the AUC of its certiﬁed accuracy curve is 43 .
Revised: 45 of the baseline and the AUC of its certiﬁed accuracy curve is 43 .

Original: 1% larger.
Revised: 1% larger.

Original: Fig.also highlights the usefulness of the linear program we derived in Section 5:
Revised: Fig.also highlights the usefulness of the linear program we derived in Section 5:

Original: Evaluating the localized smoothing base certiﬁcates independently, i.e. computing the na¨ıve collective certiﬁcate (dotted orange line), is not sufﬁcient for outperforming the baseline.
Revised: Evaluating the localized smoothing base certiﬁcates independently, i.e. computing the na¨ıve collective certiﬁcate (dotted orange line), is not sufﬁcient for outperforming the baseline.

Original: But combining them via the proposed linear program drastically increases the certiﬁed accuracy
Revised: But combining them via the proposed linear program drastically increases the certiﬁed accuracy

Original: The results for all other combinations of smoothing distribution parameters, both baselines and both metrics of certiﬁcate strength can be found in Section E.2.3.
Revised: The results for all other combinations of smoothing distribution parameters, both baselines and both metrics of certiﬁcate strength can be found in Section E.2.3.

Original: Tables 1 and 2 summarize the ﬁrst part of our evaluation procedure, in which we optimize the localized smoothing parameters.
Revised: Tables 1 and 2 summarize the ﬁrst part of our evaluation procedure, in which we optimize the localized smoothing parameters.

Original: Safe for one exception (with σ = 0 . 2 , center smoothing has a lower accuracy, but slightly larger certiﬁed ratio), the locally smoothed models have the same or higher accuracy, but provide stronger robustness guarantees.
Revised: Safe for one exception (with σ = 0 . 2 , center smoothing has a lower accuracy, but slightly larger certiﬁed ratio), the locally smoothed models have the same or higher accuracy, but provide stronger robustness guarantees.

Original: The difference is particularly large for σ ∈ { 0 . 3 , 0 . 4 , 0 . 5 } , where the accuracy of models smoothed with i.i.d.
Revised: The difference is particularly large for σ ∈ { 0 . 3 , 0 . 4 , 0 . 5 } , where the accuracy of models smoothed with i.i.d.

Original: noise drops off, while our localized smoothing distribution preserves the most relevant parts of the image to allow for high accuracy.
Revised: noise drops off, while our localized smoothing distribution preserves the most relevant parts of the image to allow for high accuracy.

Original: Table 5 summarizes the second part of our evaluation scheme, in which we perform a ﬁne-grained search over [0 , 0 . 5] .
Revised: Table 5 summarizes the second part of our evaluation scheme, in which we perform a ﬁne-grained search over [0 , 0 . 5] .

Original: We ﬁnd that there is no σ such that either of the i.i.d.
Revised: We ﬁnd that there is no σ such that either of the i.i.d.

Original: baselines can outperform any of the chosen locally smoothed models w.r.t.
Revised: baselines can outperform any of the chosen locally smoothed models w.r.t.

Original: AUC of their certiﬁed accuracy or certiﬁed ratio curves.
Revised: AUC of their certiﬁed accuracy or certiﬁed ratio curves.

Original: This is ample evidence for our claim that localized smoothing offers a better trade-off than i.i.d.
Revised: This is ample evidence for our claim that localized smoothing offers a better trade-off than i.i.d.

Original: smoothing.
Revised: smoothing.

Original: Also, this better trade-off was achieved without adding much computational cost (an average 0 . 68 s per LP).
Revised: Also, the collective LPs caused little computational overhead (avg. 0 . 68 s per LP, more details in Section E.2.3).

Original: 8. N ODE CLASSIFICATION
Revised: 8. N ODE CLASSIFICATION

Original: Dataset and model.
Revised: Dataset and model.

Original: We evaluate our certiﬁcate for binary data on the Cora-ML node classiﬁcation dataset.
Revised: We evaluate our certiﬁcate for binary data on the Cora-ML node classiﬁcation dataset.

Original: We use two different base-models: Approximate Personalized Propagation of Neural Predictions (APPNP)
Revised: We use two different base-models: Approximate Personalized Propagation of Neural Predictions (APPNP)

Original: (Klicpera et al., 2019) and a 6-layer Graph Convolutional network (GCN) (Kipf & Welling, 2017).
Revised: (Klicpera et al., 2019) and a 6-layer Graph Convolutional network (GCN) (Kipf & Welling, 2017).

Original: Both models have a receptive ﬁeld that covers most or all of the graph, meaning they are softly local.
Revised: Both models have a receptive ﬁeld that covers most or all of the graph, meaning they are softly local.

Original: For details on model and training parameters, see Section E.3.1.
Revised: For details on model and training parameters, see Section E.3.1.

Original: As center smoothing has only been derived for Gaussian smoothing, we only compare to the na¨ıve baseline.
Revised: As center smoothing has only been derived for Gaussian smoothing, we only compare to the na¨ıve baseline.

Original: For both, the baseline and our localized smoothing certiﬁcate, we use sparsity-aware randomized smoothing (Bojchevski et al., 2020) , i.e. ﬂip 1 -bits and 0 -bits with different probabilities ( θ − and θ + , respectively), which allows us to certify different levels of robustness to deletions and additions of bits.
Revised: For both, the baseline and our localized smoothing certiﬁcate, we use sparsity-aware randomized smoothing (Bojchevski et al., 2020) , i.e. ﬂip 1 -bits and 0 -bits with different probabilities ( θ − and θ + , respectively), which allows us to certify different levels of robustness to deletions and additions of bits.

Original: With localized randomized smoothing, we use the variance smoothing base certiﬁcate derived in Section B.2.2.
Revised: With localized randomized smoothing, we use the variance smoothing base certiﬁcate derived in Section B.2.2.

Original: We choose the distribution parameters for localized smoothing based on an assumption of homophily, i.e. nearby nodes are most relevant for classifying a node.
Revised: We choose the distribution parameters for localized smoothing based on an assumption of homophily, i.e. nearby nodes are most relevant for classifying a node.

Original: We partition the graph into 5 clusters and deﬁne parameters θ ± min and θ ± max .
Revised: We partition the graph into 5 clusters and deﬁne parameters θ ± min and θ ± max .

Original: When classifying a node in cluster i , we randomly smooth attributes in cluster j with θ + ij , θ − ij that are based on linearly interpolating in [ θ − min , θ − max ] and [ θ − min , θ − max ] based on the afﬁnity of the clusters (details in Section E.3.1).
Revised: When classifying a node in cluster i , we randomly smooth attributes in cluster j with θ + ij , θ − ij that are based on linearly interpolating in [ θ − min , θ − max ] and [ θ − min , θ − max ] based on the afﬁnity of the clusters (details in Section E.3.1).

Original: Evaluation.
Revised: Evaluation.

Original: We ﬁrst evaluate the new variance-based certiﬁcate and compare it to the certiﬁcate derived by Bojchevski et al.
Revised: We ﬁrst evaluate the new variance-based certiﬁcate and compare it to the certiﬁcate derived by Bojchevski et al.

Original: For this, we use only one cluster, meaning we use the same smoothing distribution for both.
Revised: For this, we use only one cluster, meaning we use the same smoothing distribution for both.

Original: Fig.
Revised: Fig.

Original: 11 in Section E.3 shows that the variance certiﬁcate is weaker than the baseline for additions, but better for deletions.
Revised: 11 in Section E.3 shows that the variance certiﬁcate is weaker than the baseline for additions, but better for deletions.

Original: It appears sufﬁciently effective to be used as a base certiﬁcate and integrated into a stronger, collective certiﬁcate.
Revised: It appears sufﬁciently effective to be used as a base certiﬁcate and integrated into a stronger, collective certiﬁcate.

Original: The parameter space of our smoothing distributions is large.
Revised: The parameter space of our smoothing distributions is large.

Original: For the localized approach we have four continuous parameters, as we have to specify both the minimal and maximal noise values.
Revised: For the localized approach we have four continuous parameters, as we have to specify both the minimal and maximal noise values.

Original: [Figure]
Revised: [Figure]

Original: C e r ti ﬁ e d acc u r ac y
Revised: C e r ti ﬁ e d acc u r ac y

Original: LP localized
Revised: LP localized

Original: Na¨ıve localized
Revised: Na¨ıve localized

Original: Na¨ıve i.i.d.
Revised: Na¨ıve i.i.d.

Original: Number of adversarial deletions 0 .0 .0 .0 .0 .1 .
Revised: Number of adversarial deletions 0 .0 .0 .0 .0 .1 .

Original: C e r ti ﬁ e d acc u r ac y
Revised: C e r ti ﬁ e d acc u r ac y

Original: LP localized
Revised: LP localized

Original: Na¨ıve localized
Revised: Na¨ıve localized

Original: Na¨ıve i.i.d.
Revised: Na¨ıve i.i.d.

Original: Therefore, it is difﬁcult to show that our approach achieves a better accuracy-robustness trade-off over the whole noise space.
Revised: Therefore, it is difﬁcult to show that our approach achieves a better accuracy-robustness trade-off over the whole noise space.

Original: However, we can investigate the accuracy-robustness trade-off within some areas of this space.
Revised: However, we can investigate the accuracy-robustness trade-off within some areas of this space.

Original: For the localized approach we choose a few ﬁxed combinations of the noise parameters θ ± min and θ ± max .
Revised: For the localized approach we choose a few ﬁxed combinations of the noise parameters θ ± min and θ ± max .

Original: To show our claim, we then optimise the baselines with parameters in an interval around our θ +min and θ − min .
Revised: To show our claim, we then optimise the baselines with parameters in an interval around our θ +min and θ − min .

Original: This is a smaller space, as the baselines only have two parameters.
Revised: This is a smaller space, as the baselines only have two parameters.

Original: We select the baseline whose certiﬁed accuracy curve has the largest AUC.
Revised: We select the baseline whose certiﬁed accuracy curve has the largest AUC.

Original: We perform the search for the best baseline for the addition and deletion scenario independently, i.e., the best baseline model for addition and deletion does not have to be the same.
Revised: We perform the search for the best baseline for the addition and deletion scenario independently, i.e., the best baseline model for addition and deletion does not have to be the same.

Original: In Fig.
Revised: In Fig.

Original: 3, we see the certiﬁed accuracy of an APPNP model for a varying number of attribute additions and deletions (left and right respectively).
Revised: 3, we see the certiﬁed accuracy of an APPNP model for a varying number of attribute additions and deletions (left and right respectively).

Original: To ﬁnd the best distribution parameters for the baselines, we evaluated combinations of θ + ∈ { 0 . 04 , 0 . 055 , 0 . 07 } and θ − ∈ [0 . 1 , . . .
Revised: To ﬁnd the best distribution parameters for the baselines, we evaluated combinations of θ + ∈ { 0 . 04 , 0 . 055 , 0 . 07 } and θ − ∈ [0 . 1 , . . .

Original: 827] , using 11 equally spaced values for the interval.
Revised: 827] , using 11 equally spaced values for the interval.

Original: For adversarial additions, the best baseline yields a certiﬁed accuracy curve with an AUC of 4 . 51 compared to our 5 . 65 .
Revised: For adversarial additions, the best baseline yields a certiﬁed accuracy curve with an AUC of 4 . 51 compared to our 5 . 65 .

Original: The best baseline for deletions has an AUC of 7 .
Revised: The best baseline for deletions has an AUC of 7 .

Original: 76 compared to our 16 .
Revised: 76 compared to our 16 .

Original: Our method outperforms these optimized baselines for most adversarial budgets, while maintaining the same clean accuracy (i.e. certiﬁed accuracy at (cid:15) = 0 ).
Revised: Our method outperforms these optimized baselines for most adversarial budgets, while maintaining the same clean accuracy (i.e. certiﬁed accuracy at (cid:15) = 0 ).

Original: Experiments with different noise parameters and classiﬁers can be found in Section E.3.
Revised: Experiments with different noise parameters and classiﬁers can be found in Section E.3.

Original: In general, we ﬁnd that we signiﬁcantly outperform the baseline when certifying robustness to deletions, but often have weaker certiﬁcates for additions (which may be inherent to the variance smoothing base certiﬁcates).
Revised: In general, we ﬁnd that we signiﬁcantly outperform the baseline when certifying robustness to deletions, but often have weaker certiﬁcates for additions (which may be inherent to the variance smoothing base certiﬁcates).

Original: Due to the large continuous parameter space, we cannot claim that localized smoothing outperforms the na ¨ ıve baseline everywhere.
Revised: Due to the large continuous parameter space, we cannot claim that localized smoothing outperforms the na ¨ ıve baseline everywhere.

Original: However, our results show that, for the tested parameter regions, localized smoothing can provide a signiﬁcantly better accuracy-robustness trade-off.
Revised: However, our results show that, for the tested parameter regions, localized smoothing can provide a signiﬁcantly better accuracy-robustness trade-off.

Original: In all experiments, we found that using the collective LP instead of na¨ıvely combining the base certiﬁcates resulted in much stronger certiﬁcates: The AUC of the certiﬁed accuracy curve (averaged over all experiments) increased by 38 . 8% and 33 .
Revised: We found that using the collective LP instead of na¨ıvely combining the base certiﬁcates can result in much stronger certiﬁcates: The AUC of the certiﬁed accuracy curve (averaged over all experiments) increased by 38 . 8% and 33 .

Original: 6% for addition and deletion, respectively.
Revised: 6% for addition and deletion, respectively.

Original: The  computational overhead for solving each LP was small ( 10 . 9 s on average).
Revised: The collective LPs caused little computational overhead (avg. 10 . 9 s per LP, more details in Section E.3.3).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we have proposed the ﬁrst collective robustness certiﬁcate for softly local multi-output classiﬁers.
Revised: In this work, we have proposed the ﬁrst collective robustness certiﬁcate for softly local multi-output classiﬁers.

Original: It is based on localized randomized smoothing, i.e randomly smoothing different outputs using different non-i.i.d.
Revised: It is based on localized randomized smoothing, i.e. randomly smoothing different outputs using different non-i.i.d.

Original: smoothing distributions matching the model’s locality.
Revised: smoothing distributions matching the model’s locality.

Original: We have shown how per-output certiﬁcates based on localized smoothing can be computed and that they share a common interface.
Revised: We have shown how per-output certiﬁcates based on localized smoothing can be computed and that they share a common interface.

Original: This interface allows them to be combined into a strong collective robustness certiﬁcate.
Revised: This interface allows them to be combined into a strong collective robustness certiﬁcate.

Original: Experiments on image segmentation and node classiﬁcation tasks demonstrate that localized smoothing can offer a better robustness-accuracy trade-off than existing randomized smoothing techniques.
Revised: Experiments on image segmentation and node classiﬁcation tasks demonstrate that localized smoothing can offer a better robustness-accuracy trade-off than existing randomized smoothing techniques.

Original: Our results show that locality is linked to robustness, which suggests the research direction of building more effective local models to robustly solve multi-output tasks.
Revised: Our results show that locality is linked to robustness, which suggests the research direction of building more effective local models to robustly solve multi-output tasks.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: We prove all theoretic results that were not already derived in the main text in Appendices A to C. To ensure reproducibility of the experimental results we provide detailed descriptions of the evaluation process with the respective parameters in Section E.2 and Section E.3.
Revised: We prove all theoretic results that were not already derived in the main text in Appendices A to C. To ensure reproducibility of the experimental results we provide detailed descriptions of the evaluation process with the respective parameters in Section E.2 and Section E.3.

Original: Code will be made available to reviewers via an anonymous link posted on OpenReview, as suggested by the guidelines.
Revised: Code will be made available to reviewers via an anonymous link posted on OpenReview, as suggested by the guidelines.

Original:  E THICS STATEMENT
Revised:  E THICS STATEMENT

Original: In this paper we propose a method to increase the robustness of machine learning models against adversarial perturbations and to certify their robustness.
Revised: In this paper we propose a method to increase the robustness of machine learning models against adversarial perturbations and to certify their robustness.

Original: We see this as an important step towards general usage of models in practice, as many existing methods are brittle to crafted attacks.
Revised: We see this as an important step towards general usage of models in practice, as many existing methods are brittle to crafted attacks.

Original: Through the proposed method, we hope to contribute to the safe usage of machine learning.
Revised: Through the proposed method, we hope to contribute to the safe usage of machine learning.

Original: However, robust models also have to be seen with caution.
Revised: However, robust models also have to be seen with caution.

Original: As they are harder to fool, harmful purposes like mass surveillance are harder to avoid.
Revised: As they are harder to fool, harmful purposes like mass surveillance are harder to avoid.

Original: We believe that it is still necessary to further research robustness of machine learning models as the positive effects can outweigh the negatives, but it is necessary to discuss the ethical implications of the usage in any speciﬁc application area
Revised: We believe that it is still necessary to further research robustness of machine learning models as the positive effects can outweigh the negatives, but it is necessary to discuss the ethical implications of the usage in any speciﬁc application area

Original: A I MPROVING EFFICIENCY
Revised: A I MPROVING EFFICIENCY

Original: In this section, we discuss different modiﬁcations to our collective certiﬁcate that improve its sample efﬁciency and allow us ﬁne-grained control over the size of the collective linear program.
Revised: In this section, we discuss different modiﬁcations to our collective certiﬁcate that improve its sample efﬁciency and allow us ﬁne-grained control over the size of the collective linear program.

Original: We further discuss a linear relaxation of our collective linear program.
Revised: We further discuss a linear relaxation of our collective linear program.

Original: All of the modiﬁcations preserve the soundness of our collective certiﬁcate, i.e. we still obtain a provable bound on the number of predictions that can be simultaneously attacked by an adversary.
Revised: All of the modiﬁcations preserve the soundness of our collective certiﬁcate, i.e. we still obtain a provable bound on the number of predictions that can be simultaneously attacked by an adversary.

Original: To avoid constant case distinctions, we ﬁrst present all results for real-valued data, i.e. X =
Revised: To avoid constant case distinctions, we ﬁrst present all results for real-valued data, i.e. X =

Original: R , before mentioning any additional precautions that may be needed when working with binary data.
Revised: R , before mentioning any additional precautions that may be needed when working with binary data.

Original: A. S HARING SMOOTHING DISTRIBUTIONS AMONG OUTPUTS
Revised: A. S HARING SMOOTHING DISTRIBUTIONS AMONG OUTPUTS

Original: In principle, our proposed certiﬁcate allows a different smoothing distribution Ψ ( n ) to be used per output g n of our base model.
Revised: In principle, our proposed certiﬁcate allows a different smoothing distribution Ψ ( n ) to be used per output g n of our base model.

Original: In practice, where we have to estimate properties of the smoothed classiﬁer using Monte Carlo methods, this is problematic:
Revised: In practice, where we have to estimate properties of the smoothed classiﬁer using Monte Carlo methods, this is problematic:

Original: Samples cannot be re-used, each of the many outputs requires its own round of sampling.
Revised: Samples cannot be re-used, each of the many outputs requires its own round of sampling.

Original: We can increase the efﬁciency of our localized smoothing approach by partitioning our D out outputs into N out subsets that share the same smoothing distribution.
Revised: We can increase the efﬁciency of our localized smoothing approach by partitioning our D out outputs into N out subsets that share the same smoothing distribution.

Original: When making smoothed predictions or computing base certiﬁcates, we can then reuse the same samples for all outputs within each subsets.
Revised: When making smoothed predictions or computing base certiﬁcates, we can then reuse the same samples for all outputs within each subsets.

Original: More formally, we partition our D out output dimensions into sets K (1) , . . .
Revised: More formally, we partition our D out output dimensions into sets K (1) , . . .

Original: , K ( N out) with
Revised: , K ( N out) with

Original: [Equation]
Revised: [Equation]

Original: We then associate each set K ( i ) with a smoothing distribution Ψ ( i ) .
Revised: We then associate each set K ( i ) with a smoothing distribution Ψ ( i ) .

Original: For each base model output g n with n ∈ K ( i ) , we then use smoothing distribution Ψ ( i ) to construct the smoothed output f n , e.g. f n ( x ) =
Revised: For each base model output g n with n ∈ K ( i ) , we then use smoothing distribution Ψ ( i ) to construct the smoothed output f n , e.g. f n ( x ) =

Original: argmax y ∈ Y Pr z ∼ Ψ
Revised: argmax y ∈ Y Pr z ∼ Ψ

Original: ( i )
Revised: ( i )

Original: [ f ( x + z )
Revised: [ f ( x + z )

Original: =
Revised: =

Original: y ] (note that we use a different smoothing paradigm for binary data, see Section 5).
Revised: y ] (note that we use a different smoothing paradigm for binary data, see Section 5).

Original: A. Q UANTIZING CERTIFICATE PARAMETERS
Revised: A. Q UANTIZING CERTIFICATE PARAMETERS

Original: Recall that our base certiﬁcates from Section 5 are deﬁned by a linear inequality: A prediction y n = f n ( x ) is robust to a perturbed input x
Revised: Recall that our base certiﬁcates from Section 5 are deﬁned by a linear inequality: A prediction y n = f n ( x ) is robust to a perturbed input x

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: X D in if (cid:80) Dd =1 w ( n ) d · | x (cid:48)
Revised: X D in if (cid:80) Dd =1 w ( n ) d · | x (cid:48)

Original: d − x d | p < η ( n ) , for some p ≥ 0 .
Revised: d − x d | p < η ( n ) , for some p ≥ 0 .

Original: The weight vectors w ( n ) ∈ R D in only depend on the smoothing distributions.
Revised: The weight vectors w ( n ) ∈ R D in only depend on the smoothing distributions.

Original: A side of effect of sharing the same smoothing Ψ ( i ) among all outputs from a set K ( i ) , as discussed in the previous section, is that the outputs also share the same weight vector w ( i
Revised: A side of effect of sharing the same smoothing Ψ ( i ) among all outputs from a set K ( i ) , as discussed in the previous section, is that the outputs also share the same weight vector w ( i

Original: ) ∈ R D in with ∀ n ∈ K ( i ) :
Revised: ) ∈ R D in with ∀ n ∈ K ( i ) :

Original: w ( i ) =
Revised: w ( i ) =

Original: w ( n ) .
Revised: w ( n ) .

Original: Thus, for all smoothed outputs f n with n ∈ K ( i ) , the smoothed prediction y n is robust if (cid:80) Dd =1 w ( i ) d · | x (cid:48)
Revised: Thus, for all smoothed outputs f n with n ∈ K ( i ) , the smoothed prediction y n is robust if (cid:80) Dd =1 w ( i ) d · | x (cid:48)

Original: d − x d | p < η ( n ) .
Revised: d − x d | p < η ( n ) .

Original: Evidently, the base certiﬁcates for outputs from a set K ( i ) only differ in their parameter η ( n ) .
Revised: Evidently, the base certiﬁcates for outputs from a set K ( i ) only differ in their parameter η ( n ) .

Original: Recall that in our collective linear program we use a vector of variables t ∈ { 0 , 1 } D out to indicate which predictions are robust according to their base certiﬁcates (see Section 6).
Revised: Recall that in our collective linear program we use a vector of variables t ∈ { 0 , 1 } D out to indicate which predictions are robust according to their base certiﬁcates (see Section 6).

Original: If there are two outputs f n and f m with η ( n )
Revised: If there are two outputs f n and f m with η ( n )

Original: =
Revised: =

Original: η ( m ) , then f n and f m have the same base certiﬁcate and their robustness can be modelled by the same indicator variable.
Revised: η ( m ) , then f n and f m have the same base certiﬁcate and their robustness can be modelled by the same indicator variable.

Original: Conversely, for each set of outputs K ( i ) , we only need one indicator variable per unique η ( n ) .
Revised: Conversely, for each set of outputs K ( i ) , we only need one indicator variable per unique η ( n ) .

Original: By quantizing the η ( n ) within each subset K ( i ) (for example by deﬁning equally sized bins between min n ∈ K ( i ) η ( n ) and max n ∈ K ( i ) η ( n ) ), we can ensure that there is always a ﬁxed number N bins of indicator variables per subset.
Revised: By quantizing the η ( n ) within each subset K ( i ) (for example by deﬁning equally sized bins between min n ∈ K ( i ) η ( n ) and max n ∈ K ( i ) η ( n ) ), we can ensure that there is always a ﬁxed number N bins of indicator variables per subset.

Original: This way, we can reduce the number of indicator variables from D out to N out · N bins .
Revised: This way, we can reduce the number of indicator variables from D out to N out · N bins .

Original: To implement this idea, we deﬁne matrix of thresholds E ∈ R N out × N bins with ∀ i : min { E i, : } ≤ min n ∈ K ( i ) (cid:0)(cid:8) η ( n ) | n ∈ K
Revised: To implement this idea, we deﬁne matrix of thresholds E ∈ R N out × N bins with ∀ i : min { E i, : } ≤ min n ∈ K ( i ) (cid:0)(cid:8) η ( n ) | n ∈ K

Original: ( i ) (cid:9)(cid:1) .
Revised: ( i ) (cid:9)(cid:1) .

Original: We then deﬁne a function ξ : {
Revised: We then deﬁne a function ξ : {

Original: [Equation]
Revised: [Equation]

Original: , N out } × R
Revised: , N out } × R

Original: → R with
Revised: → R with

Original: [Equation]
Revised: [Equation]

Original: that quantizes base certiﬁcate parameter η from output subset K ( i ) by mapping it to the next smallest threshold in E i, : .
Revised: that quantizes base certiﬁcate parameter η from output subset K ( i ) by mapping it to the next smallest threshold in E i, : .

Original: For feasibility, like in Section 6 we need to compute the constant η ( i ) =
Revised: For feasibility, like in Section 6 we need to compute the constant η ( i ) =

Original: min b ∈ R D in+ b T w ( i ) d s.t. sum { b } ≤ (cid:15) p to ensure feasibility of the problem.
Revised: min b ∈ R D in+ b T w ( i ) d s.t. sum { b } ≤ (cid:15) p to ensure feasibility of the problem.

Original: Note that, because all outputs from a subset K ( i ) share the same weight vector w ( i ) , we only have to compute this constant once per subset.
Revised: Note that, because all outputs from a subset K ( i ) share the same weight vector w ( i ) , we only have to compute this constant once per subset.

Original: We can bound the collective robustness of the targeted dimensions T of our vector of predictions y
Revised: We can bound the collective robustness of the targeted dimensions T of our vector of predictions y

Original: =
Revised: =

Original: f
Revised: f

Original: ( x ) as follows:
Revised: ( x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Constraint Eq.
Revised: Constraint Eq.

Original: 9 ensures that T i,j is only set to 

Revised: 9 ensures that T i,j is only set to

Original: if b T w ( i ) ≥ E i,j , i.e. all predictions from subset
Revised: if b T w ( i ) ≥ E i,j , i.e. all predictions from subset

Original: K ( i ) whose base certiﬁcate parameter η ( n ) is quantized to E i,j are no longer robust.
Revised: K ( i ) whose base certiﬁcate parameter η ( n ) is quantized to E i,j are no longer robust.

Original: When this is the case, the objective function decreases by the number of these predictions.
Revised: When this is the case, the objective function decreases by the number of these predictions.

Original: For N out =
Revised: For N out =

Original: D out ,
Revised: D out ,

Original: N bins = 1 and E n, 1 = η ( n ) , we recover our general certiﬁcate from Section 6.
Revised: N bins = 1 and E n, 1 = η ( n ) , we recover our general certiﬁcate from Section 6.

Original: Note that, if the quantization maps any parameter η ( n ) to a smaller number, the set H ( n ) becomes more restrictive, i.e. y n is considered robust to a smaller set of perturbed inputs.
Revised: Note that, if the quantization maps any parameter η ( n ) to a smaller number, the set H ( n ) becomes more restrictive, i.e. y n is considered robust to a smaller set of perturbed inputs.

Original: Thus, Eq. 8 is a lower bound on our general certiﬁcate from Section 6.
Revised: Thus, Eq. 8 is a lower bound on our general certiﬁcate from Section 6.

Original: A. S HARING NOISE LEVELS AMONG INPUTS
Revised: A. S HARING NOISE LEVELS AMONG INPUTS

Original: Similar to how partitioning the output dimensions allows us to control the number of output variables t , partitioning the input dimensions and using the same noise level within each partition allows us to control the number of variables
Revised: Similar to how partitioning the output dimensions allows us to control the number of output variables t , partitioning the input dimensions and using the same noise level within each partition allows us to control the number of variables

Original: b that model the allocation of adversarial budget.
Revised: b that model the allocation of adversarial budget.

Original: Assume that we have partitioned our output dimensions into N out subsets K (1) , . .
Revised: Assume that we have partitioned our output dimensions into N out subsets K (1) , . .

Original: , K ( N out , with outputs in each subset sharing the same smoothing distribution Ψ ( i ) , as explained in Section A.1.
Revised: , K ( N out , with outputs in each subset sharing the same smoothing distribution Ψ ( i ) , as explained in Section A.1.

Original: Let us now deﬁne N in input subsets J (1) , . . .
Revised: Let us now deﬁne N in input subsets J (1) , . . .

Original: , J ( N in ) with
Revised: , J ( N in ) with

Original: [Equation]
Revised: [Equation]

Original: Recall that a prediction y n = f n ( x ) with n ∈ K ( i ) is robust to a perturbed input x
Revised: Recall that a prediction y n = f n ( x ) with n ∈ K ( i ) is robust to a perturbed input x

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: X D in if (cid:80) Dd =1 w ( i ) d
Revised: X D in if (cid:80) Dd =1 w ( i ) d

Original: · | x (cid:48)
Revised: · | x (cid:48)

Original: d − x d | p < η ( n ) and that the weight vectors w ( i ) only depend on the smoothing distributions.
Revised: d − x d | p < η ( n ) and that the weight vectors w ( i ) only depend on the smoothing distributions.

Original: Assume that we choose each smoothing distribution Ψ ( i ) such that ∀ l ∈ { 1 , . .
Revised: Assume that we choose each smoothing distribution Ψ ( i ) such that ∀ l ∈ { 1 , . .

Original: , N in } , ∀ d, d (cid:48)
Revised: , N in } , ∀ d, d (cid:48)

Original: ∈ J ( l ) : w ( i ) d
Revised: ∈ J ( l ) : w ( i ) d

Original: = w ( i ) d (cid:48) , i.e. all input dimensions within each set J ( l ) have the same weight.
Revised: = w ( i ) d (cid:48) , i.e. all input dimensions within each set J ( l ) have the same weight.

Original: This can be achieved by choosing Ψ ( i ) so that all dimensions in each input subset J l are smoothed with the noise level (note that we can still use different Ψ ( i ) , i.e. different noise levels for smoothing different sets of outputs).
Revised: This can be achieved by choosing Ψ ( i ) so that all dimensions in each input subset J l are smoothed with the noise level (note that we can still use different Ψ ( i ) , i.e. different noise levels for smoothing different sets of outputs).

Original: For example, one could use a Gaussian distribution with covariance matrix Σ = diag ( σ ) 2 with ∀ l ∈ { 1 , . . .
Revised: For example, one could use a Gaussian distribution with covariance matrix Σ = diag ( σ ) 2 with ∀ l ∈ { 1 , . . .

Original: , N in } , ∀ d, d (cid:48)
Revised: , N in } , ∀ d, d (cid:48)

Original: ∈ J ( l ) : σ d = σ d (cid:48) .
Revised: ∈ J ( l ) : σ d = σ d (cid:48) .

Original: In this case, the evaluation of our base certiﬁcates can be simpliﬁed.
Revised: In this case, the evaluation of our base certiﬁcates can be simpliﬁed.

Original: Prediction y n = f n ( x ) is robust to a perturbed input x
Revised: Prediction y n = f n ( x ) is robust to a perturbed input x

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: X D in if
Revised: X D in if

Original: D
Revised: D

Original: [Equation]
Revised: [Equation]

Original: (12) =
Revised: (12) =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 13) with u ∈ R N in and ∀ i ∈ { 1 , . . .
Revised: 13) with u ∈ R N in and ∀ i ∈ { 1 , . . .

Original: , N out } , ∀ l ∈ { 1 , . . .
Revised: , N out } , ∀ l ∈ { 1 , . . .

Original: , N in } , ∀ d ∈ J : u il = w i
Revised: , N in } , ∀ d ∈ J : u il = w i

Original: d .
Revised: d .

Original: That is, we can replace each weight vector w ( i ) that has one weight w ( i ) d per input dimension d with a smaller weight vector u ( i ) with one weight u ( i ) l per input subset J ( l ) .
Revised: That is, we can replace each weight vector w ( i ) that has one weight w ( i ) d per input dimension d with a smaller weight vector u ( i ) with one weight u ( i ) l per input subset J ( l ) .

Original: For our linear program, this means that we no longer need a budget vector b ∈ R D in + to model the element-wise distance | x (cid:48)
Revised: For our linear program, this means that we no longer need a budget vector b ∈ R D in + to model the element-wise distance | x (cid:48)

Original: d − x d | p in each dimension d .
Revised: d − x d | p in each dimension d .

Original: Instead, we can use a smaller budget vector b ∈ R N in + to model the overall distance within each input subset J ( l ) , i.e.
Revised: Instead, we can use a smaller budget vector b ∈ R N in + to model the overall distance within each input subset J ( l ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: Combined with the quantization of certiﬁcate parameters from the previous section, our optimization problem becomes
Revised: Combined with the quantization of certiﬁcate parameters from the previous section, our optimization problem becomes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with u ∈ R N in and ∀ i ∈ { 1 , . . .
Revised: with u ∈ R N in and ∀ i ∈ { 1 , . . .

Original: , N out } , ∀ l ∈ { 1 , . . .
Revised: , N out } , ∀ l ∈ { 1 , . . .

Original: , N in } , ∀ d ∈ J : ω i l = w i
Revised: , N in } , ∀ d ∈ J : ω i l = w i

Original: [Equation]
Revised: [Equation]

Original: For N out =
Revised: For N out =

Original: D out ,
Revised: D out ,

Original: N in = D in , N bins = 1 and E n, 1 = η ( n ) , we recover our general certiﬁcate from Section 6.
Revised: N in = D in , N bins = 1 and E n, 1 = η ( n ) , we recover our general certiﬁcate from Section 6.

Original: When certifying robustness for binary data, we impose different constraints on b .
Revised: When certifying robustness for binary data, we impose different constraints on b .

Original: To model that the adversary can not ﬂip more bits than are present within each subset, we use a budget vector b ∈ N N inwith ∀ l ∈ { 1 , . .
Revised: To model that the adversary can not ﬂip more bits than are present within each subset, we use a budget vector b ∈ N N inwith ∀ l ∈ { 1 , . .

Original: , N in } : b l ≤ (cid:12)(cid:12)
Revised: , N in } : b l ≤ (cid:12)(cid:12)

Original: J
Revised: J

Original: ( l ) (cid:12)(cid:12) , instead of a continuous budget vector b ∈ R N in + .
Revised: ( l ) (cid:12)(cid:12) , instead of a continuous budget vector b ∈ R N in + .

Original: A.
Revised: A.

Original: L INEAR RELAXATION
Revised: L INEAR RELAXATION

Original: Combining the previous steps allows us to reduce the number of problem variables and linear constraints from D in + D out
Revised: Combining the previous steps allows us to reduce the number of problem variables and linear constraints from D in + D out

Original: and D out + 1 to N in + N out · N bins and N out · N bins + 1 , respectively.
Revised: and D out + 1 to N in + N out · N bins and N out · N bins + 1 , respectively.

Original: Still, ﬁnding an optimal solution to the mixed-integer linear program may be too expensive.
Revised: Still, ﬁnding an optimal solution to the mixed-integer linear program may be too expensive.

Original: One can obtain a lower bound on the optimal value and thus a valid, albeit more pessimistic, robustness certiﬁcate by relaxing all to be continuous.
Revised: One can obtain a lower bound on the optimal value and thus a valid, albeit more pessimistic, robustness certiﬁcate by relaxing all to be continuous.

Original: When using the general certiﬁcate from Section 6, the binary vector t ∈ { 0 , 1 } D out can be relaxed to t ∈ [0 , 1] D out .
Revised: When using the general certiﬁcate from Section 6, the binary vector t ∈ { 0 , 1 } D out can be relaxed to t ∈ [0 , 1] D out .

Original: When using the certiﬁcate with quantized base certiﬁcate parameters from Section A.2 or Section A.3, the binary matrix T ∈ [0 , 1] N out × N bins can be relaxed to T ∈ [0 , 1] N out × N bins .
Revised: When using the certiﬁcate with quantized base certiﬁcate parameters from Section A.2 or Section A.3, the binary matrix T ∈ [0 , 1] N out × N bins can be relaxed to T ∈ [0 , 1] N out × N bins .

Original: Conceptually, this means that predictions can be partially certiﬁed, i.e. t n ∈ (0 , 1) or T i,j ∈ (0 , 1) .
Revised: Conceptually, this means that predictions can be partially certiﬁed, i.e. t n ∈ (0 , 1) or T i,j ∈ (0 , 1) .

Original: In particular, a prediction can be partially certiﬁed even if we know that is impossible to attack under the collective perturbation model B x =
Revised: In particular, a prediction can be partially certiﬁed even if we know that is impossible to attack under the collective perturbation model B x =

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: − x |
Revised: − x |

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:9) .
Revised: (cid:9) .

Original: Just like Schuchardt et al.
Revised: Just like Schuchardt et al.

Original: (2021), who encountered the same problem with their collective certiﬁcate, we circumvent this issue by ﬁrst computing a set L ⊆ T of all targeted predictions in T that are guaranteed to always be robust:
Revised: (2021), who encountered the same problem with their collective certiﬁcate, we circumvent this issue by ﬁrst computing a set L ⊆ T of all targeted predictions in T that are guaranteed to always be robust:

Original: [Equation]
Revised: [Equation]

Original: The equality follows from the fact that the most effective way of attacking a prediction is to allocate all adversarial budget to the least robust dimension, i.e. the dimension with the largest weight – unless all weights are negative.
Revised: The equality follows from the fact that the most effective way of attacking a prediction is to allocate all adversarial budget to the least robust dimension, i.e. the dimension with the largest weight – unless all weights are negative.

Original: Because we know that all predictions with indices in L are robust, we do not have to include them in the collective optimization problem and can instead compute
Revised: Because we know that all predictions with indices in L are robust, we do not have to include them in the collective optimization problem and can instead compute

Original: [Equation]
Revised: [Equation]

Original: The r.h.s.
Revised: The r.h.s.

Original: optimization can be solved using the general collective certiﬁcate from Section 6 or any of the more efﬁcient, modiﬁed certiﬁcates from previous sections.
Revised: optimization can be solved using the general collective certiﬁcate from Section 6 or any of the more efﬁcient, modiﬁed certiﬁcates from previous sections.

Original: When using the general collective certiﬁcate from Section 6 with binary data, the budget variables b ∈ { 0 , 1 } D in can be relaxed to b ∈ [0 , 1] D in .
Revised: When using the general collective certiﬁcate from Section 6 with binary data, the budget variables b ∈ { 0 , 1 } D in can be relaxed to b ∈ [0 , 1] D in .

Original: When using the modiﬁed collective certiﬁcate from Section A.3, the budget variables with b ∈ N N incan be relaxed to b ∈ R N in + .
Revised: When using the modiﬁed collective certiﬁcate from Section A.3, the budget variables with b ∈ N N incan be relaxed to b ∈ R N in + .

Original: The additional constraint ∀ l ∈ { 1 , . . .
Revised: The additional constraint ∀ l ∈ { 1 , . . .

Original: , N in } : b l ≤ (cid:12)(cid:12)
Revised: , N in } : b l ≤ (cid:12)(cid:12)

Original: J
Revised: J

Original: ( l ) (cid:12)(cid:12) can be kept in order to model that the adversary cannot ﬂip (or partially ﬂip) more bits than are present within each input subset J ( l ) .
Revised: ( l ) (cid:12)(cid:12) can be kept in order to model that the adversary cannot ﬂip (or partially ﬂip) more bits than are present within each input subset J ( l ) .

Original: B B ASE CERTIFICATES
Revised: B B ASE CERTIFICATES

Original: In the following, we show why the base certiﬁcates presented in Section 5 hold and present alternatives for other collective perturbation models.
Revised: In the following, we show why the base certiﬁcates presented in Section 5 hold and present alternatives for other collective perturbation models.

Original: B. G AUSSIAN SMOOTHING FOR l 2 PERTURBATIONS OF CONTINUOUS DATA
Revised: B. G AUSSIAN SMOOTHING FOR l 2 PERTURBATIONS OF CONTINUOUS DATA

Original: Proposition 1.
Revised: Proposition 1.

Original: Given an output g n : R D in → Y , let f n ( x ) =
Revised: Given an output g n : R D in → Y , let f n ( x ) =

Original: argmax y ∈ Y Pr z ∼N ( x , Σ ) [ g n ( z ) =
Revised: argmax y ∈ Y Pr z ∼N ( x , Σ ) [ g n ( z ) =

Original: y ] be the corresponding smoothed output with Σ = diag ( σ ) 2 and σ ∈ R D in + .
Revised: y ] be the corresponding smoothed output with Σ = diag ( σ ) 2 and σ ∈ R D in + .

Original: Given an input x ∈ R D in and smoothed prediction y n = f n ( x ) , let q =
Revised: Given an input x ∈ R D in and smoothed prediction y n = f n ( x ) , let q =

Original: Pr z ∼N ( x , Σ )
Revised: Pr z ∼N ( x , Σ )

Original: [ g n ( z ) =
Revised: [ g n ( z ) =

Original: y n ] .
Revised: y n ] .

Original: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =
Revised: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =

Original: y n with H ( n ) deﬁned as in Eq. 2, w d = 1 σ d 2 , η = (cid:0) Φ ( − 1) ( q ) (cid:1) 2 and κ = 2 .
Revised: y n with H ( n ) deﬁned as in Eq. 2, w d = 1 σ d 2 , η = (cid:0) Φ ( − 1) ( q ) (cid:1) 2 and κ = 2 .

Original: Proof.
Revised: Proof.

Original: Based on the deﬁnition of the base certiﬁcate interface, we need to show that, ∀ x (cid:48) ∈ H : f n ( x (cid:48) )
Revised: Based on the deﬁnition of the base certiﬁcate interface, we need to show that, ∀ x (cid:48) ∈ H : f n ( x (cid:48) )

Original: =
Revised: =

Original: y n with
Revised: y n with

Original: [Equation]
Revised: [Equation]

Original: Eiras et al.
Revised: Eiras et al.

Original: (2021) have shown that under the same conditions as above, but with a general covariance matrix Σ ∈ R D in × D in + , a prediction y n is certiﬁably robust to a perturbed input x (cid:48) if
Revised: (2021) have shown that under the same conditions as above, but with a general covariance matrix Σ ∈ R D in × D in + , a prediction y n is certiﬁably robust to a perturbed input x (cid:48) if

Original: [Equation]
Revised: [Equation]

Original: where q (cid:48)
Revised: where q (cid:48)

Original: =
Revised: =

Original: max y (cid:48) n (cid:54) =
Revised: max y (cid:48) n (cid:54) =

Original: y n Pr z ∼N ( x , Σ ) [ g n ( z ) =
Revised: y n Pr z ∼N ( x , Σ ) [ g n ( z ) =

Original: y (cid:48) n ] is the probability of the second most likely prediction under the smoothing distribution.
Revised: y (cid:48) n ] is the probability of the second most likely prediction under the smoothing distribution.

Original: Because the probabilities of all possible predictions have to sum up to 1 , we have q (cid:48) ≤ 1 − q .
Revised: Because the probabilities of all possible predictions have to sum up to 1 , we have q (cid:48) ≤ 1 − q .

Original: Since Φ − 1 is monotonically increasing, we can obtain a lower bound on the r.h.s.
Revised: Since Φ − 1 is monotonically increasing, we can obtain a lower bound on the r.h.s.

Original: of Eq.
Revised: of Eq.

Original: 21 and thus a more pessimistic certiﬁcate by substituting 1 − q for q (cid:48) (deriving such a ”binary certiﬁcate” from a ”multiclass certiﬁcate” is common in randomized smoothing and was already discussed in (Cohen et al., 2019)):
Revised: 21 and thus a more pessimistic certiﬁcate by substituting 1 − q for q (cid:48) (deriving such a ”binary certiﬁcate” from a ”multiclass certiﬁcate” is common in randomized smoothing and was already discussed in (Cohen et al., 2019)):

Original: [Equation]
Revised: [Equation]

Original: In our case, Σ is a diagonal matrix diag ( σ ) 2 with σ ∈ R D in + .
Revised: In our case, Σ is a diagonal matrix diag ( σ ) 2 with σ ∈ R D in + .

Original: Thus Eq.
Revised: Thus Eq.

Original: 22 is equivalent to
Revised: 22 is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Finally, using the fact that Φ − 1 ( q ) − Φ − 1 (1 − q ) = 2Φ − 1 ( q ) and eliminating the square root shows that we are certiﬁably robust if
Revised: Finally, using the fact that Φ − 1 ( q ) − Φ − 1 (1 − q ) = 2Φ − 1 ( q ) and eliminating the square root shows that we are certiﬁably robust if

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B.1. U NIFORM SMOOTHING FOR l 1 PERTURBATIONS OF CONTINUOUS DATA
Revised: B.1. U NIFORM SMOOTHING FOR l 1 PERTURBATIONS OF CONTINUOUS DATA

Original: An alternative base certiﬁcate for l 1 perturbations is again due to Eiras et al.
Revised: An alternative base certiﬁcate for l 1 perturbations is again due to Eiras et al.

Original: Using uniform instead of Gaussian noise later allows us to collective certify robustness to l 1 -norm-bound perturbations.
Revised: Using uniform instead of Gaussian noise later allows us to collective certify robustness to l 1 -norm-bound perturbations.

Original: In the following U ( x , λ ) with x ∈ R D , λ ∈ R D + refers to a vector-valued random distribution in which the d -th element is uniformly distributed in [ x d − λ d , x d + λ d ] .
Revised: In the following U ( x , λ ) with x ∈ R D , λ ∈ R D + refers to a vector-valued random distribution in which the d -th element is uniformly distributed in [ x d − λ d , x d + λ d ] .

Original: Proposition 2.
Revised: Proposition 2.

Original: Given an output g n : R D in → Y , let f ( x ) =
Revised: Given an output g n : R D in → Y , let f ( x ) =

Original: argmax y ∈ Y
Revised: argmax y ∈ Y

Original: Pr z ∼U
Revised: Pr z ∼U

Original: ( x , λ ) [ g ( z ) =
Revised: ( x , λ ) [ g ( z ) =

Original: y ] be the corresponding smoothed classiﬁer with λ ∈ R D in + .
Revised: y ] be the corresponding smoothed classiﬁer with λ ∈ R D in + .

Original: Given an input x ∈ R D in and smoothed prediction y =
Revised: Given an input x ∈ R D in and smoothed prediction y =

Original: f ( x ) , let p =
Revised: f ( x ) , let p =

Original: Pr z
Revised: Pr z

Original: ∼U ( x , λ ) [ g ( z ) =
Revised: ∼U ( x , λ ) [ g ( z ) =

Original: y ] .
Revised: y ] .

Original: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =
Revised: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =

Original: y n with H ( n ) deﬁned as in Eq. 2, w d = 1 /λ d , η = Φ − 1 ( q ) and κ = 1 .
Revised: y n with H ( n ) deﬁned as in Eq. 2, w d = 1 /λ d , η = Φ − 1 ( q ) and κ = 1 .

Original: Proof.
Revised: Proof.

Original: Based on the deﬁnition of H ( n ) , we need to prove that ∀ x (cid:48) ∈ H : f n ( x (cid:48) )
Revised: Based on the deﬁnition of H ( n ) , we need to prove that ∀ x (cid:48) ∈ H : f n ( x (cid:48) )

Original: =
Revised: =

Original: y n with
Revised: y n with

Original: [Equation]
Revised: [Equation]

Original: Eiras et al.
Revised: Eiras et al.

Original: (2021) have shown that under the same conditions as above, a prediction y n is certiﬁably robust to a perturbed input x (cid:48) if
Revised: (2021) have shown that under the same conditions as above, a prediction y n is certiﬁably robust to a perturbed input x (cid:48) if

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where q (cid:48)
Revised: where q (cid:48)

Original: =
Revised: =

Original: max y (cid:48) n (cid:54) =
Revised: max y (cid:48) n (cid:54) =

Original: y n Pr z ∼U ( x , λ ) [ g n ( z ) =
Revised: y n Pr z ∼U ( x , λ ) [ g n ( z ) =

Original: y (cid:48) n ] is the probability of the second most likely prediction under the smoothing distribution.
Revised: y (cid:48) n ] is the probability of the second most likely prediction under the smoothing distribution.

Original: As in our previous proof for Gaussian smoothing, we can obtain a more pessimistic certiﬁcate by substituting 1 − q for q (cid:48) .
Revised: As in our previous proof for Gaussian smoothing, we can obtain a more pessimistic certiﬁcate by substituting 1 − q for q (cid:48) .

Original: Since Φ − 1 ( q ) − Φ − 1 (1 − q ) = 2Φ − 1 ( q ) and all λ d are non-negative, we know that our prediction is certiﬁably robust if
Revised: Since Φ − 1 ( q ) − Φ − 1 (1 − q ) = 2Φ − 1 ( q ) and all λ d are non-negative, we know that our prediction is certiﬁably robust if

Original: [Equation]
Revised: [Equation]

Original: B. V ARIANCE SMOOTHING
Revised: B. V ARIANCE SMOOTHING

Original: We propose variance smoothing as a base certiﬁcate for binary data.
Revised: We propose variance smoothing as a base certiﬁcate for binary data.

Original: Variance smoothing certiﬁes predictions based on the mean and variance of the softmax score associated with a predicted label.
Revised: Variance smoothing certiﬁes predictions based on the mean and variance of the softmax score associated with a predicted label.

Original: It is in principle applicable to arbitrary data types.
Revised: It is in principle applicable to arbitrary data types.

Original: We focus on discrete data, but all results can be generalized from discrete to continuous data by replacing any sum over probability mass functions with integrals over probability density functions.
Revised: We focus on discrete data, but all results can be generalized from discrete to continuous data by replacing any sum over probability mass functions with integrals over probability density functions.

Original: We ﬁrst derive a general form of variance smoothing before discussing our certiﬁcates for binary data in Section B.2.1 and Section B.2.2.
Revised: We ﬁrst derive a general form of variance smoothing before discussing our certiﬁcates for binary data in Section B.2.1 and Section B.2.2.

Original: Variance smoothing assumes that we make predictions by randomly smoothing a base model’s softmax scores.
Revised: Variance smoothing assumes that we make predictions by randomly smoothing a base model’s softmax scores.

Original: That is, given base model g : X → ∆ | Y | mapping from an arbitrary discrete input space X to scores from the | Y | -dimensional probability simplex ∆ | Y | , we deﬁne the smoothed classiﬁer f ( x ) = argmax y ∈ Y E
Revised: That is, given base model g : X → ∆ | Y | mapping from an arbitrary discrete input space X to scores from the | Y | -dimensional probability simplex ∆ | Y | , we deﬁne the smoothed classiﬁer f ( x ) = argmax y ∈ Y E

Original: z ∼ Ψ
Revised: z ∼ Ψ

Original: ( x ) [ g ( z ) y ] .
Revised: ( x ) [ g ( z ) y ] .

Original: Here, Ψ( x ) is an arbitrary distribution over X parameterized by x , e.g a Normal distribution with mean x .
Revised: Here, Ψ( x ) is an arbitrary distribution over X parameterized by x , e.g a Normal distribution with mean x .

Original: The smoothed classiﬁer does not return the most likely prediction, but the prediction associated with the highest expected softmax score.
Revised: The smoothed classiﬁer does not return the most likely prediction, but the prediction associated with the highest expected softmax score.

Original: Given an input x ∈ X , smoothed prediction y =
Revised: Given an input x ∈ X , smoothed prediction y =

Original: f ( x ) and a perturbed input x (cid:48) ∈ X , we want to determine whether f ( x (cid:48) )
Revised: f ( x ) and a perturbed input x (cid:48) ∈ X , we want to determine whether f ( x (cid:48) )

Original: =
Revised: =

Original: y .
Revised: y .

Original: By deﬁnition of our smoothed classiﬁer, we know that f ( x (cid:48) )
Revised: By deﬁnition of our smoothed classiﬁer, we know that f ( x (cid:48) )

Original: =
Revised: =

Original: y if y is the label with the highest expected softmax score.
Revised: y if y is the label with the highest expected softmax score.

Original: In particular, we know that f ( x (cid:48) )
Revised: In particular, we know that f ( x (cid:48) )

Original: =
Revised: =

Original: y if y ’s softmax score is larger than all other softmax scores combined,
Revised: y if y ’s softmax score is larger than all other softmax scores combined,

Original: i.e.
Revised: i.e.

Original: [Equation]
Revised: [Equation]

Original: Computing E z ∼ Ψ
Revised: Computing E z ∼ Ψ

Original: ( x (cid:48) )
Revised: ( x (cid:48) )

Original: [ g ( z ) y ] exactly is usually not tractable – especially if we later want to evaluate robustness to many x (cid:48) from a whole perturbation model B ⊆ X .
Revised: [ g ( z ) y ] exactly is usually not tractable – especially if we later want to evaluate robustness to many x (cid:48) from a whole perturbation model B ⊆ X .

Original: Therefore, we compute a lower bound on E z ∼ Ψ
Revised: Therefore, we compute a lower bound on E z ∼ Ψ

Original: ( x (cid:48) )
Revised: ( x (cid:48) )

Original: [ g ( z ) y ] .
Revised: [ g ( z ) y ] .

Original: If even this lower bound is larger than 0 .
Revised: If even this lower bound is larger than 0 .

Original: 5 , we know that prediction y is certainly robust.
Revised: 5 , we know that prediction y is certainly robust.

Original: For this, we deﬁne a set of functions H with g y ∈ H and compute the minimum softmax score across all functions from H :
Revised: For this, we deﬁne a set of functions H with g y ∈ H and compute the minimum softmax score across all functions from H :

Original: [Equation]
Revised: [Equation]

Original: For our variance smoothing approach, we deﬁne H to be the set of all functions that have a larger or equal expected value and a smaller or equal variance, compared to our base model g applied to unperturbed input x .
Revised: For our variance smoothing approach, we deﬁne H to be the set of all functions that have a larger or equal expected value and a smaller or equal variance, compared to our base model g applied to unperturbed input x .

Original: Let µ = E
Revised: Let µ = E

Original: z ∼ Ψ
Revised: z ∼ Ψ

Original: ( x ) [ g ( z ) y ] be the expected softmax score of our base model g for label y .
Revised: ( x ) [ g ( z ) y ] be the expected softmax score of our base model g for label y .

Original: Let σ 2 =
Revised: Let σ 2 =

Original: E z ∼ Ψ
Revised: E z ∼ Ψ

Original: ( x ) (cid:104) ( g ( z ) y − ν ) 2 (cid:105) be the expected squared distance of the softmax score from a scalar ν ∈ R .
Revised: ( x ) (cid:104) ( g ( z ) y − ν ) 2 (cid:105) be the expected squared distance of the softmax score from a scalar ν ∈ R .

Original: (Choosing ν = µ yields the variance of the softmax score.
Revised: (Choosing ν = µ yields the variance of the softmax score.

Original: An arbitrary ν is only needed for technical reasons related to Monte Carlo estimation Section C.2).
Revised: An arbitrary ν is only needed for technical reasons related to Monte Carlo estimation Section C.2).

Original: Then, we deﬁne
Revised: Then, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Clearly, by the deﬁnition of µ and σ 2 , we have g y ∈ H .
Revised: Clearly, by the deﬁnition of µ and σ 2 , we have g y ∈ H .

Original: Note that we do not restrict functions from H to the domain [0 , 1] , but allow arbitrary real-valued outputs.
Revised: Note that we do not restrict functions from H to the domain [0 , 1] , but allow arbitrary real-valued outputs.

Original: By evaluating Eq.
Revised: By evaluating Eq.

Original: with H deﬁned as in Eq.
Revised: with H deﬁned as in Eq.

Original: 29, we can determine if our prediciton is robust.
Revised: 29, we can determine if our prediciton is robust.

Original: To compute the optimal value , we need the following two Lemmata:
Revised: To compute the optimal value , we need the following two Lemmata:

Original: Lemma 1.
Revised: Lemma 1.

Original: Given a discrete set X and the set Π of all probability mass functions over X , any two probability mass functions π 1 , π 2 ∈ Π fulﬁll
Revised: Given a discrete set X and the set Π of all probability mass functions over X , any two probability mass functions π 1 , π 2 ∈ Π fulﬁll

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For a ﬁxed probability mass function π 1 , Eq. 31 is lower-bounded by the minimal expected likelihood ratio that can be achieved by another ˜ π ( z ) ∈ Π :
Revised: For a ﬁxed probability mass function π 1 , Eq. 31 is lower-bounded by the minimal expected likelihood ratio that can be achieved by another ˜ π ( z ) ∈ Π :

Original: [Equation]
Revised: [Equation]

Original: The r.h.s.
Revised: The r.h.s.

Original: term can be expressed as the constrained optimization problem
Revised: term can be expressed as the constrained optimization problem

Original: [Equation]
Revised: [Equation]

Original: with the corresponding dual problem
Revised: with the corresponding dual problem

Original: [Equation]
Revised: [Equation]

Original: The inner problem is convex in each ˜ π
Revised: The inner problem is convex in each ˜ π

Original: ( z ) .
Revised: ( z ) .

Original: Taking the gradient w.r.t.
Revised: Taking the gradient w.r.t.

Original: to ˜ π
Revised: to ˜ π

Original: ( z ) for all z ∈ X shows that it has its minimum at ∀ z ∈ X : ˜ π
Revised: ( z ) for all z ∈ X shows that it has its minimum at ∀ z ∈ X : ˜ π

Original: ( z ) = − λπ 1 ( z ).
Revised: ( z ) = − λπ 1 ( z ).

Original: Substituting into Eq.
Revised: Substituting into Eq.

Original: 34 results in
Revised: 34 results in

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 37 follows from the fact that π 1 ( z ) is a valid probability mass function.
Revised: 37 follows from the fact that π 1 ( z ) is a valid probability mass function.

Original: Due to duality, the optimal dual value 1 is a lower bound on the optimal value of our primal problem Eq. 31.
Revised: Due to duality, the optimal dual value 1 is a lower bound on the optimal value of our primal problem Eq. 31.

Original: Lemma 2.
Revised: Lemma 2.

Original: Given a probability distribution D over a R and a scalar ν ∈ R , let µ = E z ∼D and ξ =
Revised: Given a probability distribution D over a R and a scalar ν ∈ R , let µ = E z ∼D and ξ =

Original: E z ∼D (cid:104) ( z − ν ) 2 (cid:105) .
Revised: E z ∼D (cid:104) ( z − ν ) 2 (cid:105) .

Original: Then ξ ≥ ( µ − ν )
Revised: Then ξ ≥ ( µ − ν )

Original: Proof.
Revised: Proof.

Original: Using the deﬁnitions of µ and ξ , as well as some simple algebra, we can show:
Revised: Using the deﬁnitions of µ and ξ , as well as some simple algebra, we can show:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: It is well known for the variance that E z ∼D (cid:2) ( z − µ ) 2 (cid:3) =
Revised: It is well known for the variance that E z ∼D (cid:2) ( z − µ ) 2 (cid:3) =

Original: E z ∼D (cid:2) z 2 (cid:3) − µ 2 .
Revised: E z ∼D (cid:2) z 2 (cid:3) − µ 2 .

Original: Because the variance is always non-negative, the above inequality holds.
Revised: Because the variance is always non-negative, the above inequality holds.

Original: Using the previously described approach and lemmata, we can show the soundness of the following robustness certiﬁcate:
Revised: Using the previously described approach and lemmata, we can show the soundness of the following robustness certiﬁcate:

Original: Theorem 3.
Revised: Theorem 3.

Original: Given a model g : X → ∆ | Y | mapping from discrete set X to scores from the | Y | -dimensional probability simplex, let f ( x ) = argmax y ∈ Y E
Revised: Given a model g : X → ∆ | Y | mapping from discrete set X to scores from the | Y | -dimensional probability simplex, let f ( x ) = argmax y ∈ Y E

Original: z ∼ Ψ
Revised: z ∼ Ψ

Original: ( x ) [ g ( z ) y ] be the corresponding smoothed classiﬁer with smoothing distribution Ψ( x ) and probability mass function π x ( z ) =
Revised: ( x ) [ g ( z ) y ] be the corresponding smoothed classiﬁer with smoothing distribution Ψ( x ) and probability mass function π x ( z ) =

Original: Pr ˜ z ∼ Ψ( x )
Revised: Pr ˜ z ∼ Ψ( x )

Original: [˜ z = z ] .
Revised: [˜ z = z ] .

Original: Given an input x ∈ X and smoothed prediction y =
Revised: Given an input x ∈ X and smoothed prediction y =

Original: f ( x ) , let µ = E
Revised: f ( x ) , let µ = E

Original: z ∼ Ψ
Revised: z ∼ Ψ

Original: ( x ) [ g ( z ) y ] and σ 

Revised: ( x ) [ g ( z ) y ] and σ

Original: = E z ∼ Ψ
Revised: = E z ∼ Ψ

Original: ( x ) (cid:104) ( g ( z ) y − ν ) 2 (cid:105) with ν ∈ R .
Revised: ( x ) (cid:104) ( g ( z ) y − ν ) 2 (cid:105) with ν ∈ R .

Original: If ν ≤ µ , we know that f ( x (cid:48) )
Revised: If ν ≤ µ , we know that f ( x (cid:48) )

Original: =
Revised: =

Original: y
Revised: y

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Following our discussion above, we know that f ( x (cid:48) )
Revised: Following our discussion above, we know that f ( x (cid:48) )

Original: =
Revised: =

Original: y if E z ∼ Ψ( x (cid:48) )
Revised: y if E z ∼ Ψ( x (cid:48) )

Original: [ g ( z ) y ] > 0 .
Revised: [ g ( z ) y ] > 0 .

Original: 5 with H deﬁned as in Section 5.
Revised: 5 with H deﬁned as in Section 5.

Original: We can compute a (tight) lower bound on min h ∈
Revised: We can compute a (tight) lower bound on min h ∈

Original: H E z ∼ Ψ
Revised: H E z ∼ Ψ

Original: ( x (cid:48) ) by following the functional optimization approach for randomized smoothing proposed by Zhang et al.
Revised: ( x (cid:48) ) by following the functional optimization approach for randomized smoothing proposed by Zhang et al.

Original: That is, we solve a dual problem in which we optimize the value
Revised: That is, we solve a dual problem in which we optimize the value

Original: h
Revised: h

Original: ( z ) for each z ∈ X .
Revised: ( z ) for each z ∈ X .

Original: By the deﬁnition of the set H , our optimization problem is
Revised: By the deﬁnition of the set H , our optimization problem is

Original: [Equation]
Revised: [Equation]

Original: The corresponding dual problem with dual variables α, β ≥ 0 is
Revised: The corresponding dual problem with dual variables α, β ≥ 0 is

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst move move all terms that don’t involve h out of the inner optimization problem:
Revised: We ﬁrst move move all terms that don’t involve h out of the inner optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Writing out the expectation terms and combining them into one sum (or – in the case of continuous X – one integral), our dual problem becomes
Revised: Writing out the expectation terms and combining them into one sum (or – in the case of continuous X – one integral), our dual problem becomes

Original: [Equation]
Revised: [Equation]

Original: (recall that π x (cid:48) and π x (cid:48) refer to the probability mass functions of the smoothing distributions).
Revised: (recall that π x (cid:48) and π x (cid:48) refer to the probability mass functions of the smoothing distributions).

Original: The inner optimization problem can be solved by ﬁnding the optimal h ( z ) in each point z :
Revised: The inner optimization problem can be solved by ﬁnding the optimal h ( z ) in each point z :

Original: [Equation]
Revised: [Equation]

Original: Because β ≥ 0 , each inner optimization problem is convex in h ( z ) .
Revised: Because β ≥ 0 , each inner optimization problem is convex in h ( z ) .

Original: We can thus ﬁnd the optimal h ∗ ( z ) by setting the derivative to zero:
Revised: We can thus ﬁnd the optimal h ∗ ( z ) by setting the derivative to zero:

Original: [Equation]
Revised: [Equation]

Original: Substituting into Eq.
Revised: Substituting into Eq.

Original: 48 and simplifying leaves us with the dual problem
Revised: 48 and simplifying leaves us with the dual problem

Original: [Equation]
Revised: [Equation]

Original: In the following, let us use ρ = (cid:80)
Revised: In the following, let us use ρ = (cid:80)

Original: z ∈
Revised: z ∈

Original: X π x (cid:48) ( z ) 2 π x ( z ) as a shorthand for the expected likelihood ratio.
Revised: X π x (cid:48) ( z ) 2 π x ( z ) as a shorthand for the expected likelihood ratio.

Original: The problem is concave in α .
Revised: The problem is concave in α .

Original: We can thus ﬁnd the optimum α ∗ by setting the derivative to zero, which gives us α ∗ = 2 β ( µ − ν ) + 1 .
Revised: We can thus ﬁnd the optimum α ∗ by setting the derivative to zero, which gives us α ∗ = 2 β ( µ − ν ) + 1 .

Original: Because β ≥ 0 and ou theorem assumes that ν ≤ µ , α ∗ is a feasible solution to the dual problem.
Revised: Because β ≥ 0 and ou theorem assumes that ν ≤ µ , α ∗ is a feasible solution to the dual problem.

Original: Substituting into Eq.
Revised: Substituting into Eq.

Original: and simplifying results in
Revised: and simplifying results in

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma 1 shows that the expected likelihood ratio ρ is always greater than or equal to 1 .
Revised: Lemma 1 shows that the expected likelihood ratio ρ is always greater than or equal to 1 .

Original: Lemmashows that ( µ − ν ) 2 − σ 2 ≤ 0 .
Revised: Lemmashows that ( µ − ν ) 2 − σ 2 ≤ 0 .

Original: Therefore Eq.
Revised: Therefore Eq.

Original: 55 is concave in β .
Revised: 55 is concave in β .

Original: The optimal value of β can again be found by setting the derivative to zero:
Revised: The optimal value of β can again be found by setting the derivative to zero:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Recall that our theorem assumes σ 2 ≥ ( µ − ν ) 2 and thus β ∗ is real valued.
Revised: Recall that our theorem assumes σ 2 ≥ ( µ − ν ) 2 and thus β ∗ is real valued.

Original: Substituting into Eq.shows that the maximum of our dual problem is
Revised: Substituting into Eq.shows that the maximum of our dual problem is

Original: [Equation]
Revised: [Equation]

Original: By duality, this is a lower bound on our primal problem min h ∈ H
Revised: By duality, this is a lower bound on our primal problem min h ∈ H

Original: E z ∼ Ψ
Revised: E z ∼ Ψ

Original: ( x
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: [ h ( z )] .
Revised: [ h ( z )] .

Original: We know that our prediction is certiﬁably robust, i.e. f ( x ) =
Revised: We know that our prediction is certiﬁably robust, i.e. f ( x ) =

Original: y , if min
Revised: y , if min

Original: [Equation]
Revised: [Equation]

Original: ∈ H E
Revised: ∈ H E

Original: z ∼ Ψ
Revised: z ∼ Ψ

Original: ( x (cid:48) )
Revised: ( x (cid:48) )

Original: [ h ( z )] > 0 . 5 .
Revised: [ h ( z )] > 0 . 5 .

Original: So, in particular, our prediction is robust if
Revised: So, in particular, our prediction is robust if

Original: [Equation]
Revised: [Equation]

Original: The last equivalence is the result of inserting the deﬁnition of the expected likelihood ratio ρ .
Revised: The last equivalence is the result of inserting the deﬁnition of the expected likelihood ratio ρ .

Original: With Theorem 3 in place, we can certify robustness for arbitrary smoothing distributions, assuming we can compute the expected likelihood ratio.
Revised: With Theorem 3 in place, we can certify robustness for arbitrary smoothing distributions, assuming we can compute the expected likelihood ratio.

Original: When we are working with discrete data and the smoothing distributions factorize (but are not necessarily i.i.d.), this can be done efﬁciently, as the two following base certiﬁcates for binary data demonstrate.
Revised: When we are working with discrete data and the smoothing distributions factorize (but are not necessarily i.i.d.), this can be done efﬁciently, as the two following base certiﬁcates for binary data demonstrate.

Original: B.2. B ERNOULLI V ARIANCE SMOOTHING FOR PERTURBATIONS OF BINARY DATA
Revised: B.2. B ERNOULLI V ARIANCE SMOOTHING FOR PERTURBATIONS OF BINARY DATA

Original: We begin by proving the base certiﬁcate presented in Section 5.
Revised: We begin by proving the base certiﬁcate presented in Section 5.

Original: Recall that we we use a smoothing distribution F ( x , θ ) with θ ∈ [0 , 1] D in that independently ﬂips the d ’th bit with probability θ d , i.e. for x , z ∈ { 0 , 1 } D in and z ∼ F ( x , θ ) we have Pr[ z d (cid:54) =
Revised: Recall that we we use a smoothing distribution F ( x , θ ) with θ ∈ [0 , 1] D in that independently ﬂips the d ’th bit with probability θ d , i.e. for x , z ∈ { 0 , 1 } D in and z ∼ F ( x , θ ) we have Pr[ z d (cid:54) =

Original: x d ] = θ d .
Revised: x d ] = θ d .

Original: Theorem 1.
Revised: Theorem 1.

Original: Given an output g n : { 0 , 1 } D in → ∆ | Y | mapping to scores from the | Y | -dimensional probability simplex, let f n ( x ) = argmax
Revised: Given an output g n : { 0 , 1 } D in → ∆ | Y | mapping to scores from the | Y | -dimensional probability simplex, let f n ( x ) = argmax

Original: y ∈ Y E
Revised: y ∈ Y E

Original: z ∼F ( x , θ ) [ g n ( z ) y ] be the corresponding smoothed classiﬁer with θ ∈ [0 , 1] D in .
Revised: z ∼F ( x , θ ) [ g n ( z ) y ] be the corresponding smoothed classiﬁer with θ ∈ [0 , 1] D in .

Original: Given an input x ∈ { 0 , 1 } D in and smoothed prediction y n = f n ( x ) , let µ = E z ∼F ( x , θ )
Revised: Given an input x ∈ { 0 , 1 } D in and smoothed prediction y n = f n ( x ) , let µ = E z ∼F ( x , θ )

Original: [ g n ( z ) y ] and σ 2 =
Revised: [ g n ( z ) y ] and σ 2 =

Original: Var z ∼F ( x , θ ) [ g n ( z ) y ] .
Revised: Var z ∼F ( x , θ ) [ g n ( z ) y ] .

Original: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =
Revised: Then, ∀ x (cid:48) ∈ H ( n ) : f n ( x (cid:48) ) =

Original: y n with H ( n ) deﬁned as in Eq. 2, w d = ln (cid:16) (
Revised: y n with H ( n ) deﬁned as in Eq. 2, w d = ln (cid:16) (

Original: [Equation]
Revised: [Equation]

Original: 2 θ
Revised: 2 θ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Based on our deﬁnition of the base certiﬁcate interface from Section 5, we must show that ∀ x (cid:48) ∈ H : f n ( x (cid:48) )
Revised: Based on our deﬁnition of the base certiﬁcate interface from Section 5, we must show that ∀ x (cid:48) ∈ H : f n ( x (cid:48) )

Original: =
Revised: =

Original: y n with
Revised: y n with

Original: H =
Revised: H =

Original: [Equation]
Revised: [Equation]

Original: Because all bits are ﬂipped independently, our probability mass function π x ( z ) =
Revised: Because all bits are ﬂipped independently, our probability mass function π x ( z ) =

Original: Pr ˜ z ∼ Ψ( x )
Revised: Pr ˜ z ∼ Ψ( x )

Original: [˜ z = z ] factorizes:
Revised: [˜ z = z ] factorizes:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: Thus, our expected likelihood ratio can be written as
Revised: Thus, our expected likelihood ratio can be written as

Original: [Equation]
Revised: [Equation]

Original: For each dimension d , we can distinguish two cases: If both the perturbed and unperturbed input are the same in dimension d , i.e. x (cid:48)
Revised: For each dimension d , we can distinguish two cases: If both the perturbed and unperturbed input are the same in dimension d , i.e. x (cid:48)

Original: d = x d , then π x (cid:48)
Revised: d = x d , then π x (cid:48)

Original: d ( z ) π xd ( z ) =
Revised: d ( z ) π xd ( z ) =

Original: and thus
Revised: and thus

Original: [Equation]
Revised: [Equation]

Original: If the perturbed and unperturbed input differ in dimension d , then
Revised: If the perturbed and unperturbed input differ in dimension d , then

Original: [Equation]
Revised: [Equation]

Original: Therefore, the expected likelihood ratio is
Revised: Therefore, the expected likelihood ratio is

Original: [Equation]
Revised: [Equation]

Original: Due to Theorem 3 (and using ν = µ when computing the variance)
Revised: Due to Theorem 3 (and using ν = µ when computing the variance)

Original: [Equation]
Revised: [Equation]

Original: we know that our prediction is robust
Revised: we know that our prediction is robust

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e
Revised: e

Original: [Equation]
Revised: [Equation]

Original: f n ( x (cid:48) )
Revised: f n ( x (cid:48) )

Original: =
Revised: =

Original: y n , if
Revised: y n , if

Original: [Equation]
Revised: [Equation]

Original: Because x d and x (cid:48)
Revised: Because x d and x (cid:48)

Original: d are binary, the last inequality is equivalent to
Revised: d are binary, the last inequality is equivalent to

Original: [Equation]
Revised: [Equation]

Original: B.2. S PARSITY - AWARE V ARIANCE SMOOTHING FOR PERTURBATIONS OF BINARY DATA
Revised: B.2. S PARSITY - AWARE V ARIANCE SMOOTHING FOR PERTURBATIONS OF BINARY DATA

Original: Sparsity-aware randomized smoothing (Bojchevski et al., 2020) is an alternative smoothing approach for binary data.
Revised: Sparsity-aware randomized smoothing (Bojchevski et al., 2020) is an alternative smoothing approach for binary data.

Original: It uses different probabilities for randomly deleting ( 1 → 0 ) and adding ( 0 → 1 ) bits to preserve data sparsity.
Revised: It uses different probabilities for randomly deleting ( 1 → 0 ) and adding ( 0 → 1 ) bits to preserve data sparsity.

Original: For a random variable z distributed according to the sparsity-aware distribution S ( x , θ + , θ − ) with x ∈ { 0 , 1 } D in and addition and deletion probabilities θ + , θ − ∈ [0 , 1] D in , we have:
Revised: For a random variable z distributed according to the sparsity-aware distribution S ( x , θ + , θ − ) with x ∈ { 0 , 1 } D in and addition and deletion probabilities θ + , θ − ∈ [0 , 1] D in , we have:

Original: [Equation]
Revised: [Equation]

Original: The Bernoulli smoothing distribution we discussed in the previous section is a special case of sparsity-aware smoothing with θ + = θ − .
Revised: The Bernoulli smoothing distribution we discussed in the previous section is a special case of sparsity-aware smoothing with θ + = θ − .

Original: The runtime of the robustness certiﬁcate derived by
Revised: The runtime of the robustness certiﬁcate derived by

Original: Bojchevski et al.
Revised: Bojchevski et al.

Original: (2020) increases exponentially with the number of unique values in θ
Revised: (2020) increases exponentially with the number of unique values in θ

Original: + and θ − , which makes it unsuitable for localized smoothing.
Revised: + and θ − , which makes it unsuitable for localized smoothing.

Original: Variance smoothing, on the other hand, allows us to efﬁciently compute a certiﬁcate in closed form.
Revised: Variance smoothing, on the other hand, allows us to efﬁciently compute a certiﬁcate in closed form.

Original: Theorem 2.
Revised: Theorem 2.

Original: Given an output g n : R D in → ∆ | Y | mapping to scores from the | Y | -dimensional probability simplex, let f n ( x ) =
Revised: Given an output g n : R D in → ∆ | Y | mapping to scores from the | Y | -dimensional probability simplex, let f n ( x ) =

Original: argmax
Revised: argmax

Original: y ∈ Y E z ∼S ( x , θ + , θ − )
Revised: y ∈ Y E z ∼S ( x , θ + , θ − )

Original: [ g n ( z ) y ] be the corresponding smoothed classiﬁer with θ + , θ − ∈ [0 , 1] D in .
Revised: [ g n ( z ) y ] be the corresponding smoothed classiﬁer with θ + , θ − ∈ [0 , 1] D in .

Original: Given an input x ∈ { 0 , 1 } D in and smoothed prediction y n = f n ( x ) , let µ = E z ∼S ( x , θ + , θ − )
Revised: Given an input x ∈ { 0 , 1 } D in and smoothed prediction y n = f n ( x ) , let µ = E z ∼S ( x , θ + , θ − )

Original: [ g n ( z ) y ] and σ 2 =
Revised: [ g n ( z ) y ] and σ 2 =

Original: Var z ∼S ( x , θ + , θ − )
Revised: Var z ∼S ( x , θ + , θ − )

Original: [ g n ( z ) y ] .
Revised: [ g n ( z ) y ] .

Original: Then, ∀ x (cid:48) ∈ H : f n ( x (cid:48) )
Revised: Then, ∀ x (cid:48) ∈ H : f n ( x (cid:48) )

Original: =
Revised: =

Original: y for
Revised: y for

Original: [Equation]
Revised: [Equation]

Original: where γ + , γ − ∈ R D in , γ + d = ln (cid:18) ( θ − d ) 2 1 − θ + d + (
Revised: where γ + , γ − ∈ R D in , γ + d = ln (cid:18) ( θ − d ) 2 1 − θ + d + (

Original: [Equation]
Revised: [Equation]

Original: − d
Revised: − d

Original: [Equation]
Revised: [Equation]

Original: + d
Revised: + d

Original: [Equation]
Revised: [Equation]

Original: , γ
Revised: , γ

Original: [Equation]
Revised: [Equation]

Original: d = ln (cid:18) ( 1 − θ + d ) 2 θ − d
Revised: d = ln (cid:18) ( 1 − θ + d ) 2 θ − d

Original: + (
Revised: + (

Original: [Equation]
Revised: [Equation]

Original: )1 − θ − d .
Revised: )1 − θ − d .

Original: (cid:19) and η = ln (cid:16) 1 + 1 σ 2 (cid:0) µ − 1 2 (cid:1) 2 (cid:17) .
Revised: (cid:19) and η = ln (cid:16) 1 + 1 σ 2 (cid:0) µ − 1 2 (cid:1) 2 (cid:17) .

Original: Proof.
Revised: Proof.

Original: Just like with the Bernoulli distribution we discussed in the previous section, all bits are ﬂipped independently, meaning our probability mass function
Revised: Just like with the Bernoulli distribution we discussed in the previous section, all bits are ﬂipped independently, meaning our probability mass function

Original: π x ( z ) =
Revised: π x ( z ) =

Original: Pr ˜ z ∼ Ψ
Revised: Pr ˜ z ∼ Ψ

Original: [Equation]
Revised: [Equation]

Original: x ) [˜ z = z ] factorizes:
Revised: x ) [˜ z = z ] factorizes:

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: As before, our expected likelihood ratio can be written as
Revised: As before, our expected likelihood ratio can be written as

Original: [Equation]
Revised: [Equation]

Original: We can now distinguish three cases.
Revised: We can now distinguish three cases.

Original: If both the perturbed and unperturbed input are the same in dimension d , i.e. x (cid:48)
Revised: If both the perturbed and unperturbed input are the same in dimension d , i.e. x (cid:48)

Original: d = x d , then π x (cid:48)
Revised: d = x d , then π x (cid:48)

Original: d ( z ) π xd ( z ) =
Revised: d ( z ) π xd ( z ) =

Original: and thus
Revised: and thus

Original: [Equation]
Revised: [Equation]

Original: If x (cid:48) d = 1 and x d = 0 , i.e. a bit was added, then
Revised: If x (cid:48) d = 1 and x d = 0 , i.e. a bit was added, then

Original: [Equation]
Revised: [Equation]

Original: If x (cid:48) d = 0 and x d = 1 ,
Revised: If x (cid:48) d = 0 and x d = 1 ,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. a bit was deleted
Revised: e. a bit was deleted

Original: [Equation]
Revised: [Equation]

Original: then (cid:88)
Revised: then (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Therefore, the expected likelihood ratio is
Revised: Therefore, the expected likelihood ratio is

Original: [Equation]
Revised: [Equation]

Original: In the last equation
Revised: In the last equation

Original: [Equation]
Revised: [Equation]

Original: we have simple used the shorthands γ + d and γ − d , as deﬁned in Theorem 2 Due to Theorem 3 (and using ν = µ when computing the variance)
Revised: we have simple used the shorthands γ + d and γ − d , as deﬁned in Theorem 2 Due to Theorem 3 (and using ν = µ when computing the variance)

Original: [Equation]
Revised: [Equation]

Original: we know that our prediction is robust,
Revised: we know that our prediction is robust,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. f n ( x (cid:48) )
Revised: e. f n ( x (cid:48) )

Original: =
Revised: =

Original: y n , if
Revised: y n , if

Original: [Equation]
Revised: [Equation]

Original: ⇐⇒
Revised: ⇐⇒

Original: [Equation]
Revised: [Equation]

Original: ⇐⇒
Revised: ⇐⇒

Original: [Equation]
Revised: [Equation]

Original: It should be noted that this certiﬁcate does not comply with our interface for base certiﬁcates Section 5, meaning we can not directly use it to certify robustness to norm-bound perturbations using our collective linear program from Section 6.
Revised: It should be noted that this certiﬁcate does not comply with our interface for base certiﬁcates Section 5, meaning we can not directly use it to certify robustness to norm-bound perturbations using our collective linear program from Section 6.

Original: We can however use it to certify collective robustness to the more reﬁned threat model used in (Schuchardt et al., 2021):
Revised: We can however use it to certify collective robustness to the more reﬁned threat model used in (Schuchardt et al., 2021):

Original: Let the set of admissible perturbed inputs be
Revised: Let the set of admissible perturbed inputs be

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: + , (cid:15) y ∈ N 0 specifying the number of bits the adversary is allowed to add or delete.
Revised: + , (cid:15) y ∈ N 0 specifying the number of bits the adversary is allowed to add or delete.

Original: We can now follow the procedure outlined in Section 6 to combine the per-prediction base certiﬁcates into a collective certiﬁcate for our new collective perturbation model.
Revised: We can now follow the procedure outlined in Section 6 to combine the per-prediction base certiﬁcates into a collective certiﬁcate for our new collective perturbation model.

Original: As discussed in, we can bound the number of predictions that are robust to simultaneous attacks by minimizing the number of predictions that are certiﬁably robust according to their base certiﬁcates:
Revised: As discussed in, we can bound the number of predictions that are robust to simultaneous attacks by minimizing the number of predictions that are certiﬁably robust according to their base certiﬁcates:

Original: [Equation]
Revised: [Equation]

Original: Inserting the linear inequalities characterizing our perturbation model and base certiﬁcates results in:
Revised: Inserting the linear inequalities characterizing our perturbation model and base certiﬁcates results in:

Original: [Equation]
Revised: [Equation]

Original: Instead of optimizing over the perturbed input x (cid:48)
Revised: Instead of optimizing over the perturbed input x (cid:48)

Original: [Equation]
Revised: [Equation]

Original: we can deﬁne two vectors b + , b − ∈ { 0 , 1 } D in that indicate in which dimension bits were added or deleted.
Revised: we can deﬁne two vectors b + , b − ∈ { 0 , 1 } D in that indicate in which dimension bits were added or deleted.

Original: Using these new variables
Revised: Using these new variables

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 86 can be rewritten as
Revised: 86 can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: The last two constraints ensure that bits can only be deleted where x d = 1 and bits can only be added where x d = 0 .
Revised: The last two constraints ensure that bits can only be deleted where x d = 1 and bits can only be added where x d = 0 .

Original: Finally, we can use the procedure for replacing the indicator functions with indicator variables that we discussed in Section 6).
Revised: Finally, we can use the procedure for replacing the indicator functions with indicator variables that we discussed in Section 6).

Original: Let η ( n ) be the minimum values the weighted sums in the objective function can take on:
Revised: Let η ( n ) be the minimum values the weighted sums in the objective function can take on:

Original: [Equation]
Revised: [Equation]

Original: The above problem can be solved by ﬁnding all negative entries of γ + and γ − and then summing up the min (cid:0) sum { 1 − x } , b + d (cid:1) (or min (cid:0) sum { x } , b − d (cid:1) , respectively) smallest ones.
Revised: The above problem can be solved by ﬁnding all negative entries of γ + and γ − and then summing up the min (cid:0) sum { 1 − x } , b + d (cid:1) (or min (cid:0) sum { x } , b − d (cid:1) , respectively) smallest ones.

Original: Using these η ( n ) and binary indicator variables t ∈ { 0 , 1 } D out , we can restate the above problem as the mixed-integer problem
Revised: Using these η ( n ) and binary indicator variables t ∈ { 0 , 1 } D out , we can restate the above problem as the mixed-integer problem

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst constraint ensures that t n can only be set to 0 if the l.h.s.
Revised: The ﬁrst constraint ensures that t n can only be set to 0 if the l.h.s.

Original: is greater or equal η n , i.e. only when the base certiﬁcate can no longer guarantee robustness.
Revised: is greater or equal η n , i.e. only when the base certiﬁcate can no longer guarantee robustness.

Original: The efﬁciency of the certiﬁcate can be improved by applying any of the techniques discussed in Section A.
Revised: The efﬁciency of the certiﬁcate can be improved by applying any of the techniques discussed in Section A.

Original: C M ONTE C ARLO RANDOMIZED SMOOTHING
Revised: C M ONTE C ARLO RANDOMIZED SMOOTHING

Original: To make predictions and certify robustness, randomized smoothing requires computing certain properties of the distribution of a base model
Revised: To make predictions and certify robustness, randomized smoothing requires computing certain properties of the distribution of a base model

Original: ’s output, given an input smoothing distribution.
Revised: ’s output, given an input smoothing distribution.

Original: For example, the certiﬁcate of Cohen et al.
Revised: For example, the certiﬁcate of Cohen et al.

Original: (2019) assumes that the smoothed model f predicts the most likely label output by base model g , given a smoothing distribution N ( 0 , σ · 1 ) :
Revised: (2019) assumes that the smoothed model f predicts the most likely label output by base model g , given a smoothing distribution N ( 0 , σ · 1 ) :

Original: f ( x ) = argmax y ∈ Y Pr z ∼N ( 0 ,σ · 1 ) [ g ( x + z ) =
Revised: f ( x ) = argmax y ∈ Y Pr z ∼N ( 0 ,σ · 1 ) [ g ( x + z ) =

Original: y ] .
Revised: y ] .

Original: To certify the robustness of a smoothed prediction y
Revised: To certify the robustness of a smoothed prediction y

Original: = f
Revised: = f

Original: ( x ) for a speciﬁc input x, we have to compute the probability q =
Revised: ( x ) for a speciﬁc input x, we have to compute the probability q =

Original: Pr z ∼N ( 0 ,σ · 1 ) [ g ( x + z ) =
Revised: Pr z ∼N ( 0 ,σ · 1 ) [ g ( x + z ) =

Original: y ] to then calculate the maximum certiﬁable radius σ Φ − 1 ( q ) with standard-normal inverse CDF Φ − 1 .
Revised: y ] to then calculate the maximum certiﬁable radius σ Φ − 1 ( q ) with standard-normal inverse CDF Φ − 1 .

Original: For complicated models like deep neural networks, computing such properties in closed form is usually not tractable.
Revised: For complicated models like deep neural networks, computing such properties in closed form is usually not tractable.

Original: Instead, they have to be estimated using Monte Carlo sampling.
Revised: Instead, they have to be estimated using Monte Carlo sampling.

Original: The result are predictions and certiﬁcates that only hold with a certain probability.
Revised: The result are predictions and certiﬁcates that only hold with a certain probability.

Original: Randomized smoothing with Monte Carlo sampling usually consists of three distinct steps.
Revised: Randomized smoothing with Monte Carlo sampling usually consists of three distinct steps.

Original: First, a small number of samples N 1 from the smoothing distribution are used to generate a candidate prediction
Revised: First, a small number of samples N 1 from the smoothing distribution are used to generate a candidate prediction

Original: ˆ y , e.g. the most frequently predicted class.
Revised: ˆ y , e.g. the most frequently predicted class.

Original: Then, a second round of N 2 samples is taken and a statistical test is used to determine whether the candidate prediction is likely to be the actual prediction of smoothed classiﬁer f , i.e. whether ˆ y = f ( x ) with a certain probability (1 - α 1 ).
Revised: Then, a second round of N 2 samples is taken and a statistical test is used to determine whether the candidate prediction is likely to be the actual prediction of smoothed classiﬁer f , i.e. whether ˆ y = f ( x ) with a certain probability (1 - α 1 ).

Original: If this is not the case, one has to abstain from making a prediction (or generate a new candidate prediction).
Revised: If this is not the case, one has to abstain from making a prediction (or generate a new candidate prediction).

Original: To certify the robustness of prediction ˆ y , a ﬁnal round of N 3 samples is taken to estimate all quantities needed for the certiﬁcate.
Revised: To certify the robustness of prediction ˆ y , a ﬁnal round of N 3 samples is taken to estimate all quantities needed for the certiﬁcate.

Original: In the case of (Cohen et al., 2019), we need to estimate the probability q =
Revised: In the case of (Cohen et al., 2019), we need to estimate the probability q =

Original: Pr z ∼N ( 0 ,σ · 1 ) [ g ( x + z )
Revised: Pr z ∼N ( 0 ,σ · 1 ) [ g ( x + z )

Original: =
Revised: =

Original: ˆ y ] to compute the certiﬁcate σ Φ − 1 ( q ) , whose strength is monotonically increasing in q .
Revised: ˆ y ] to compute the certiﬁcate σ Φ − 1 ( q ) , whose strength is monotonically increasing in q .

Original: To ensure that the certiﬁcate holds with high probability (1 - α 2 ) , we have to compute a probabilistic lower bound q ≤ q .
Revised: To ensure that the certiﬁcate holds with high probability (1 - α 2 ) , we have to compute a probabilistic lower bound q ≤ q .

Original: If one wants to certify robustness for all predictions, as we do in our experiments, one can also use the same samples for the abstention test and the certiﬁcates.
Revised: If one wants to certify robustness for all predictions, as we do in our experiments, one can also use the same samples for the abstention test and the certiﬁcates.

Original: One particularly simple abstention mechanism is to just compute the Monte
Revised: One particularly simple abstention mechanism is to just compute the Monte

Original: Carlo randomized smoothing certiﬁcate to determine whether ∀ x (cid:48) ∈ { x } : f ( x (cid:48) )
Revised: Carlo randomized smoothing certiﬁcate to determine whether ∀ x (cid:48) ∈ { x } : f ( x (cid:48) )

Original: =
Revised: =

Original: ˆ y with high probability, i.e. whether the prediction is robust to input x (cid:48) that is the result of ”perturbing” clean input x with zero adversarial budget.
Revised: ˆ y with high probability, i.e. whether the prediction is robust to input x (cid:48) that is the result of ”perturbing” clean input x with zero adversarial budget.

Original: In the following, we discuss how we perform Monte Carlo randomized smoothing for our base certiﬁcates, as well as the baselines we use for our experimental evaluation.
Revised: In the following, we discuss how we perform Monte Carlo randomized smoothing for our base certiﬁcates, as well as the baselines we use for our experimental evaluation.

Original: In Section C.4, we discuss how we account for the multiple comparisons problem, i.e. the fact that we are not just trying to probabilistically certify a single prediction, but multiple predictions at once.
Revised: In Section C.4, we discuss how we account for the multiple comparisons problem, i.e. the fact that we are not just trying to probabilistically certify a single prediction, but multiple predictions at once.

Original: C. M ONTE C ARLO BASE CERTIFICATES FOR CONTINUOUS DATA
Revised: C. M ONTE C ARLO BASE CERTIFICATES FOR CONTINUOUS DATA

Original: For our base certiﬁcates for continuous data, we follow the approach we already discussed in the previous paragraphs (recall that the certiﬁcate of Cohen et al.
Revised: For our base certiﬁcates for continuous data, we follow the approach we already discussed in the previous paragraphs (recall that the certiﬁcate of Cohen et al.

Original: (2019) is a special case of our certiﬁcate with Gaussian noise for l 2 perturbations).
Revised: (2019) is a special case of our certiﬁcate with Gaussian noise for l 2 perturbations).

Original: We are given an input space X D in , label space Y , base model (or – in the case of multi-output classiﬁers – base model output) g : X D in → Y and smoothing distribution Ψ( x ) (either multivariate Gaussian or multivariate uniform).
Revised: We are given an input space X D in , label space Y , base model (or – in the case of multi-output classiﬁers – base model output) g : X D in → Y and smoothing distribution Ψ( x ) (either multivariate Gaussian or multivariate uniform).

Original: To generate a candidate prediction, we apply the base classiﬁer to N 1 samples from the smoothing distribution in order to obtain predictions (cid:0) y (1) , . . .
Revised: To generate a candidate prediction, we apply the base classiﬁer to N 1 samples from the smoothing distribution in order to obtain predictions (cid:0) y (1) , . . .

Original: y ( N 1 ) (cid:1) and compute the majority prediction
Revised: y ( N 1 ) (cid:1) and compute the majority prediction

Original: ˆ y = argmax y ∈ Y (cid:8) n | y ( n ) =
Revised: ˆ y = argmax y ∈ Y (cid:8) n | y ( n ) =

Original: ˆ y (cid:9) .
Revised: ˆ y (cid:9) .

Original: Recall that for Gaussian and uniform noise, our certiﬁcate guarantees ∀ x (cid:48) ∈ H : f ( x ) =
Revised: Recall that for Gaussian and uniform noise, our certiﬁcate guarantees ∀ x (cid:48) ∈ H : f ( x ) =

Original: ˆ y with
Revised: ˆ y with

Original: [Equation]
Revised: [Equation]

Original: with η = (cid:0)
Revised: with η = (cid:0)

Original: Φ − 1 ( q ) (cid:1) 2 or η = Φ − 1 ( q ) (depending on the distribution), q =
Revised: Φ − 1 ( q ) (cid:1) 2 or η = Φ − 1 ( q ) (depending on the distribution), q =

Original: Pr z ∼N ( 0 ,σ · 1 ) [ g ( x + z )
Revised: Pr z ∼N ( 0 ,σ · 1 ) [ g ( x + z )

Original: =
Revised: =

Original: ˆ y ] and standard-normal inverse CDF Φ − 1 .
Revised: ˆ y ] and standard-normal inverse CDF Φ − 1 .

Original: To obtain a probabilistic certiﬁcate that holds with high probability 1 − α , we need a probabilistic lower bound on η .
Revised: To obtain a probabilistic certiﬁcate that holds with high probability 1 − α , we need a probabilistic lower bound on η .

Original: Both η are monotonically increasing in q , i.e. we can bound them by ﬁnding a lower bound q on q .
Revised: Both η are monotonically increasing in q , i.e. we can bound them by ﬁnding a lower bound q on q .

Original: For this, we take N 2 more samples from the smoothing distribution and compute a Clopper-Pearson lower conﬁdence bound (Clopper & Pearson, 1934) on q .
Revised: For this, we take N 2 more samples from the smoothing distribution and compute a Clopper-Pearson lower conﬁdence bound (Clopper & Pearson, 1934) on q .

Original: For abstentions, we use the aforementioned simple mechanism: We test whether x ∈ H .
Revised: For abstentions, we use the aforementioned simple mechanism: We test whether x ∈ H .

Original: Given the deﬁnition of H , this is equivalent to testing whether
Revised: Given the deﬁnition of H , this is equivalent to testing whether

Original: [Equation]
Revised: [Equation]

Original: If q ≤ 0 .
Revised: If q ≤ 0 .

Original: 5 , we abstain.
Revised: 5 , we abstain.

Original: C. M ONTE C ARLO VARIANCE SMOOTHING
Revised: C. M ONTE C ARLO VARIANCE SMOOTHING

Original: In variance smoothing, we smooth a model’s softmax scores.
Revised: In variance smoothing, we smooth a model’s softmax scores.

Original: That is, we are given an input space X D in , label space Y , base model (or – in the case of multi-output classiﬁers – base model output) g : X D in → ∆ | Y | with | Y | -dimensional probability simplex ∆ | Y | and smoothing distribution Ψ( x ) (Bernoullli or sparsity-aware noise, in the case of binary data).
Revised: That is, we are given an input space X D in , label space Y , base model (or – in the case of multi-output classiﬁers – base model output) g : X D in → ∆ | Y | with | Y | -dimensional probability simplex ∆ | Y | and smoothing distribution Ψ( x ) (Bernoullli or sparsity-aware noise, in the case of binary data).

Original: To generate a candidate prediction, we apply the base classiﬁer to N 1 samples from the smoothing distribution in order to obtain vectors (cid:0) s (1) , . . .
Revised: To generate a candidate prediction, we apply the base classiﬁer to N 1 samples from the smoothing distribution in order to obtain vectors (cid:0) s (1) , . . .

Original: , s ( N 1 ) (cid:1) with s ∈ ∆ | Y | , compute the average softmax scores s = 1 N 1 (cid:80)
Revised: , s ( N 1 ) (cid:1) with s ∈ ∆ | Y | , compute the average softmax scores s = 1 N 1 (cid:80)

Original: Nn =1 s and select the label with the highest score ˆ y = arg max y s y .
Revised: Nn =1 s and select the label with the highest score ˆ y = arg max y s y .

Original: Recall that our certiﬁcate guarantees robustness if the optimal value of the following optimization problem is greater than 0 .
Revised: Recall that our certiﬁcate guarantees robustness if the optimal value of the following optimization problem is greater than 0 .

Original: [Equation]
Revised: [Equation]

Original: with µ = E z ∼ Ψ
Revised: with µ = E z ∼ Ψ

Original: ( x ) [ g ( z ) ˆ y ] , σ 2 =
Revised: ( x ) [ g ( z ) ˆ y ] , σ 2 =

Original: E z ∼ Ψ
Revised: E z ∼ Ψ

Original: ( x ) (cid:104) ( g ( z ) ˆ y − ν ) 2 (cid:105) and a ﬁxed scalar ν ∈ R .
Revised: ( x ) (cid:104) ( g ( z ) ˆ y − ν ) 2 (cid:105) and a ﬁxed scalar ν ∈ R .

Original: To obtain a probabilistic certiﬁcate, we have to compute a probabilistic lower bound on the optimal value of the optimization problem.
Revised: To obtain a probabilistic certiﬁcate, we have to compute a probabilistic lower bound on the optimal value of the optimization problem.

Original: Because it is a minimization problem, this can be achieved by loosening its constraints, i.e. computing a probabilistic lower bound µ on µ and a probabilistic upper bound σon σ 2 .
Revised: Because it is a minimization problem, this can be achieved by loosening its constraints, i.e. computing a probabilistic lower bound µ on µ and a probabilistic upper bound σon σ 2 .

Original: Like in CDF-smoothing (Kumar et al., 2020), we bound the parameters using CDF-based nonparametric conﬁdence intervals.
Revised: Like in CDF-smoothing (Kumar et al., 2020), we bound the parameters using CDF-based nonparametric conﬁdence intervals.

Original: Let F ( s ) =
Revised: Let F ( s ) =

Original: Pr z ∼ Ψ
Revised: Pr z ∼ Ψ

Original: ( x ) [ g ( z ) ˆ y ≤ s ] be the CDF of the distribution of g ˆ y under the smoothing distribution Ψ( x ) .
Revised: ( x ) [ g ( z ) ˆ y ≤ s ] be the CDF of the distribution of g ˆ y under the smoothing distribution Ψ( x ) .

Original: Deﬁne M thresholds ≤ 0 τ 1 ≤ τ 2 . .
Revised: Deﬁne M thresholds ≤ 0 τ 1 ≤ τ 2 . .

Original: , τ M − 1 ≤ τ M ≤with ∀ m : τ m ∈ [0 , 1] .
Revised: , τ M − 1 ≤ τ M ≤with ∀ m : τ m ∈ [0 , 1] .

Original: We then take N 2 samples x (1) , . . .
Revised: We then take N 2 samples x (1) , . . .

Original: , x ( N 2 ) from the smoothing distribution to compute the empirical CDF ˜ F ( s ) = (cid:80) N 2 n =1 I (cid:2) g ( z ( n ) )
Revised: , x ( N 2 ) from the smoothing distribution to compute the empirical CDF ˜ F ( s ) = (cid:80) N 2 n =1 I (cid:2) g ( z ( n ) )

Original: ˆ y ≤ s (cid:3) .
Revised: ˆ y ≤ s (cid:3) .

Original: We can then use the DvoretzkyKeifer-Wolfowitz inequality (Dvoretzky et al., 1956) to compute an upper bound ˆ F and a lower bound F on the CDF of g ˆ y :
Revised: We can then use the DvoretzkyKeifer-Wolfowitz inequality (Dvoretzky et al., 1956) to compute an upper bound ˆ F and a lower bound F on the CDF of g ˆ y :

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: with υ = (cid:113) ln2 /α 2 · N 2 , which holds with high probability (1 − α ) .
Revised: with υ = (cid:113) ln2 /α 2 · N 2 , which holds with high probability (1 − α ) .

Original: Using these bounds on the CDF, we can bound µ = E z ∼ Ψ
Revised: Using these bounds on the CDF, we can bound µ = E z ∼ Ψ

Original: ( x ) [ g ( z ) ˆ y ] as follows (Anderson, 1969):
Revised: ( x ) [ g ( z ) ˆ y ] as follows (Anderson, 1969):

Original: [Equation]
Revised: [Equation]

Original: The parameter σ 2 = E z ∼ Ψ
Revised: The parameter σ 2 = E z ∼ Ψ

Original: ( x ) (cid:104) ( g ( z ) ˆ y − ν ) 2 (cid:105) can be bounded in a similar fashion.
Revised: ( x ) (cid:104) ( g ( z ) ˆ y − ν ) 2 (cid:105) can be bounded in a similar fashion.

Original: Deﬁne ξ 0 , . .
Revised: Deﬁne ξ 0 , . .

Original: ξ M ∈ R + with:
Revised: ξ M ∈ R + with:

Original: [Equation]
Revised: [Equation]

Original: i.e. compute the maximum squared distance to ν within each bin [ τ m , τ m +1 ] .
Revised: i.e. compute the maximum squared distance to ν within each bin [ τ m , τ m +1 ] .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: with probability (1 − α ) .
Revised: with probability (1 − α ) .

Original: In the ﬁrst inequality, we bound the expected squared distance from ν by assuming that the probability mass in each bin [ τ m , τ m +1 ] is concentrated at the farthest point from ν .
Revised: In the ﬁrst inequality, we bound the expected squared distance from ν by assuming that the probability mass in each bin [ τ m , τ m +1 ] is concentrated at the farthest point from ν .

Original: The equality is a result of reordering the telescope sum.
Revised: The equality is a result of reordering the telescope sum.

Original: In the second inequality, we upper-bound the CDF where it is multiplied with a non-negative value and lower-bound it where it is multiplied with a negative value.
Revised: In the second inequality, we upper-bound the CDF where it is multiplied with a non-negative value and lower-bound it where it is multiplied with a negative value.

Original: With the probabilistic bounds µ and σ 2 we can now – in principle – evaluate our robustness certiﬁcate, i.e. check whether
Revised: With the probabilistic bounds µ and σ 2 we can now – in principle – evaluate our robustness certiﬁcate, i.e. check whether

Original: [Equation]
Revised: [Equation]

Original: where the π are the probability mass functions of smoothing distributions Ψ( x ) and Ψ( x (cid:48) ) .
Revised: where the π are the probability mass functions of smoothing distributions Ψ( x ) and Ψ( x (cid:48) ) .

Original: But one crucial detail of Theorem 3 underlying the certiﬁcate was that it only holds for ν ≤ µ , i.e. only when this condition is fulﬁlled can we compute the certiﬁcate in closed form by solving the corresponding dual problem.
Revised: But one crucial detail of Theorem 3 underlying the certiﬁcate was that it only holds for ν ≤ µ , i.e. only when this condition is fulﬁlled can we compute the certiﬁcate in closed form by solving the corresponding dual problem.

Original: To use the method with Monte Carlo sampling, one has to ensure that ν ≤ µ by ﬁrst computing µ and then choosing some smaller ν .
Revised: To use the method with Monte Carlo sampling, one has to ensure that ν ≤ µ by ﬁrst computing µ and then choosing some smaller ν .

Original: In our experiments, we use an alternative method that allows us to use arbitrary ν : From our proof of Theorem 3we know that the dual problem of Eq.
Revised: In our experiments, we use an alternative method that allows us to use arbitrary ν : From our proof of Theorem 3we know that the dual problem of Eq.

Original: 98 is
Revised: 98 is

Original: [Equation]
Revised: [Equation]

Original: Instead of trying to ﬁnd an optimal α (which causes problems in subsequent derivations if ν (cid:2) µ ), we can simply choose α = 1 .
Revised: Instead of trying to ﬁnd an optimal α (which causes problems in subsequent derivations if ν (cid:2) µ ), we can simply choose α = 1 .

Original: By duality, the result is still a lower bound on the primal problem, i.e. the certiﬁcate remains valid.
Revised: By duality, the result is still a lower bound on the primal problem, i.e. the certiﬁcate remains valid.

Original: The dual problem becomes
Revised: The dual problem becomes

Original: [Equation]
Revised: [Equation]

Original: The problem is concave in β (because the expected likelihood ratio is ≥ 1 ).
Revised: The problem is concave in β (because the expected likelihood ratio is ≥ 1 ).

Original: Finding the optimal β , comparing the result to 0 .
Revised: Finding the optimal β , comparing the result to 0 .

Original: 5 and solving for the expected likelihood ratio, shows that a prediction is robust if (cid:88)
Revised: 5 and solving for the expected likelihood ratio, shows that a prediction is robust if (cid:88)

Original: [Equation]
Revised: [Equation]

Original: For our abstention mechanism, like in the previous section, we compute the certiﬁcate H and then test whether x ∈ H .
Revised: For our abstention mechanism, like in the previous section, we compute the certiﬁcate H and then test whether x ∈ H .

Original: In the case of Bernoulli smoothing and sparsity-aware smoothing), this corresponds to testing whether
Revised: In the case of Bernoulli smoothing and sparsity-aware smoothing), this corresponds to testing whether

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: C. M ONTE C ARLO CENTER SMOOTHING
Revised: C. M ONTE C ARLO CENTER SMOOTHING

Original: While we can not use center smoothing as a base certiﬁcate, we benchmark our method against it during our experimental evaluation.
Revised: While we can not use center smoothing as a base certiﬁcate, we benchmark our method against it during our experimental evaluation.

Original: The generation of candidate predictions, the abstention mechanism and the certiﬁcate are explained in (Kumar & Goldstein, 2021).
Revised: The generation of candidate predictions, the abstention mechanism and the certiﬁcate are explained in (Kumar & Goldstein, 2021).

Original: The authors allow multiple options for generating candidate predictions.
Revised: The authors allow multiple options for generating candidate predictions.

Original: We use the ” β minimum enclosing ball” with β =that is based on pair-wise distance calculations.
Revised: We use the ” β minimum enclosing ball” with β =that is based on pair-wise distance calculations.

Original: C. M ULTIPLE COMPARISONS PROBLEM
Revised: C. M ULTIPLE COMPARISONS PROBLEM

Original: The ﬁrst step of our collective certiﬁcate is to compute one base certiﬁcate for each of the D out predictions of the multi-output classiﬁer.
Revised: The ﬁrst step of our collective certiﬁcate is to compute one base certiﬁcate for each of the D out predictions of the multi-output classiﬁer.

Original: With Monte Carlo randomized smoothing, we want all of these probabilistic certiﬁcates to simultaneously hold with a high probability (1 − α ) .
Revised: With Monte Carlo randomized smoothing, we want all of these probabilistic certiﬁcates to simultaneously hold with a high probability (1 − α ) .

Original: But as the number of certiﬁcates increases, so does the probability of at least one of them being invalid.
Revised: But as the number of certiﬁcates increases, so does the probability of at least one of them being invalid.

Original: To account for this multiple comparisons problem , we use Bonferroni (Bonferroni, 1936) correction, i.e. compute each Monte Carlo certiﬁcate such that it holds with probability (1 − αn ) .
Revised: To account for this multiple comparisons problem , we use Bonferroni (Bonferroni, 1936) correction, i.e. compute each Monte Carlo certiﬁcate such that it holds with probability (1 − αn ) .

Original: For base certiﬁcates that only depend on q n = Pr z ∼ Ψ ( n ) [ g n ( z ) =
Revised: For base certiﬁcates that only depend on q n = Pr z ∼ Ψ ( n ) [ g n ( z ) =

Original: ˆ y n ] , i.e. the probability of the base classiﬁer predicting a particular label ˆ y n under the smoothing distribution, one can also use the strictly better Holm correction (Holm, 1979).
Revised: ˆ y n ] , i.e. the probability of the base classiﬁer predicting a particular label ˆ y n under the smoothing distribution, one can also use the strictly better Holm correction (Holm, 1979).

Original: This includes our Gaussian and uniform smoothing certiﬁcates for continuous data.
Revised: This includes our Gaussian and uniform smoothing certiﬁcates for continuous data.

Original: Holm correction is a procedure than can be used to correct for the multiple comparisons problem when performing multiple arbitrary hypothesis tests.
Revised: Holm correction is a procedure than can be used to correct for the multiple comparisons problem when performing multiple arbitrary hypothesis tests.

Original: Given N
Revised: Given N

Original: hypotheses
Revised: hypotheses

Original: , their p -values are ordered in ascending order p 1 , . .
Revised: , their p -values are ordered in ascending order p 1 , . .

Original: . , p N .
Revised: . , p N .

Original: Starting at i = 1 , the i ’th hypothesis is rejected if p i <
Revised: Starting at i = 1 , the i ’th hypothesis is rejected if p i <

Original: α N +1 − i , until one reaches an i such that p i ≥
Revised: α N +1 − i , until one reaches an i such that p i ≥

Original: α N
Revised: α N

Original: +1 − i .
Revised: +1 − i .

Original: Fischer et al.
Revised: Fischer et al.

Original: (2021) proposed to use Holm correction as part of their procedure for certifying that all (non-abstaining) predictions of an image segmentation model are robust to adversarial perturbations.
Revised: (2021) proposed to use Holm correction as part of their procedure for certifying that all (non-abstaining) predictions of an image segmentation model are robust to adversarial perturbations.

Original: In the following, we ﬁrst summarize their approach and then discuss how Holm correction can be used for certifying our notion of collective robustness, i.e. certifying the number of robust predictions.
Revised: In the following, we ﬁrst summarize their approach and then discuss how Holm correction can be used for certifying our notion of collective robustness, i.e. certifying the number of robust predictions.

Original: As in Section C.1, the goal is to obtain a lower bound q n on q n =
Revised: As in Section C.1, the goal is to obtain a lower bound q n on q n =

Original: Pr z ∼ Ψ ( n ) [ g n ( z )
Revised: Pr z ∼ Ψ ( n ) [ g n ( z )

Original: =
Revised: =

Original: ˆ y n ] for each of the D out classiﬁer outputs.
Revised: ˆ y n ] for each of the D out classiﬁer outputs.

Original: Assume we take N 2 samples z (1) , . .
Revised: Assume we take N 2 samples z (1) , . .

Original: , z ( N 2 ) from the smoothing distribution.
Revised: , z ( N 2 ) from the smoothing distribution.

Original: Let ν n = (cid:80) N 2 i =1 I (cid:2)
Revised: Let ν n = (cid:80) N 2 i =1 I (cid:2)

Original: g n ( z ( i ) )
Revised: g n ( z ( i ) )

Original: =
Revised: =

Original: ˆ y n (cid:3) and let π : { 1 , . . .
Revised: ˆ y n (cid:3) and let π : { 1 , . . .

Original: , D out } → { 1 , . . .
Revised: , D out } → { 1 , . . .

Original: , D out } be a bijection that orders the ν n in descending order, i.e. ν π (1) ≥ ν π (2) · · · ≥ ν π ( D out ) .
Revised: , D out } be a bijection that orders the ν n in descending order, i.e. ν π (1) ≥ ν π (2) · · · ≥ ν π ( D out ) .

Original: Instead of using
Revised: Instead of using

Original: Clopper-Pearson conﬁdence intervals to obtain tight lower bounds on the q n , Fischer et al.
Revised: Clopper-Pearson conﬁdence intervals to obtain tight lower bounds on the q n , Fischer et al.

Original: (2021) deﬁne a threshold τ ∈ [0 . 5 , 1) and use Binomial tests to determine for which n the bound τ ≤ q n holds with high-probability.
Revised: (2021) deﬁne a threshold τ ∈ [0 . 5 , 1) and use Binomial tests to determine for which n the bound τ ≤ q n holds with high-probability.

Original: Let BinP ( ν n , N 2 , ≤ , τ ) be the p-value of the one-sided binomial test, which is monotonically decreasing in ν n .
Revised: Let BinP ( ν n , N 2 , ≤ , τ ) be the p-value of the one-sided binomial test, which is monotonically decreasing in ν n .

Original: Following the Holm correction scheme, the authors test whether
Revised: Following the Holm correction scheme, the authors test whether

Original: [Equation]
Revised: [Equation]

Original: for k = 1 , . . .
Revised: for k = 1 , . . .

Original: , D out until reaching a k ∗ for which the null-hypothesis can no longer be rejected, i.e. the p-value is g.e.q.
Revised: , D out until reaching a k ∗ for which the null-hypothesis can no longer be rejected, i.e. the p-value is g.e.q.

Original: α D out +1 − k ∗ .
Revised: α D out +1 − k ∗ .

Original: They then know that with probability 1 − α , the bound τ ≤ q n holds for all n ∈ { π ( k ) | k ∈ { 1 , . . .
Revised: They then know that with probability 1 − α , the bound τ ≤ q n holds for all n ∈ { π ( k ) | k ∈ { 1 , . . .

Original: , k ∗ } .
Revised: , k ∗ } .

Original: For these outputs, they use the lower bound τ to compute robustness certiﬁcates.
Revised: For these outputs, they use the lower bound τ to compute robustness certiﬁcates.

Original: They abstain with all other outputs.
Revised: They abstain with all other outputs.

Original: This approach is sensible when one is concerned with the least robust prediction from a set of predictions.
Revised: This approach is sensible when one is concerned with the least robust prediction from a set of predictions.

Original: But our collective certiﬁcate beneﬁts from having tight robustness guarantees for each of the individual predictions.
Revised: But our collective certiﬁcate beneﬁts from having tight robustness guarantees for each of the individual predictions.

Original: Holm correction can be used with arbitrary hypothesis tests.
Revised: Holm correction can be used with arbitrary hypothesis tests.

Original: For instance, we can use a different threshold τ n per output g n , i.e. test whether
Revised: For instance, we can use a different threshold τ n per output g n , i.e. test whether

Original: [Equation]
Revised: [Equation]

Original: for k = 1 , . . .
Revised: for k = 1 , . . .

Original: , D out .
Revised: , D out .

Original: In particular, we can use
Revised: In particular, we can use

Original: [Equation]
Revised: [Equation]

Original: i.e. choose the largest threshold such that the null hypothesis can still be rejected.
Revised: i.e. choose the largest threshold such that the null hypothesis can still be rejected.

Original: Eq.
Revised: Eq.

Original: 114 is the lower Clopper-Pearson conﬁdence bound with signiﬁcance α D out +1 − π − 1 ( n ) .
Revised: 114 is the lower Clopper-Pearson conﬁdence bound with signiﬁcance α D out +1 − π − 1 ( n ) .

Original: This means that, instead of performing hypothesis tests, we can obtain probabilistic lower bounds q n ≤ q n by computing
Revised: This means that, instead of performing hypothesis tests, we can obtain probabilistic lower bounds q n ≤ q n by computing

Original: Clopper-Pearson conﬁdence bounds with signiﬁcance parameters α D out , . . .
Revised: Clopper-Pearson conﬁdence bounds with signiﬁcance parameters α D out , . . .

Original: , α 1 .
Revised: , α 1 .

Original: The q n can then be used to compute the base certiﬁcates.
Revised: The q n can then be used to compute the base certiﬁcates.

Original: Due to the deﬁnition of the τ n , all of the null hypotheses are rejected, i.e. we obtain valid probabilistic lower bounds on all q n .
Revised: Due to the deﬁnition of the τ n , all of the null hypotheses are rejected, i.e. we obtain valid probabilistic lower bounds on all q n .

Original: We can thus use the abstention mechanism from Section C.1, i.e. only abstain if q n ≤ 0 .
Revised: We can thus use the abstention mechanism from Section C.1, i.e. only abstain if q n ≤ 0 .

Original: A direct consequence of the results above is that using Clopper-Pearson conﬁdence intervals and
Revised: A direct consequence of the results above is that using Clopper-Pearson conﬁdence intervals and

Original: Holm correction will yield stronger per-prediction robustness guarantees and lower abstention rates than the method of Fischer et al.
Revised: Holm correction will yield stronger per-prediction robustness guarantees and lower abstention rates than the method of Fischer et al.

Original: The Clopper-Pearson-based method only abstains if one cannot guarantee that q n > 0 .
Revised: The Clopper-Pearson-based method only abstains if one cannot guarantee that q n > 0 .

Original: 5 with high probability, while their method abstains if one cannot guarantee that q n ≥ τ with τ ≥ 0 .
Revised: 5 with high probability, while their method abstains if one cannot guarantee that q n ≥ τ with τ ≥ 0 .

Original: 5 (or speciﬁc other predictions abstain).
Revised: 5 (or speciﬁc other predictions abstain).

Original: For all non-abstaining predictions, the Clopper-Pearson-based certiﬁcate will be at least as strong as that obtained using a single threshold τ , as it computes the tightest bound for which the null hypothesis can still be rejected (see Eq. 114).
Revised: For all non-abstaining predictions, the Clopper-Pearson-based certiﬁcate will be at least as strong as that obtained using a single threshold τ , as it computes the tightest bound for which the null hypothesis can still be rejected (see Eq. 114).

Original: Consequently, a na¨ıve collective robustness certiﬁcate (i.e. counting the number of predictions whose robustness are guaranteed by the base certiﬁcates) based on ClopperPearson bounds will also be stronger.
Revised: Consequently, a na¨ıve collective robustness certiﬁcate (i.e. counting the number of predictions whose robustness are guaranteed by the base certiﬁcates) based on ClopperPearson bounds will also be stronger.

Original: It should however be noted that this is only relevant for our notion of collective robustness, not for the one considered by Fischer et al.
Revised: It should however be noted that this is only relevant for our notion of collective robustness, not for the one considered by Fischer et al.

Original: It should also be noted that their method could potentially be used with other methods of family-wise error rate correction, although they state that “these methods do not scale to realistic segmentation problems” and do not discuss any further details.
Revised: It should also be noted that their method could potentially be used with other methods of family-wise error rate correction, although they state that “these methods do not scale to realistic segmentation problems” and do not discuss any further details.

Original: In our experiments (see Section 8), we use Holm correction for our  na¨ıve collective certiﬁcate  and  the  weaker Bonferroni correction for our base certiﬁcates.
Revised: Our above discussion shows that, for our notion of collective robustness, the na¨ıve collective certiﬁcate using Clopper-Pearson conﬁdence bounds and Holm correction is at least as strong as that of Fischer et al. (2021) (or alternatively: the certiﬁcate of Fischer et al. (2021) is a na¨ıve collective certiﬁcate with Holm correction, but with weaker per-prediction certiﬁcates). Conversely, the certiﬁcate based on Clopper-Pearson conﬁdence bounds is at least as strong as that of Fischer et al.

Revised: (2021) when certifying their notion of collective robustness, i.e. determining whether all non-abstaining predictions are robust, given adversarial budget (cid:15) .

Revised: To certify this notion of robustness, they simply iterate over all predictions and determine whether all non-abstaining predictions are certiﬁably robust, given (cid:15) .

Revised: Naturally, as the Clopper-Pearson-based certiﬁcates are stronger, any prediction that is robust according to (Fischer et al., 2021) is also robust acccording to the Clopper-Pearson-based certiﬁcates.

Revised: The only difference is that, for τ > 0 . 5 , their method will have more abstaining predictions.

Revised: But, due to the direct correspondence of Clopper-Pearson conﬁdence bounds and Binomial tests, we can modify our abstention mechanism to obtain exactly the same set of abstaining predictions: We simply have to use q n ≤ τ instead of q n ≤ 0 .

Revised: 5 as our abstention criterion.

Revised: Finally, it should be noted that the proposed collective certiﬁcate is at least as strong as the na ¨ ıve collective certiﬁcate (see Section 4).

Revised: Thus, letting the set of targeted predictions T be the set of all non-abstaining predictions and checking whether the collective certiﬁcate guarantees robustness for all of T will also result in a certiﬁcate that is at least as strong as that of Fischer et al.

Revised: (2021) in their setting.

Revised: In our experiments (see Section 8), we use Holm correction for our na ¨ ıve collective certiﬁcate and the weaker Bonferroni correction for our base certiﬁcates.

Original: This is only meant to slightly skew the results in favor of our baselines.
Revised: This is only meant to slightly skew the results in favor of our baselines.

Original: Holm correction can in principle also be used when computing the base certiﬁcates, in order to improve our proposed collective cert.
Revised: Holm correction can in principle also be used when computing the base certiﬁcates, in order to improve our proposed collective cert.

Original: D C OMPARISON TO THE COLLECTIVE CERTIFICATE OF S CHUCHARDT ET AL .
Revised: D C OMPARISON TO THE COLLECTIVE CERTIFICATE OF S CHUCHARDT ET AL .

Original: In the following, we ﬁrst present the collective certiﬁcate for binary graph-structured data proposed by Schuchardt et al.
Revised: In the following, we ﬁrst present the collective certiﬁcate for binary graph-structured data proposed by Schuchardt et al.

Original: (see Section D.1.
Revised: (see Section D.1.

Original: We then show that, when using sparsity-aware smoothing distributions (Bojchevski et al., 2020) – the family of smoothing distributions used both in our work and that of Schuchardt et al.
Revised: We then show that, when using sparsity-aware smoothing distributions (Bojchevski et al., 2020) – the family of smoothing distributions used both in our work and that of Schuchardt et al.

Original: (2021) – our certiﬁcate subsumes their certiﬁcate.
Revised: (2021) – our certiﬁcate subsumes their certiﬁcate.

Original: That is, our collective robustness certiﬁcate based on localized randomized smoothing can provide the same robustness guarantees (see Section D.2).
Revised: That is, our collective robustness certiﬁcate based on localized randomized smoothing can provide the same robustness guarantees (see Section D.2).

Original: D. T HE COLLECTIVE CERTIFICATE
Revised: D. T HE COLLECTIVE CERTIFICATE

Original: Their certiﬁcate assumes the input space to be G = { 0 , 1 } N × D ×{ 0 , 1 } N × N – the set of undirected attributed graphs with N nodes and D attributes per node.
Revised: Their certiﬁcate assumes the input space to be G = { 0 , 1 } N × D ×{ 0 , 1 } N × N – the set of undirected attributed graphs with N nodes and D attributes per node.

Original: The model is assumed to be a multioutput classiﬁer f : G → Y N that assigns a label from label set Y to each of the nodes.
Revised: The model is assumed to be a multioutput classiﬁer f : G → Y N that assigns a label from label set Y to each of the nodes.

Original: Given an input graph G =
Revised: Given an input graph G =

Original: ( X , A ) and a corresponding prediction y =
Revised: ( X , A ) and a corresponding prediction y =

Original: f ( G ) , they want to certify collective robustness to a set of perturbed graphs B ⊆ G .
Revised: f ( G ) , they want to certify collective robustness to a set of perturbed graphs B ⊆ G .

Original: The perturbation model B is characterized by four scalar parameters r + X , r − X , r + A , r + A ∈ N 0 , specifying the number of bits the adversary is allowed to add ( 0 → 1 ) and delete ( 1 → 0 ) in the attribute and adjacency matrix, respectively.
Revised: The perturbation model B is characterized by four scalar parameters r + X , r − X , r + A , r + A ∈ N 0 , specifying the number of bits the adversary is allowed to add ( 0 → 1 ) and delete ( 1 → 0 ) in the attribute and adjacency matrix, respectively.

Original: It can also be extended to feature additional constraints (e.g. per-node budgets).
Revised: It can also be extended to feature additional constraints (e.g. per-node budgets).

Original: We discuss how these can be integrated after showing our main result.
Revised: We discuss how these can be integrated after showing our main result.

Original: A formal deﬁnition of the perturbation model can be found in Section B of (Schuchardt et al., 2021).
Revised: A formal deﬁnition of the perturbation model can be found in Section B of (Schuchardt et al., 2021).

Original: The goal of their work is to certify collective robustness for a set of targeted nodes T ⊆ { 1 , . . .
Revised: The goal of their work is to certify collective robustness for a set of targeted nodes T ⊆ { 1 , . . .

Original: , N } , i.e. compute a lower bound on
Revised: , N } , i.e. compute a lower bound on

Original: [Equation]
Revised: [Equation]

Original: Their approach to obtaining this lower-bound shares the same high-level idea as ours: Combining per-prediction base certiﬁcates and leveraging some notion of locality.
Revised: Their approach to obtaining this lower-bound shares the same high-level idea as ours: Combining per-prediction base certiﬁcates and leveraging some notion of locality.

Original: But while our method uses localized randomized smoothing, i.e. smoothing different outputs with different non-i.i.d.
Revised: But while our method uses localized randomized smoothing, i.e. smoothing different outputs with different non-i.i.d.

Original: smoothing distributions, to obtain base certiﬁcates that encode locality, their method uses a-priori knowledge about the strict locality of the classiﬁer f .
Revised: smoothing distributions, to obtain base certiﬁcates that encode locality, their method uses a-priori knowledge about the strict locality of the classiﬁer f .