Original: Invariance-Aware Randomized Smoothing Certificates
Revised: Invariance-Aware Randomized Smoothing Certificates

Original: Abstract
Revised: Abstract

Original: Building models that comply with the invariances inherent to different domains, such as invariance under translation or rotation, is a key aspect of applying machine learning to real world problems like molecular property prediction, medical imaging, protein folding or LiDAR classification.
Revised: Building models that comply with the invariances inherent to different domains, such as invariance under translation or rotation, is a key aspect of applying machine learning to real world problems like molecular property prediction, medical imaging, protein folding or LiDAR classification.

Original: For the first time, we study how the invariances of a model can be leveraged to provably guarantee the robustness of its predictions.
Revised: For the first time, we study how the invariances of a model can be leveraged to provably guarantee the robustness of its predictions.

Original: We propose a gray-box approach, enhancing the powerful black-box randomized smoothing technique with white-box knowledge about invariances.
Revised: We propose a gray-box approach, enhancing the powerful black-box randomized smoothing technique with white-box knowledge about invariances.

Original: First, we develop gray-box certificates based on group orbits, which can be applied to arbitrary models with invariance under permutation and Euclidean isometries.
Revised: First, we develop gray-box certificates based on group orbits, which can be applied to arbitrary models with invariance under permutation and Euclidean isometries.

Original: Then, we derive provably tight gray-box certificates.
Revised: Then, we derive provably tight gray-box certificates.

Original: We experimentally demonstrate that the provably tight certificates can offer much stronger guarantees, but that in practical scenarios the orbit-based method is a good approximation.
Revised: We experimentally demonstrate that the provably tight certificates can offer much stronger guarantees, but that in practical scenarios the orbit-based method is a good approximation.

Original:  Introduction
Revised:  Introduction

Original: It is well-established that machine learning models are susceptible to adversarial attacks [1–4].
Revised: It is well-established that machine learning models are susceptible to adversarial attacks [1–4].

Original: Even without malevolent actors, adversarial attacks can be considered worst case scenarios in environments with noisy, erroneous or otherwise corrupted data, thus necessitating robust machine learning methods.
Revised: Even without malevolent actors, adversarial attacks can be considered worst case scenarios in environments with noisy, erroneous or otherwise corrupted data, thus necessitating robust machine learning methods.

Original: Invariance is a central design principle that has so far received little dedicated attention in the realm of adversarially robust machine learning.
Revised: Invariance is a central design principle that has so far received little dedicated attention in the realm of adversarially robust machine learning.

Original: Over the past decades, there has been ongoing research into developing machine learning models that comply with the invariances inherent to different data types and tasks.
Revised: Over the past decades, there has been ongoing research into developing machine learning models that comply with the invariances inherent to different data types and tasks.

Original: Prominent recent examples include Deep Sets [5], PointNet [6], Group Equivariant
Revised: Prominent recent examples include Deep Sets [5], PointNet [6], Group Equivariant

Original: CNNs [7], Spherical CNNs [8] and Graph Convolutional Networks [9] but the study of invariant models significantly precedes the surge in popularity of deep learning methods [10–15].
Revised: CNNs [7], Spherical CNNs [8] and Graph Convolutional Networks [9] but the study of invariant models significantly precedes the surge in popularity of deep learning methods [10–15].

Original: For the first time we explore the following question: Can a-priori knowledge about invariances be leveraged in deriving provable guarantees for a model’s robustness to adversarial attacks?
Revised: For the first time we explore the following question: Can a-priori knowledge about invariances be leveraged in deriving provable guarantees for a model’s robustness to adversarial attacks?

Original: Going by a loose categorization of prior work, we could adopt one of two possible approaches for our exploration: A white-box or black-box one.
Revised: Going by a loose categorization of prior work, we could adopt one of two possible approaches for our exploration: A white-box or black-box one.

Original: White-box certificates (e.g. [16–23]) analyze a model’s internals, such at its weights and non-linearities, to provably guarantee that a prediction does not change under adversarial attack.
Revised: White-box certificates (e.g. [16–23]) analyze a model’s internals, such at its weights and non-linearities, to provably guarantee that a prediction does not change under adversarial attack.

Original: Black box certificates – specifically randomized smoothing [24–26] – use statistical methods to provide provable guarantees that hold for all models sharing the same prediction probabilities under a random input distribution, irrespective of their internals.
Revised: Black box certificates – specifically randomized smoothing [24–26] – use statistical methods to provide provable guarantees that hold for all models sharing the same prediction probabilities under a random input distribution, irrespective of their internals.

Original: We opt for a randomized smoothing approach, as it allows us to focus on the interplay between invariances and robustness, rather than the specific means of implementing these invariances.
Revised: We opt for a randomized smoothing approach, as it allows us to focus on the interplay between invariances and robustness, rather than the specific means of implementing these invariances.

Original: By combining white-box knowledge about invariances with black-box knowledge about prediction probabilities we obtain gray-box certificates.
Revised: By combining white-box knowledge about invariances with black-box knowledge about prediction probabilities we obtain gray-box certificates.

Original: We first derive a gray-box certificate that follows a post-processing paradigm: It takes an existing black-box certificate and augments the certified region using information about the model’s invariances.
Revised: We first derive a gray-box certificate that follows a post-processing paradigm: It takes an existing black-box certificate and augments the certified region using information about the model’s invariances.

Original: This orbit-based certificate serves as a baseline for the provably tight gray-box certificates we derive in subsequent sections.
Revised: This orbit-based certificate serves as a baseline for the provably tight gray-box certificates we derive in subsequent sections.

Original: For our exploration, we focus on models operating on spatial data, rather than structured data (e.g. images and sequences) – both because spatial symmetries can be elegantly formalized using algebraic concepts and because there is an ongoing trend towards using machine learning for real-world applications with inherent spatial invariances, e.g. molecular property prediction [27–34], LiDAR classification [35–38], drug discovery [39–41], particle physics [42–44] and protein folding [45–49].
Revised: For our exploration, we focus on models operating on spatial data, rather than structured data (e.g. images and sequences) – both because spatial symmetries can be elegantly formalized using algebraic concepts and because there is an ongoing trend towards using machine learning for real-world applications with inherent spatial invariances, e.g. molecular property prediction [27–34], LiDAR classification [35–38], drug discovery [39–41], particle physics [42–44] and protein folding [45–49].

Original: Our main contributions are
Revised: Our main contributions are

Original: • the first study on the interplay of invariance and certifiable robustness, • a principled method for deriving tight invariance-aware randomized smoothing certificates, • tight certificates for models invariant to translations and rotations.
Revised: • the first study on the interplay of invariance and certifiable robustness, • a principled method for deriving tight invariance-aware randomized smoothing certificates, • tight certificates for models invariant to translations and rotations.

Original: We further demonstrate experimentally that the orbit-based certificates offer a good approximation of our tight certificates, if the variance of the smoothing distribution is small.
Revised: We further demonstrate experimentally that the orbit-based certificates offer a good approximation of our tight certificates, if the variance of the smoothing distribution is small.

Original:  Related work
Revised:  Related work

Original: Invariant machine learning.
Revised: Invariant machine learning.

Original: Given the diversity of approaches to invariant machine learning, and the fact that our approach is model-agnostic, we refrain from attempting a survey and instead refer to [50] for a principled, high-level introduction into the realm of learning with invariances and equivariances.
Revised: Given the diversity of approaches to invariant machine learning, and the fact that our approach is model-agnostic, we refrain from attempting a survey and instead refer to [50] for a principled, high-level introduction into the realm of learning with invariances and equivariances.

Original: Invariance and robustness.
Revised: Invariance and robustness.

Original: Two recent empirical studies [51, 52] demonstrate that data augmentation meant to increase robustness to ℓ p -norm adversarial attacks reduces robustness to semantically meaningful transformations (e.g. rotation) and vice-versa, suggesting an inherent invariancerobustness trade-off.
Revised: Two recent empirical studies [51, 52] demonstrate that data augmentation meant to increase robustness to ℓ p -norm adversarial attacks reduces robustness to semantically meaningful transformations (e.g. rotation) and vice-versa, suggesting an inherent invariancerobustness trade-off.

Original: However, neither study models that are invariant by design.
Revised: However, neither study models that are invariant by design.

Original: In [53], the negative effect of translation-invariance on the robustness of image classifiers is investigated.
Revised: In [53], the negative effect of translation-invariance on the robustness of image classifiers is investigated.

Original: Note that our work is not meant to resolve potential trade-offs, but to tightly bound the actual robustness of models.
Revised: Note that our work is not meant to resolve potential trade-offs, but to tightly bound the actual robustness of models.

Original: Gray-box certificates.
Revised: Gray-box certificates.

Original: While we propose the first gray-box certificate for invariant models, there exists prior work on combining white-box knowledge with black-box certification.
Revised: While we propose the first gray-box certificate for invariant models, there exists prior work on combining white-box knowledge with black-box certification.

Original: In [54] and [55], knowledge about a classifier’s gradients is used to derive tighter randomized smoothing certificates.
Revised: In [54] and [55], knowledge about a classifier’s gradients is used to derive tighter randomized smoothing certificates.

Original: In [56], knowledge about a graph neural network’s receptive fields is used to derive collective gray-box certificates for multiple predictions.
Revised: In [56], knowledge about a graph neural network’s receptive fields is used to derive collective gray-box certificates for multiple predictions.

Original: In [57], the message-passing scheme of graph neural networks is used for gray-box certification against adversaries that control all features of multiple nodes.
Revised: In [57], the message-passing scheme of graph neural networks is used for gray-box certification against adversaries that control all features of multiple nodes.

Original: Adversarial attacks on spatial data.
Revised: Adversarial attacks on spatial data.

Original: Adversarial attacks on spatial data either modify [58–66], insert [65, 67, 68], or delete [69–71] points in space and have been particularly actively studied for point cloud-classification.
Revised: Adversarial attacks on spatial data either modify [58–66], insert [65, 67, 68], or delete [69–71] points in space and have been particularly actively studied for point cloud-classification.

Original: While the earliest work simply adopted gradient-based attacks from the image domain [58] more recent work has developed a rich assortment of domain-specific methods, for example to preserve object smoothness [64, 72], leverage critical points [69, 70] or craft physically realizable attacks (e.g. to attack models through LiDAR sensors) [61, 73–75].
Revised: While the earliest work simply adopted gradient-based attacks from the image domain [58], more recent work has developed a rich assortment of domain-specific methods, for example to preserve object smoothness [64, 72], leverage critical points [69, 70] or craft physically realizable attacks (e.g. to attack models through LiDAR sensors) [61, 73–75].

Original: Of particular note are attacks via isometries (e.g. rotations and translations)
Revised: Of particular note are attacks via isometries (e.g. rotations and translations)

Original: [76, 77], whose effectiveness motivates the use of invariant models.
Revised: [76, 77], whose effectiveness motivates the use of invariant models.

Original: Note that we certify the robustness of invariant models to arbitrary point modification attacks – not just isometry attacks (see also “orthogonal research directions” below).
Revised: Note that we certify the robustness of invariant models to arbitrary point modification attacks – not just isometry attacks (see also “orthogonal research directions” below).

Original: Like in other domains, empirical defenses have been proposed [59, 78, 79] and subsequently broken [80–82], motivating the development of black-box [83–85] and white-box [86] robustness certificates for spatial data.
Revised: Like in other domains, empirical defenses have been proposed [59, 78, 79] and subsequently broken [80–82], motivating the development of black-box [83–85] and white-box [86] robustness certificates for spatial data.

Original: It should be noted that Gaussian randomized smoothing, without invariance information, has already been used in prior work – either as a baseline [83] or as a special case of the respective certificate [85].
Revised: It should be noted that Gaussian randomized smoothing, without invariance information, has already been used in prior work – either as a baseline [83] or as a special case of the respective certificate [85].

Original: Orthogonal research directions.
Revised: Orthogonal research directions.

Original: One related but orthogonal research direction is transformationspecific certification [23, 85–93].
Revised: One related but orthogonal research direction is transformationspecific certification [23, 85–93].

Original: There, a model is assumed to potentially change its prediction under adversarial parametric transformations (e.g. rotations) and one certifies robustness for specific parameter ranges.
Revised: There, a model is assumed to potentially change its prediction under adversarial parametric transformations (e.g. rotations) and one certifies robustness for specific parameter ranges.

Original: White-box methods [23, 86–89] over-approximate the set of inputs reachable by a transformation and then propagate it through a model using existing white-box techniques.
Revised: White-box methods [23, 86–89] over-approximate the set of inputs reachable by a transformation and then propagate it through a model using existing white-box techniques.

Original: Black-box methods – namely transformation-specific randomized smoothing [90, 94] – randomize the transformation parameters to provide robustness guarantees for arbitrary models via statistical methods.
Revised: Black-box methods – namely transformation-specific randomized smoothing [90, 94] – randomize the transformation parameters to provide robustness guarantees for arbitrary models via statistical methods.

Original: Later work generalized this principle to spatial data [85], vector-field deformations [92] and multiplicative parameters [93].
Revised: Later work generalized this principle to spatial data [85], vector-field deformations [92] and multiplicative parameters [93].

Original: Different from these methods, we assume our model to be invariant under a set of transformations, i.e. never change its prediction.
Revised: Different from these methods, we assume our model to be invariant under a set of transformations, i.e. never change its prediction.

Original: We use this property as a tool for certification against arbitrary perturbations.
Revised: We use this property as a tool for certification against arbitrary perturbations.

Original: Aside from that, there exist white-box certification techniques for specific operations with invariances (e.g. global max-pooling [86], message passing [95, 96] and batch normalization [97]).
Revised: Aside from that, there exist white-box certification techniques for specific operations with invariances (e.g. global max-pooling [86], message passing [95, 96] and batch normalization [97]).

Original: But prior work does not use or even discuss this property – it treats these invariant operations as coincidental building blocks of the models it is trying to certify.
Revised: But prior work does not use or even discuss this property – it treats these invariant operations as coincidental building blocks of the models it is trying to certify.

Original: Our work is the first to study invariance itself in the context of provable robustness and how to leverage it for certification.
Revised: Our work is the first to study invariance itself in the context of provable robustness and how to leverage it for certification.

Original:  Background 3.1 Randomized smoothing
Revised:  Background 3.1 Randomized smoothing

Original: Randomized smoothing is a black-box certification technique that can be adapted to various data types, tasks and threat models [98–103, 91, 104–106].
Revised: Randomized smoothing is a black-box certification technique that can be adapted to various data types, tasks and threat models [98–103, 91, 104–106].

Original: Instead of directly certifying a classifier g , it constructs a smoothed classifier f that returns the most likely prediction under random perturbations of its input.
Revised: Instead of directly certifying a classifier g , it constructs a smoothed classifier f that returns the most likely prediction under random perturbations of its input.

Original: It then certifies the robustness of this smoothed classifier.
Revised: It then certifies the robustness of this smoothed classifier.

Original: We present the tight Gaussian smoothing certificate derived by Cohen et al.
Revised: We present the tight Gaussian smoothing certificate derived by Cohen et al.

Original: [26] and its generalization to matrix data [83, 85].
Revised: [26] and its generalization to matrix data [83, 85].

Original: Assume a continuous ( N × D ) -dimensional input space R N × D , label set Y and base classifier g :
Revised: Assume a continuous ( N × D ) -dimensional input space R N × D , label set Y and base classifier g :

Original: R N × D → Y .
Revised: R N × D → Y .

Original: Let µ X ( Z ) =
Revised: Let µ X ( Z ) =

Original: (cid:81)
Revised: (cid:81)

Original: Dd =1 N (cid:0) Z : ,d | X : ,d , σ 2 I N (cid:1) be the isotropic matrix normal distribution with mean X and standard deviation σ .
Revised: Dd =1 N (cid:0) Z : ,d | X : ,d , σ 2 I N (cid:1) be the isotropic matrix normal distribution with mean X and standard deviation σ .

Original: Let p X ,y =
Revised: Let p X ,y =

Original: Pr Z ∼ µ X [ g ( Z ) =
Revised: Pr Z ∼ µ X [ g ( Z ) =

Original: y ] be the probability of g predicting class y under this smoothing distribution .
Revised: y ] be the probability of g predicting class y under this smoothing distribution .

Original: One can then define a smoothed classifier f
Revised: One can then define a smoothed classifier f

Original: ( X ) =
Revised: ( X ) =

Original: argmax y ∈ Y p X ,y that returns the most likely prediction of g under µ X .
Revised: argmax y ∈ Y p X ,y that returns the most likely prediction of g under µ X .

Original: Let y ∗ =
Revised: Let y ∗ =

Original: f ( X ) be a smoothed prediction and X ′ =
Revised: f ( X ) be a smoothed prediction and X ′ =

Original: X + ∆ a perturbed input.
Revised: X + ∆ a perturbed input.

Original: One can show f ( X ′ )
Revised: One can show f ( X ′ )

Original: =
Revised: =

Original: y ∗ by proving that, for perturbed input X ′ , y ∗ is more likely than all other classes combined.
Revised: y ∗ by proving that, for perturbed input X ′ , y ∗ is more likely than all other classes combined.

Original: That is, p X ′ ,y ∗ > 0 .
Revised: That is, p X ′ ,y ∗ > 0 .

Original: A tighter certificate can be obtained by proving that y ∗ is more likely than the second most likely class, i.e. p X ′ ,y ∗ >
Revised: A tighter certificate can be obtained by proving that y ∗ is more likely than the second most likely class, i.e. p X ′ ,y ∗ >

Original: max y ′ ̸
Revised: max y ′ ̸

Original: =
Revised: =

Original: y ∗ p X ′ ,y ′ .
Revised: y ∗ p X ′ ,y ′ .

Original: For the sake of exposition, we use the first approach throughout the main text and generalize all results to the second one in Appendix H.
Revised: For the sake of exposition, we use the first approach throughout the main text and generalize all results to the second one in Appendix H.

Original: One can lower-bound p X ′ ,y ∗ by finding the worst-case classifier from a set of functions H with g ∈ H :
Revised: One can lower-bound p X ′ ,y ∗ by finding the worst-case classifier from a set of functions H with g ∈ H :

Original: [Equation]
Revised: [Equation]

Original: For H = (cid:8) h : R N × D
Revised: For H = (cid:8) h : R N × D

Original: → Y | Pr Z ∼ µ X [ h ( Z ) =
Revised: → Y | Pr Z ∼ µ X [ h ( Z ) =

Original: y ∗ ] ≥ p X ,y ∗ (cid:9) , the classifiers that are at least as likely as g to classify X as y ∗ , the exact solution is given by the Neyman-Pearson lemma [107] (see Appendix F.2).
Revised: y ∗ ] ≥ p X ,y ∗ (cid:9) , the classifiers that are at least as likely as g to classify X as y ∗ , the exact solution is given by the Neyman-Pearson lemma [107] (see Appendix F.2).

Original: The optimal value is Φ (cid:16)
Revised: The optimal value is Φ (cid:16)

Original: Φ − 1 ( p X ,y ∗ ) − || ∆ || 2 σ (cid:17) , where Φ is the standard-normal CDF and || · || 2 is the Frobenius norm.
Revised: Φ − 1 ( p X ,y ∗ ) − || ∆ || 2 σ (cid:17) , where Φ is the standard-normal CDF and || · || 2 is the Frobenius norm.

Original: If || ∆ || 2 < σ Φ − 1 ( p X ,y ∗ ) , then p X ′ ,y ∗ > 0 .
Revised: If || ∆ || 2 < σ Φ − 1 ( p X ,y ∗ ) , then p X ′ ,y ∗ > 0 .

Original: and the prediction is provably robust.
Revised: and the prediction is provably robust.

Original: Because Eq.
Revised: Because Eq.

Original: (1) was solved exactly, this is a tight certificate , i.e. the best possible certificate that can be obtained by only using black-box knowledge about prediction probability p X ,y ∗ .
Revised: (1) was solved exactly, this is a tight certificate , i.e. the best possible certificate that can be obtained by only using black-box knowledge about prediction probability p X ,y ∗ .

Original: Probabilistic certificates.
Revised: Probabilistic certificates.

Original: For neural networks, the prediction probability p X ,y ∗ can usually not be computed analytically.
Revised: For neural networks, the prediction probability p X ,y ∗ can usually not be computed analytically.

Original: Instead, one has to use Monte Carlo sampling to compute a lower confidence bound p x ′ ,y ∗ that holds with high probability 1 − α .
Revised: Instead, one has to use Monte Carlo sampling to compute a lower confidence bound p x ′ ,y ∗ that holds with high probability 1 − α .

Original: The resulting certificate is a probabilistic one.
Revised: The resulting certificate is a probabilistic one.

Original: 3.2 Group invariance
Revised: 3.2 Group invariance

Original: Let f : R N × D
Revised: Let f : R N × D

Original: → Y be a classifier and T a group, i.e. a set and associated operator · : T × T → T that is closed and associative under the operation, has an inverse t − 1 for each element t ∈ T and features an identity element e .
Revised: → Y be a classifier and T a group, i.e. a set and associated operator · : T × T → T that is closed and associative under the operation, has an inverse t − 1 for each element t ∈ T and features an identity element e .

Original: Further let T act on the input space via a group action ◦ :
Revised: Further let T act on the input space via a group action ◦ :

Original: T × R N × D → R N × D that preserves the group structure, i.e. ( t · t ′ ) ◦
Revised: T × R N × D → R N × D that preserves the group structure, i.e. ( t · t ′ ) ◦

Original: X = t ◦ ( t ′ ◦ X ) .
Revised: X = t ◦ ( t ′ ◦ X ) .

Original: Group actions naturally partition the domain into orbits , i.e. sets that can be reached by applying group actions to inputs:
Revised: Group actions naturally partition the domain into orbits , i.e. sets that can be reached by applying group actions to inputs:

Original: Definition 1 (Orbits) .
Revised: Definition 1 (Orbits) .

Original: The orbit of an input X ∈ R N × D w.r.t.
Revised: The orbit of an input X ∈ R N × D w.r.t.

Original: a group T is [ X ] T = { t ◦ X | t ∈ T } .
Revised: a group T is [ X ] T = { t ◦ X | t ∈ T } .

Original: Classifier f is said to be invariant under group T if ∀ X ∈ R N × D , ∀ X ′ ∈
Revised: Classifier f is said to be invariant under group T if ∀ X ∈ R N × D , ∀ X ′ ∈

Original: [ X ] T : f ( X ) = f
Revised: [ X ] T : f ( X ) = f

Original: ( X ′ ) .
Revised: ( X ′ ) .

Original: 3.3 Haar measures
Revised: 3.3 Haar measures

Original: Haar measures [108] are a generalization of Lebesgue measures for integration over groups.
Revised: Haar measures [108] are a generalization of Lebesgue measures for integration over groups.

Original: Their key property is invariance, meaning the group operator does not affect the measure.
Revised: Their key property is invariance, meaning the group operator does not affect the measure.

Original: Definition 2 (Right Haar measure) .
Revised: Definition 2 (Right Haar measure) .

Original: Let η be a finite, regular measure on the Borel subsets of T .
Revised: Let η be a finite, regular measure on the Borel subsets of T .

Original: If η ( S ) =
Revised: If η ( S ) =

Original: η ( { s · t | s ∈ S } ) for all t ∈ T and Borel subsets S ⊆ T , then η is a right Haar measure.
Revised: η ( { s · t | s ∈ S } ) for all t ∈ T and Borel subsets S ⊆ T , then η is a right Haar measure.

Original: For the groups we consider, the Haar measure is unique up to a multiplicative constant [109, 110].
Revised: For the groups we consider, the Haar measure is unique up to a multiplicative constant [109, 110].

Original: In Section 6.1 we use Haar measures as a notational tool to summarize certificates for different groups.
Revised: In Section 6.1 we use Haar measures as a notational tool to summarize certificates for different groups.

Original: However, an understanding of measure theory is not required to follow any part of the derivations.
Revised: However, an understanding of measure theory is not required to follow any part of the derivations.

Original: [Figure]
Revised: [Figure]

Original:  Problem setting
Revised:  Problem setting

Original: We consider a similar setup to that described in Section 3.1, i.e. we have a smoothed classifier f : R N × D
Revised: We consider a similar setup to that described in Section 3.1, i.e. we have a smoothed classifier f : R N × D

Original: → Y that is the result of randomly smoothing a base classifier g with an isotropic matrix normal distribution µ X .
Revised: → Y that is the result of randomly smoothing a base classifier g with an isotropic matrix normal distribution µ X .

Original: Given a clean prediction y ∗
Revised: Given a clean prediction y ∗

Original: =
Revised: =

Original: f
Revised: f

Original: ( X ) with clean prediction probability p X ,y ∗ , we want to determine whether f ( X ′ )
Revised: ( X ) with clean prediction probability p X ,y ∗ , we want to determine whether f ( X ′ )

Original: =
Revised: =

Original: y ∗ for an adversarially perturbed input X ′ =
Revised: y ∗ for an adversarially perturbed input X ′ =

Original: X + ∆ .
Revised: X + ∆ .

Original: Different from all prior work, we additionally assume that the base classifier g is invariant under a group T .
Revised: Different from all prior work, we additionally assume that the base classifier g is invariant under a group T .

Original: This does not necessarily mean that the smoothed classifier f shares the same invariances.
Revised: This does not necessarily mean that the smoothed classifier f shares the same invariances.

Original: But, we can use the isotropy of smoothing distribution µ X to prove (see Appendix D) that randomized smoothing preserves invariance under Euclidean isotropies (rotation, reflection and translation) and permutation, which we shall leverage in Section 5.
Revised: But, we can use the isotropy of smoothing distribution µ X to prove (see Appendix D) that randomized smoothing preserves invariance under Euclidean isotropies (rotation, reflection and translation) and permutation, which we shall leverage in Section 5.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let base classifier g : R N × D
Revised: Let base classifier g : R N × D

Original: → Y be invariant under group T with T ⊆
Revised: → Y be invariant under group T with T ⊆

Original: E ( D ) or T ⊆ S ( N ) , where E ( D ) is the Euclidean group and S ( N ) is the permutation group.
Revised: E ( D ) or T ⊆ S ( N ) , where E ( D ) is the Euclidean group and S ( N ) is the permutation group.

Original: Then the isotropically smoothed classifier f , as defined in Section 3.1, is also invariant under T .
Revised: Then the isotropically smoothed classifier f , as defined in Section 3.1, is also invariant under T .

Original: Note that the above result also holds for subgroups of E ( D ) , such as the translation group T ( D ) , the rotation group SO
Revised: Note that the above result also holds for subgroups of E ( D ) , such as the translation group T ( D ) , the rotation group SO

Original: ( D ) and the roto-translation group SE ( D ) .
Revised: ( D ) and the roto-translation group SE ( D ) .

Original: We define these groups and their actions on R N × D more formally in Appendix C.
Revised: We define these groups and their actions on R N × D more formally in Appendix C.

Original:  Orbit-based gray-box certificates
Revised:  Orbit-based gray-box certificates

Original: Since we are the first to consider robustness certification for invariant models, we begin by defining a baseline that other certificates can be benchmarked against: It is based on the insight that certificates correspond to sets of inputs B ⊆ R N × D that preserve clean prediction y ∗ , and group T corresponds to sets of transformations that preserve predictions.
Revised: Since we are the first to consider robustness certification for invariant models, we begin by defining a baseline that other certificates can be benchmarked against: It is based on the insight that certificates correspond to sets of inputs B ⊆ R N × D that preserve clean prediction y ∗ , and group T corresponds to sets of transformations that preserve predictions.

Original: By transitivity, a perturbed input X ′ that can reach B via a transformation from T , i.e. ∃ t ∈ T : t ◦ X ′ ∈ B , must fulfill f ( X ′ )
Revised: By transitivity, a perturbed input X ′ that can reach B via a transformation from T , i.e. ∃ t ∈ T : t ◦ X ′ ∈ B , must fulfill f ( X ′ )

Original: =
Revised: =

Original: y ∗ and cannot be an adversarial example.
Revised: y ∗ and cannot be an adversarial example.

Original: Combining all such inputs, i.e. combining the orbits of elements of B (see Definition 1), yields an augmented certified region ˜ B ⊇ B (proof in Appendix E.1):
Revised: Combining all such inputs, i.e. combining the orbits of elements of B (see Definition 1), yields an augmented certified region ˜ B ⊇ B (proof in Appendix E.1):

Original: Theorem 2 (Orbit-based certificates) .
Revised: Theorem 2 (Orbit-based certificates) .

Original: Let f ∈ R N × D
Revised: Let f ∈ R N × D

Original: → Y be invariant under a group T .
Revised: → Y be invariant under a group T .

Original: Let y ∗ =
Revised: Let y ∗ =

Original: f ( X ) be a prediction that is certifiably robust to a set of perturbed inputs B ⊆ R N × D , i.e.
Revised: f ( X ) be a prediction that is certifiably robust to a set of perturbed inputs B ⊆ R N × D , i.e.

Original: ∀ Z ∈ B : f ( Z ) =
Revised: ∀ Z ∈ B : f ( Z ) =

Original: y ∗ .
Revised: y ∗ .

Original: Let ˜ B
Revised: Let ˜ B

Original: = ∪ Z ∈ B
Revised: = ∪ Z ∈ B

Original: [ Z ] T .
Revised: [ Z ] T .

Original: Then ∀ X ′ ∈ ˜ B : f ( X ′ ) =
Revised: Then ∀ X ′ ∈ ˜ B : f ( X ′ ) =

Original: y ∗ .
Revised: y ∗ .

Original: This orbit-based approach is illustrated in Fig.
Revised: This orbit-based approach is illustrated in Fig.

Original: While we focus on randomized smoothing,
Revised: While we focus on randomized smoothing,

Original: Theorem 2 holds for arbitrary models and certified regions B .
Revised: Theorem 2 holds for arbitrary models and certified regions B .

Original: However, obtaining B via other means may not be possible.
Revised: However, obtaining B via other means may not be possible.

Original: For instance, there exist no white-box certificates for rotation-invariant models.
Revised: For instance, there exist no white-box certificates for rotation-invariant models.

Original: Because randomized smoothing preserves invariance to Euclidean isometries and permutation (see Theorem 1), we may apply the orbit-based approach with T
Revised: Because randomized smoothing preserves invariance to Euclidean isometries and permutation (see Theorem 1), we may apply the orbit-based approach with T

Original: ⊆
Revised: ⊆

Original: E ( D ) or T ⊆ S ( N ) to the certified region B = { X + ∆ | || ∆ ||
Revised: E ( D ) or T ⊆ S ( N ) to the certified region B = { X + ∆ | || ∆ ||

Original: 2 < r } with r = σ Φ − 1 ( p X ,y ∗ ) that we derived in Section 3.1.
Revised: 2 < r } with r = σ Φ − 1 ( p X ,y ∗ ) that we derived in Section 3.1.

Original: While ˜ B
Revised: While ˜ B

Original: = ∪ Z ∈ B
Revised: = ∪ Z ∈ B

Original: [ Z ] T , is a valid certificate, one may desire an equivalent, but more explicit characterization of the certified region ˜ B to determine whether a specific perturbed input preserves the prediction y ∗ .
Revised: [ Z ] T , is a valid certificate, one may desire an equivalent, but more explicit characterization of the certified region ˜ B to determine whether a specific perturbed input preserves the prediction y ∗ .

Original: We discuss such explicit characterizations in Appendix E.
Revised: We discuss such explicit characterizations in Appendix E.

Original: In particular, translation invariance (i.e. T = T ( D ) ) leads to certified region ˜ B
Revised: In particular, translation invariance (i.e. T = T ( D ) ) leads to certified region ˜ B

Original: = (cid:8) X + ∆ | || ∆ − 1 N ∆ || 2 < r (cid:9) , where
Revised: = (cid:8) X + ∆ | || ∆ − 1 N ∆ || 2 < r (cid:9) , where

Original: ∆ ∈ R 1 × D are column-wise averages and r is defined as above.
Revised: ∆ ∈ R 1 × D are column-wise averages and r is defined as above.

Original: Rotation invariance (i.e. T =
Revised: Rotation invariance (i.e. T =

Original: SO ( D ) ) leads to certified region ˜ B
Revised: SO ( D ) ) leads to certified region ˜ B

Original: = (cid:8) X ′ | ||
Revised: = (cid:8) X ′ | ||

Original: X ′ R T − X
Revised: X ′ R T − X

Original: || 2 < r (cid:9) , where R is an optimal rotation matrix defined by the singular value decomposition of X T X
Revised: || 2 < r (cid:9) , where R is an optimal rotation matrix defined by the singular value decomposition of X T X

Original: ′ [111, 112] (see Appendix E.2.2).
Revised: ′ [111, 112] (see Appendix E.2.2).

Original:  Tight gray-box certificates
Revised:  Tight gray-box certificates

Original: Now that we have established a baseline for gray-box certification, one may naturally wonder about its optimality.
Revised: Now that we have established a baseline for gray-box certification, one may naturally wonder about its optimality.

Original: We answer this by deriving tight gray-box certificates, i.e. the best certificates that can be obtained for prediction y ∗ using only the invariances of base classifier g under group T and its prediction probability p X ,y ∗ under clean smoothing distribution µ X .
Revised: We answer this by deriving tight gray-box certificates, i.e. the best certificates that can be obtained for prediction y ∗ using only the invariances of base classifier g under group T and its prediction probability p X ,y ∗ under clean smoothing distribution µ X .

Original: Similar to Section 3.1, we do so by finding a worst-case classifier.
Revised: Similar to Section 3.1, we do so by finding a worst-case classifier.

Original: In addition to constraining its clean prediction probability, we constrain the classifier to be invariant under T , i.e. solve min h ∈ H T Pr Z ∼ µ X ′
Revised: In addition to constraining its clean prediction probability, we constrain the classifier to be invariant under T , i.e. solve min h ∈ H T Pr Z ∼ µ X ′

Original: [ h ( Z ) =
Revised: [ h ( Z ) =

Original: y ∗ ] with
Revised: y ∗ ] with

Original: [Equation]
Revised: [Equation]

Original: where [ Z ] T is the orbit of Z w.r.t.
Revised: where [ Z ] T is the orbit of Z w.r.t.

Original: T (see Definition 1).
Revised: T (see Definition 1).

Original: In the following, we show how to solve the above optimization problem and then apply our certification methodology to specific invariances.
Revised: In the following, we show how to solve the above optimization problem and then apply our certification methodology to specific invariances.

Original: 6.1 Certification methodology
Revised: 6.1 Certification methodology

Original: To work with invariance constraints, it is convenient to not think of h as a function, but a family of variables ( h Z ) ∈ Y indexed by R N × D .
Revised: To work with invariance constraints, it is convenient to not think of h as a function, but a family of variables ( h Z ) ∈ Y indexed by R N × D .

Original: The invariance constraint states that all variables from an orbit should have the same value, i.e. ∀ Z , ∀ Z ′ ∈
Revised: The invariance constraint states that all variables from an orbit should have the same value, i.e. ∀ Z , ∀ Z ′ ∈

Original: [ Z ] T : h Z
Revised: [ Z ] T : h Z

Original: =
Revised: =

Original: h Z ′ .
Revised: h Z ′ .

Original: Intuitively, the constraint can be enforced by replacing all these variables with a single variable.
Revised: Intuitively, the constraint can be enforced by replacing all these variables with a single variable.

Original: We propose to formalize this idea by using canonical maps, which map all inputs from an orbit to a single, distinct representative:
Revised: We propose to formalize this idea by using canonical maps, which map all inputs from an orbit to a single, distinct representative:

Original: Definition 3 (Canonical map) .
Revised: Definition 3 (Canonical map) .

Original: A canonical map for invariance under a group of transformations T is a function γ : R N × D
Revised: A canonical map for invariance under a group of transformations T is a function γ : R N × D

Original: → R N × D with
Revised: → R N × D with

Original: [Equation]
Revised: [Equation]

Original: In Appendix F.1, we prove that canonical maps let us discard the invariance constraints:
Revised: In Appendix F.1, we prove that canonical maps let us discard the invariance constraints:

Original: Lemma 1.
Revised: Lemma 1.

Original: Let g : R N × D
Revised: Let g : R N × D

Original: → Y be invariant under group T and let H T be defined as in Eq.
Revised: → Y be invariant under group T and let H T be defined as in Eq.

Original: If γ : R N × D → R N × D is a canonical map for invariance under T , then
Revised: If γ : R N × D → R N × D is a canonical map for invariance under T , then

Original: [Equation]
Revised: [Equation]

Original: Like in Section 3.1, the optimization problem without invariance constraints from Lemma 1 could now be solved exactly using the Neyman-Pearson lemma – if we could find the distribution of γ ( Z ) , i.e. the distribution of representatives 1 .
Revised: Like in Section 3.1, the optimization problem without invariance constraints from Lemma 1 could now be solved exactly using the Neyman-Pearson lemma – if we could find the distribution of γ ( Z ) , i.e. the distribution of representatives 1 .

Original: This can be achieved for groups T ( D ) , SO ( D ) and SE ( D ) via careful change of variables into an alternative parameterization of R N × D (see Appendix F.3).
Revised: This can be achieved for groups T ( D ) , SO ( D ) and SE ( D ) via careful change of variables into an alternative parameterization of R N × D (see Appendix F.3).

Original: This leads us to our main result, which we derive more formally in Appendix F.3.
Revised: This leads us to our main result, which we derive more formally in Appendix F.3.

Original: In the following, let ⟨ A , B ⟩ F be the Frobenius inner product (cid:80)
Revised: In the following, let ⟨ A , B ⟩ F be the Frobenius inner product (cid:80)

Original: Nn =1 ( A n ) T B n and recall from Section 3.3 that a right
Revised: Nn =1 ( A n ) T B n and recall from Section 3.3 that a right

Original: Haar measure is a measure for integration over a group T .
Revised: Haar measure is a measure for integration over a group T .

Original: Theorem 3.
Revised: Theorem 3.

Original: Let g : R N × D
Revised: Let g : R N × D

Original: → Y be invariant under T with T chosen from { T ( D ) , SO ( D ) , SE ( D ) } .
Revised: → Y be invariant under T with T chosen from { T ( D ) , SO ( D ) , SE ( D ) } .

Original: For SO ( D ) and SE ( D ) , let D ∈ { 2 , 3 } .
Revised: For SO ( D ) and SE ( D ) , let D ∈ { 2 , 3 } .

Original: Let H T be defined as in Eq.
Revised: Let H T be defined as in Eq.

Original: (2) and η be a right Haar measure on T .
Revised: (2) and η be a right Haar measure on T .

Original: Define the indicator function h ∗ : R N × D → { 0 , 1 } with
Revised: Define the indicator function h ∗ : R N × D → { 0 , 1 } with

Original: [Equation]
Revised: [Equation]

Original: , where β X ( Z ) = (cid:90) t ∈ T exp (cid:0) ⟨ t ◦ Z , X ⟩ F / σ 2 (cid:1) dη ( t ) (5) and κ ∈ R such that
Revised: , where β X ( Z ) = (cid:90) t ∈ T exp (cid:0) ⟨ t ◦ Z , X ⟩ F / σ 2 (cid:1) dη ( t ) (5) and κ ∈ R such that

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: The indicator function h ∗ in Eq.
Revised: The indicator function h ∗ in Eq.

Original: (5) corresponds to the worst-case invariant classifier for clean prediction probability p X ,y ∗ .
Revised: (5) corresponds to the worst-case invariant classifier for clean prediction probability p X ,y ∗ .

Original: To classify an input sample Z , it integrates Gaussian kernels of ( t ◦ Z , X ′ ) and ( t ◦ Z , X ) over group T (e.g. over all possible rotations).
Revised: To classify an input sample Z , it integrates Gaussian kernels of ( t ◦ Z , X ′ ) and ( t ◦ Z , X ) over group T (e.g. over all possible rotations).

Original: If the ratio of these integrals is below a threshold κ , it classifies Z as y ∗ .
Revised: If the ratio of these integrals is below a threshold κ , it classifies Z as y ∗ .

Original: The constraint in Eq.
Revised: The constraint in Eq.

Original: (6) ensures that the probability of predicting y ∗ under the clean smoothing distribution µ X matches that of the actual base classifier g .
Revised: (6) ensures that the probability of predicting y ∗ under the clean smoothing distribution µ X matches that of the actual base classifier g .

Original: The expected value with respect to perturbed smoothing distribution µ X ′ in Eq.
Revised: The expected value with respect to perturbed smoothing distribution µ X ′ in Eq.

Original: (7) is the optimal value of our optimization problem.
Revised: (7) is the optimal value of our optimization problem.

Original: As discussed in Section 3.1, the prediction is certifiably robust if this optimal value is greater than 1 / 2 .
Revised: As discussed in Section 3.1, the prediction is certifiably robust if this optimal value is greater than 1 / 2 .

Original: Applying the certificate to a group T requires three steps: 1.)
Revised: Applying the certificate to a group T requires three steps: 1.)

Original: Calculating the Haar integrals in Eq.
Revised: Calculating the Haar integrals in Eq.

Original: Solving Eq.
Revised: Solving Eq.

Original: (6) for classification threshold κ . 3.)
Revised: (6) for classification threshold κ . 3.)

Original: Evaluating the expected value in Eq.
Revised: Evaluating the expected value in Eq.

Original: Connection to prior work.
Revised: Connection to prior work.

Original: This result differs from black-box randomized smoothing with Gaussian noise, where the worst-case classifier is a linear model [26].
Revised: This result differs from black-box randomized smoothing with Gaussian noise, where the worst-case classifier is a linear model [26].

Original: The group averaging performed by the worst-case invariant classifier, i.e. integrating a function over group T , is a key technique for building invariant models [7, 113–117].
Revised: The group averaging performed by the worst-case invariant classifier, i.e. integrating a function over group T , is a key technique for building invariant models [7, 113–117].

Original: Group-averaged kernels have been proposed in [15].
Revised: Group-averaged kernels have been proposed in [15].

Original: It is fascinating to see them naturally materialize from nothing but an invariance constraint.
Revised: It is fascinating to see them naturally materialize from nothing but an invariance constraint.

Original: 6.2 Translation invariance
Revised: 6.2 Translation invariance

Original: In the case of translation invariance (i.e. T = T ( D ) ), we can evaluate the worst-case classifier (Eq.
Revised: In the case of translation invariance (i.e. T = T ( D ) ), we can evaluate the worst-case classifier (Eq.

Original: (5), solve for threshold κ (Eq.
Revised: (5), solve for threshold κ (Eq.

Original: and evaluate the perturbed prediction probability (Eq.
Revised: and evaluate the perturbed prediction probability (Eq.

Original: (7)) analytically.
Revised: (7)) analytically.

Original: This leads to the following result (proof in Appendix F.4.1):
Revised: This leads to the following result (proof in Appendix F.4.1):

Original: Theorem 4.
Revised: Theorem 4.

Original: Let g : R N × D
Revised: Let g : R N × D

Original: → Y be invariant under T
Revised: → Y be invariant under T

Original: = T ( D ) and H T be defined as in Eq.
Revised: = T ( D ) and H T be defined as in Eq.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where ∆ ∈ R 1 × D are the column-wise averages of ∆ =
Revised: where ∆ ∈ R 1 × D are the column-wise averages of ∆ =

Original: X ′ − X
Revised: X ′ − X

Original: and σ is the standard deviation of the isotropic matrix normal smoothing distribution µ X .
Revised: and σ is the standard deviation of the isotropic matrix normal smoothing distribution µ X .

Original: Certificate parameters.
Revised: Certificate parameters.

Original: For a fixed smoothing standard deviation σ , the certificate depends on a single parameter:
Revised: For a fixed smoothing standard deviation σ , the certificate depends on a single parameter:

Original: The norm (cid:12)(cid:12)(cid:12)(cid:12) ∆ − 1 N ∆ (cid:12)(cid:12)(cid:12)(cid:12) 2 of the mean-centered perturbation matrix ∆ .
Revised: The norm (cid:12)(cid:12)(cid:12)(cid:12) ∆ − 1 N ∆ (cid:12)(cid:12)(cid:12)(cid:12) 2 of the mean-centered perturbation matrix ∆ .

Original: Comparison to the orbit-based certificate.
Revised: Comparison to the orbit-based certificate.

Original: Substituting into robustness condition min h ∈ H T
Revised: Substituting into robustness condition min h ∈ H T

Original: Pr Z ∼ µ X ′
Revised: Pr Z ∼ µ X ′

Original: [ h ( Z ) =
Revised: [ h ( Z ) =

Original: y ∗ ] > 12 shows that f ( X ′ )
Revised: y ∗ ] > 12 shows that f ( X ′ )

Original: =
Revised: =

Original: y ∗ if (cid:12)(cid:12)(cid:12)(cid:12) ∆ − 1 N ∆ (cid:12)(cid:12)(cid:12)(cid:12) 2 <
Revised: y ∗ if (cid:12)(cid:12)(cid:12)(cid:12) ∆ − 1 N ∆ (cid:12)(cid:12)(cid:12)(cid:12) 2 <

Original: σ Φ − 1 ( p X ,y ∗ ) .
Revised: σ Φ − 1 ( p X ,y ∗ ) .

Original: This result, obtained via our tight certification methodology, is identical to the orbit-based certificate for translation invariance from the end of Section 5.
Revised: This result, obtained via our tight certification methodology, is identical to the orbit-based certificate for translation invariance from the end of Section 5.

Original: In other words: Despite its simplicity, the orbit-based certificate is the best possible gray-box certificate for translation invariance.
Revised: In other words: Despite its simplicity, the orbit-based certificate is the best possible gray-box certificate for translation invariance.

Original: 6.3 Rotation invariance in 2D
Revised: 6.3 Rotation invariance in 2D

Original: Considering the previous result, one may suspect the orbit-based certificate to be tight for arbitrary invariances.
Revised: Considering the previous result, one may suspect the orbit-based certificate to be tight for arbitrary invariances.

Original: This is not the case.
Revised: This is not the case.

Original: For instance, it is not tight for rotation invariance ( T = SO
Revised: For instance, it is not tight for rotation invariance ( T = SO

Original: ( D ) ):
Revised: ( D ) ):

Original: Theorem 5.
Revised: Theorem 5.

Original: Let g : R N × D
Revised: Let g : R N × D

Original: → Y be invariant under T =
Revised: → Y be invariant under T =

Original: SO ( D ) and H T be defined as in Eq.
Revised: SO ( D ) and H T be defined as in Eq.

Original: Assume that perturbed input X ′ is not obtained via rotation of X , i.e. ∄ R ∈
Revised: Assume that perturbed input X ′ is not obtained via rotation of X , i.e. ∄ R ∈

Original: SO ( D ) :
Revised: SO ( D ) :

Original: X ′ =
Revised: X ′ =

Original: XR T .
Revised: XR T .

Original: Further assume that p X ,y ∗ ∈ (0 , 1) .
Revised: Further assume that p X ,y ∗ ∈ (0 , 1) .

Original: Then, for all R ∈ SO ( D ) :
Revised: Then, for all R ∈ SO ( D ) :

Original: [Equation]
Revised: [Equation]

Original: In other words: The tight certificate is strictly stronger.
Revised: In other words: The tight certificate is strictly stronger.

Original: Proof in Appendix G.
Revised: Proof in Appendix G.

Original: Next, we apply Theorem 3 to obtain the strictly stronger, tight certificate for rotation invariance in 2D.
Revised: Next, we apply Theorem 3 to obtain the strictly stronger, tight certificate for rotation invariance in 2D.

Original: In the following, let R ( θ ) ∈
Revised: In the following, let R ( θ ) ∈

Original: SO (2) be the matrix that rotates counter-clockwise by angle θ and
Revised: SO (2) be the matrix that rotates counter-clockwise by angle θ and

Original: I 0 be the modified Bessel function of the first kind and order 0 .
Revised: I 0 be the modified Bessel function of the first kind and order 0 .

Original: The Haar integral from Eq.
Revised: The Haar integral from Eq.

Original: (5) that defines the worst-case classifier can be evaluated analytically (see Appendix F.4.2)):
Revised: (5) that defines the worst-case classifier can be evaluated analytically (see Appendix F.4.2)):

Original: [Equation]
Revised: [Equation]

Original: We can substitute this into Theorem 3 and use the fact that Eq.
Revised: We can substitute this into Theorem 3 and use the fact that Eq.

Original: (9) depends on linear transformations of the matrix normal random variable Z to obtain the following certificate (proof in Appendix F.4.2):
Revised: (9) depends on linear transformations of the matrix normal random variable Z to obtain the following certificate (proof in Appendix F.4.2):

Original: Theorem 6.
Revised: Theorem 6.

Original: Let g : R N × 2 → Y be invariant under T =
Revised: Let g : R N × 2 → Y be invariant under T =

Original: SO (2) and H T be defined as in Eq.
Revised: SO (2) and H T be defined as in Eq.

Original: Define the indicator function ρ : R 4 → { 0 , 1 } with
Revised: Define the indicator function ρ : R 4 → { 0 , 1 } with

Original: [Equation]
Revised: [Equation]

Original: with κ ∈ R such that
Revised: with κ ∈ R such that

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where m (1) , m (2) ∈ R 4 , Σ ∈ R 4 × 4 are linear combinations (see Appendix F.4.2) of || X
Revised: where m (1) , m (2) ∈ R 4 , Σ ∈ R 4 × 4 are linear combinations (see Appendix F.4.2) of || X

Original: || 22 / σ 2 , ||
Revised: || 22 / σ 2 , ||

Original: ∆ || 22 / σ 2 and parameters ϵ 1 = ⟨ X , ∆ ⟩ F , ϵ 2 = ⟨ XR ( − π / 2 ) T , ∆ ⟩ F .
Revised: ∆ || 22 / σ 2 and parameters ϵ 1 = ⟨ X , ∆ ⟩ F , ϵ 2 = ⟨ XR ( − π / 2 ) T , ∆ ⟩ F .

Original: Certificate parameters.
Revised: Certificate parameters.

Original: This certificate depends on the perturbation norm ||
Revised: This certificate depends on the perturbation norm ||

Original: ∆ || 2 and clean data norm ||
Revised: ∆ || 2 and clean data norm ||

Original: X
Revised: X

Original: || 2 , relative to the smoothing standard deviation σ .
Revised: || 2 , relative to the smoothing standard deviation σ .

Original: It further depends on parameters ϵ 1 and ϵ 2 , which are Frobenius inner products between ∆ and the clean input X before and after a rotation by − π / 2 .
Revised: It further depends on parameters ϵ 1 and ϵ 2 , which are Frobenius inner products between ∆ and the clean input X before and after a rotation by − π / 2 .

Original: These parameters capture the orientation of X ′ =
Revised: These parameters capture the orientation of X ′ =

Original: X + ∆ relative to X , as one would expect from a rotation-invariance aware certificate.
Revised: X + ∆ relative to X , as one would expect from a rotation-invariance aware certificate.

Original: Monte Carlo evaluation.
Revised: Monte Carlo evaluation.

Original: Evidently, we do not have a closed-form expression for the expectations in Eqs.
Revised: Evidently, we do not have a closed-form expression for the expectations in Eqs.

Original: (11) and (12).
Revised: (11) and (12).

Original: However, recall from Section 3.1 that randomized smoothing already involves an intractable expectation, namely the prediction probability p X ,y ∗ .
Revised: However, recall from Section 3.1 that randomized smoothing already involves an intractable expectation, namely the prediction probability p X ,y ∗ .

Original: One has to use Monte Carlo sampling to compute a lower confidence bound that hold with high probability 1 − α .
Revised: One has to use Monte Carlo sampling to compute a lower confidence bound that hold with high probability 1 − α .

Original: We adopt the same approach: First, we lower-bound threshold κ from Eq.
Revised: We adopt the same approach: First, we lower-bound threshold κ from Eq.

Original: (11), i.e. make the classifier slightly less likely to predict y ∗ .
Revised: (11), i.e. make the classifier slightly less likely to predict y ∗ .

Original: Then, we lower-bound bound expectation E q ∼N ( m (1) , Σ )
Revised: Then, we lower-bound bound expectation E q ∼N ( m (1) , Σ )

Original: [ ρ ( q )] from Eq.
Revised: [ ρ ( q )] from Eq.

Original: Because the resulting value is slightly smaller than than optimal value of our optimization problem
Revised: Because the resulting value is slightly smaller than the optimal value of our optimization problem

Original: min h ∈ H T
Revised: min h ∈ H T

Original: Pr Z ∼ µ X ′
Revised: Pr Z ∼ µ X ′

Original: [ h ( Z ) =
Revised: [ h ( Z ) =

Original: y ∗ ] , this procedure yields a valid certificate.
Revised: y ∗ ] , this procedure yields a valid certificate.

Original: We discuss the full algorithm and how to ensure that all bounds simultaneously hold in Appendix F.5.
Revised: We discuss the full algorithm and how to ensure that all bounds simultaneously hold in Appendix F.5.

Original: Because the expectations only require sampling from four-dimensional normal distributions and do not depend on base classifier g , one can use a large number of samples to obtain narrow bounds at little computational cost (e.g. 0 .
Revised: Because the expectations only require sampling from four-dimensional normal distributions and do not depend on base classifier g , one can use a large number of samples to obtain narrow bounds at little computational cost (e.g. 0 .

Original: 59s for 100000 samples per confidence bound on an Intel Xeon E5-2630 CPU).
Revised: 59s for 100000 samples per confidence bound on an Intel Xeon E5-2630 CPU).

Original: 6.4 Rotation invariance in 3D
Revised: 6.4 Rotation invariance in 3D

Original: To evaluate the tight certificate for 3D rotation invariance (i.e. T = SO (3) ), we adopt the same Monte
Revised: To evaluate the tight certificate for 3D rotation invariance (i.e. T = SO (3) ), we adopt the same Monte

Original: Carlo evaluation approach discussed in Section 6.3.
Revised: Carlo evaluation approach discussed in Section 6.3.

Original: Different from 2D rotation invariance, evaluating the worst-case classifier analytically is not tractable.
Revised: Different from 2D rotation invariance, evaluating the worst-case classifier analytically is not tractable.

Original: It can however be evaluated by numerical integration over an Euler angle parameterization of rotation group SO (3) (see Appendix F.4.3).
Revised: It can however be evaluated by numerical integration over an Euler angle parameterization of rotation group SO (3) (see Appendix F.4.3).

Original: 6.5 Roto-translation invariance in 2D and 3D
Revised: 6.5 Roto-translation invariance in 2D and 3D

Original: In Appendix F.4.4 we prove that additionally enforcing translation invariance (i.e. T = SE ( D ) ) is equivalent to centering X and ∆ before evaluating the certificates for rotation invariance.
Revised: In Appendix F.4.4 we prove that additionally enforcing translation invariance (i.e. T = SE ( D ) ) is equivalent to centering X and ∆ before evaluating the certificates for rotation invariance.

Original: This is consistent with our result from Section 6.2, i.e. the orbit-based approach being optimal for translation.
Revised: This is consistent with our result from Section 6.2, i.e. the orbit-based approach being optimal for translation.

Original:  Limitations and broader impact
Revised:  Limitations and broader impact

Original: The main limitation of our work lies in its exploratory nature.
Revised: The main limitation of our work lies in its exploratory nature.

Original: We have derived the first certificates for models invariant under arbitrary Euclidean isometries and/or permutation.
Revised: We have derived the first certificates for models invariant under arbitrary Euclidean isometries and/or permutation.

Original: While these invariances are of key importance to many practical applications , there is a vast swath of invariances that we have not covered, such as spatio-temporal invariances [44], invariance under graph isomorphisms [9] or invariances for planar images [7].
Revised: While these invariances are of key importance to many practical applications , there is a vast swath of invariances that we have not covered, such as spatio-temporal invariances [44], invariance under graph isomorphisms [9] or invariances for planar images [7].

Original: Furthermore, we have not yet derived tight certificates for reflections and permutations (though our certification methodology could conceivably be applied to them).
Revised: Furthermore, we have not yet derived tight certificates for reflections and permutations (though our certification methodology could conceivably be applied to them).

Original: Broader impact.
Revised: Broader impact.

Original: With the growing prevalence of machine learning in safety-critical and sensitive domains like autonomous driving [118, 119] or healthcare [120, 121], trustworthy models promise to become increasingly important.
Revised: With the growing prevalence of machine learning in safety-critical and sensitive domains like autonomous driving [118, 119] or healthcare [120, 121], trustworthy models promise to become increasingly important.

Original: Certification is one pillar of trustworthiness, alongside concepts like fairness [122, 123] or differential privacy [124, 125].
Revised: Certification is one pillar of trustworthiness, alongside concepts like fairness [122, 123] or differential privacy [124, 125].

Original: Unique to our work is that we certify models
Revised: Unique to our work is that we certify models

Original: [Figure]
Revised: [Figure]

Original: ||
Revised: ||

Original: [Figure]
Revised: [Figure]

Original: |
Revised: |

Original: [Figure]
Revised: [Figure]

Original: ||X||
Revised: ||X||

Original: [Figure]
Revised: [Figure]

Original: ||X||=
Revised: ||X||=

Original: [Figure]
Revised: [Figure]

Original: ||X||
Revised: ||X||

Original: [Figure]
Revised: [Figure]

Original: and varying || ∆ || and || X
Revised: and varying || ∆ || and || X

Original: || (smaller p min is bet- ter).
Revised: || (smaller p min is bet- ter).

Original: As || X || increases, the difference between the certificates shrinks.
Revised: As || X || increases, the difference between the certificates shrinks.

Original: [Figure]
Revised: [Figure]

Original: ˜ (cid:15)
Revised: ˜ (cid:15)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ||
Revised: ||

Original: ∆ ||
Revised: ∆ ||

Original: =1.
Revised: =1.

Original: -   ˜ (cid:15)-  1 || ∆ ||
Revised: -   ˜ (cid:15)-  1 || ∆ ||

Original: =1.
Revised: =1.

Original: 0 .0 .0 .d ( p min ) with spatial invariances.
Revised: 0 .0 .0 .d ( p min ) with spatial invariances.

Original: These invariances naturally occur in the physical sciences, including those with a direct societal impact like pharmacology and biochemistry.
Revised: These invariances naturally occur in the physical sciences, including those with a direct societal impact like pharmacology and biochemistry.

Original: Our work can be seen as a first step towards provable trustworthiness for tasks like drug discovery [39–41] or protein folding [45–49].
Revised: Our work can be seen as a first step towards provable trustworthiness for tasks like drug discovery [39–41] or protein folding [45–49].

Original:  Experimental evaluation
Revised:  Experimental evaluation

Original: We already know that the orbit-based certificate for translation invariance is tight and certifies robustness for an infinitely larger volume than black-box randomized smoothing (see Fig. 1).
Revised: We already know that the orbit-based certificate for translation invariance is tight and certifies robustness for an infinitely larger volume than black-box randomized smoothing (see Fig. 1).

Original: Therefore, we focus our experiments on the certificates for rotation invariance.
Revised: Therefore, we focus our experiments on the certificates for rotation invariance.

Original: Recall that the orbit-based certificate guarantees robustness for the set ˜ B = (cid:8) X ′ |
Revised: Recall that the orbit-based certificate guarantees robustness for the set ˜ B = (cid:8) X ′ |

Original: ||
Revised: ||

Original: X ′ R T − X
Revised: X ′ R T − X

Original: || 2 < r (cid:9) with r = σ Φ − 1 ( p X ,y ∗ ) and optimal rotation matrix R .
Revised: || 2 < r (cid:9) with r = σ Φ − 1 ( p X ,y ∗ ) and optimal rotation matrix R .

Original: In other words: It certifies robustness for perturbations with rotational components that can be eliminated to bring X ′ into distance r of
Revised: In other words: It certifies robustness for perturbations with rotational components that can be eliminated to bring X ′ into distance r of

Original: X .
Revised: X .

Original: We want to understand whether the tight certificates offer any benefit beyond that, or if the strict inequality in Theorem 5 is due to some negligible ϵ .
Revised: We want to understand whether the tight certificates offer any benefit beyond that, or if the strict inequality in Theorem 5 is due to some negligible ϵ .

Original: To this end, we first thoroughly examine the four-dimensional parameter space of the tight certificate for rotation invariance in 2D, before applying our certificates to rotation invariant point cloud classifiers.
Revised: To this end, we first thoroughly examine the four-dimensional parameter space of the tight certificate for rotation invariance in 2D, before applying our certificates to rotation invariant point cloud classifiers.

Original: All parameters and experimental details are specified in Appendix B.
Revised: All parameters and experimental details are specified in Appendix B.

Original: We use 10000 samples per confidence bound and set α = 0 .
Revised: We use 10000 samples per confidence bound and set α = 0 .

Original: 001 , i.e. all certificates hold with 99 . 9% probability.
Revised: 001 , i.e. all certificates hold with 99 . 9% probability.

Original: A reference implementation will be made available at https://www.cs.cit.tum.de/daml/invariance-smoothing.
Revised: A reference implementation will be made available at https://www.cs.cit.tum.de/daml/invariance-smoothing.

Original: 8.1 Tight certificate parameter space
Revised: 8.1 Tight certificate parameter space

Original: The tight certificate for 2D rotation invariance depends on || X
Revised: The tight certificate for 2D rotation invariance depends on || X

Original: || 2 /σ , ||
Revised: || 2 /σ , ||

Original: ∆
Revised: ∆

Original: || 2 /σ and parameters ϵ 1 and ϵ 2 , which capture the orientation of the perturbed point cloud and
Revised: || 2 /σ and parameters ϵ 1 and ϵ 2 , which capture the orientation of the perturbed point cloud and

Original: fulfill (cid:112) ϵ 

Revised: fulfill (cid:112) ϵ

Original: + ϵ 22 ≤
Revised: + ϵ 22 ≤

Original: ||
Revised: ||

Original: X
Revised: X

Original: || 2 · || ∆ || 2 (see Appendix J).
Revised: || 2 · || ∆ || 2 (see Appendix J).

Original: To avoid clutter, we define ˜ ϵ
Revised: To avoid clutter, we define ˜ ϵ

Original: k :=
Revised: k :=

Original: ϵ k / (
Revised: ϵ k / (

Original: || X || 2 · || ∆ || 2 ) .
Revised: || X || 2 · || ∆ || 2 ) .

Original: As our metric for this section, we report p min , the smallest probability p X ,y ∗ for which a prediction can still be certified
Revised: As our metric for this section, we report p min , the smallest probability p X ,y ∗ for which a prediction can still be certified

Original: Adversarial scaling.
Revised: Adversarial scaling.

Original: First, we assume that X ′ =
Revised: First, we assume that X ′ =

Original: (1 + c ) X , i.e. the input is adversarially scaled.
Revised: (1 + c ) X , i.e. the input is adversarially scaled.

Original: In this case, we have ˜ ϵ
Revised: In this case, we have ˜ ϵ

Original: = 1 and ˜ ϵ
Revised: = 1 and ˜ ϵ

Original: 2 = 0 .
Revised: 2 = 0 .

Original: We then vary || ∆ || 

Revised: We then vary || ∆ ||

Original: and || X
Revised: and || X

Original: || 

Revised: ||

Original: and evaluate our certificates.
Revised: and evaluate our certificates.

Original: Note that such attacks have no rotational component, i.e. the orbit-based certificate is identical to the black-box one.
Revised: Note that such attacks have no rotational component, i.e. the orbit-based certificate is identical to the black-box one.

Original: Fig.
Revised: Fig.

Original: 2 shows that, even in the absence of rotations, the tight certificate can yield significantly stronger guarantees.
Revised: 2 shows that, even in the absence of rotations, the tight certificate can yield significantly stronger guarantees.

Original: For σ = 0 .
Revised: For σ = 0 .

Original: 5 and || X
Revised: 5 and || X

Original: || 2 = 0 .
Revised: || 2 = 0 .

Original: 01 , the baseline can only certify robustness for a prediction with p X ,y ∗ = 0 .
Revised: 01 , the baseline can only certify robustness for a prediction with p X ,y ∗ = 0 .

Original: if || ∆ || 2 ≤ 0 . 4 .
Revised: if || ∆ || 2 ≤ 0 . 4 .

Original: The tight certificate can certify robustness up to || ∆ ||
Revised: The tight certificate can certify robustness up to || ∆ ||

Original: 2 = 0 .
Revised: 2 = 0 .

Original: However, the gap shrinks, as the norm of the clean data increases.
Revised: However, the gap shrinks as the norm of the clean data increases.

Original: Effect of data norm.
Revised: Effect of data norm.

Original: We would like to see if this is a pervasive pattern.
Revised: We would like to see if this is a pervasive pattern.

Original: To this end, we fix || ∆ || 2 , gradually increase
Revised: To this end, we fix || ∆ || 2 , gradually increase

Original: ||
Revised: ||

Original: X
Revised: X

Original: || 2 , and evaluate the tight certificate for ˜ ϵ 1 , ˜ ϵ 2 on a 100 × 100 rasterization of [0 , 1] ×
Revised: || 2 , and evaluate the tight certificate for ˜ ϵ 1 , ˜ ϵ 2 on a 100 × 100 rasterization of [0 , 1] ×

Original: We then measure the difference d ( p min ) to the black-box certificate (we will discuss
Revised: We then measure the difference d ( p min ) to the black-box certificate (we will discuss

Original: -   ˜ (cid:15)-   ˜ (cid:15)|| X
Revised: -   ˜ (cid:15)-   ˜ (cid:15)|| X

Original: || =0.
Revised: || =0.

Original: -   ˜ (cid:15)-  1 ||
Revised: -   ˜ (cid:15)-  1 ||

Original: X
Revised: X

Original: || =
Revised: || =

Original: -   ˜ (cid:15)-   ||
Revised: -   ˜ (cid:15)-   ||

Original: X
Revised: X

Original: || =
Revised: || =

Original: -   ˜ (cid:15)-  1 ||
Revised: -   ˜ (cid:15)-  1 ||

Original: X
Revised: X

Original: || =10.
Revised: || =10.

Original: .d
Revised: .d

Original: ( p min )
Revised: ( p min )

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: how these results relate to the orbit-based one shortly).
Revised: how these results relate to the orbit-based one shortly).

Original: Fig.
Revised: Fig.

Original: 4 shows that for small ||
Revised: 4 shows that for small ||

Original: X
Revised: X

Original: || 2 , the tight certificate outperforms the black-box certificate for arbitrary perturbations of norm ||
Revised: || 2 , the tight certificate outperforms the black-box certificate for arbitrary perturbations of norm ||

Original: ∆
Revised: ∆

Original: || 2 .
Revised: || 2 .

Original: But, as || X || 2 increases, the regions where it outperforms the black-box one shrink.
Revised: But, as || X || 2 increases, the regions where it outperforms the black-box one shrink.

Original: Rotational components.
Revised: Rotational components.

Original: Simple algebra (see Appendix J) shows that for any combination of || X || 2 and || ∆ || 2 , adversarial rotations, i.e. X ′ =
Revised: Simple algebra (see Appendix J) shows that for any combination of || X || 2 and || ∆ || 2 , adversarial rotations, i.e. X ′ =

Original: XR T such that || X
Revised: XR T such that || X

Original: ′ − X
Revised: ′ − X

Original: ||
Revised: ||

Original: =
Revised: =

Original: ||
Revised: ||

Original: ∆ || , correspond to two specific points with (cid:112) ˜
Revised: ∆ || , correspond to two specific points with (cid:112) ˜

Original: ϵ 

Revised: ϵ

Original: + ˜ ϵ 22 = 1 in the certificate’s parameter space.
Revised: + ˜ ϵ 22 = 1 in the certificate’s parameter space.

Original: In Fig.
Revised: In Fig.

Original: 3 we fix σ = 0 .
Revised: 3 we fix σ = 0 .

Original: 5 and || X
Revised: 5 and || X

Original: || 

Revised: ||

Original: = 1 , vary ||
Revised: = 1 , vary ||

Original: ∆ || and highlight these two points.
Revised: ∆ || and highlight these two points.

Original: We observe that the tight certificate only out- performs the black-box certificate for values of ˜ ϵ 1 , ˜ ϵ 2 that are close to adversarial rotations.
Revised: We observe that the tight certificate only out- performs the black-box certificate for values of ˜ ϵ 1 , ˜ ϵ 2 that are close to adversarial rotations.

Original: However, robustness to such attacks can also be readily certified by the orbit-based approach.
Revised: However, robustness to such attacks can also be readily certified by the orbit-based approach.

Original: Combined with our previous observations, and because the certificate only depends on norms relative to smoothing standard deviation σ ,
Revised: Combined with our previous observations, and because the certificate only depends on norms relative to smoothing standard deviation σ ,

Original: i.e. || X
Revised: i.e. || X

Original: || 2 /σ and || ∆
Revised: || 2 /σ and || ∆

Original: || 2 /σ , we expect the tight and the orbit-based certificate to perform similarly well, assuming that the smoothing standard deviation σ is small relative to ||
Revised: || 2 /σ , we expect the tight and the orbit-based certificate to perform similarly well, assuming that the smoothing standard deviation σ is small relative to ||

Original: X
Revised: X

Original: || 2 .
Revised: || 2 .

Original: 8.2 Application to point cloud classification
Revised: 8.2 Application to point cloud classification

Original: To verify whether our observations hold in practice, we apply our certificates to point cloud classification.
Revised: To verify whether our observations hold in practice, we apply our certificates to point cloud classification.

Original: We consider two datasets: 3D point cloud representations of ModelNet40 [126], which consists of CAD models from 40 different categories, and 2D point cloud representations of MNIST [127].
Revised: We consider two datasets: 3D point cloud representations of ModelNet40 [126], which consists of CAD models from 40 different categories, and 2D point cloud representations of MNIST [127].

Original: We apply the same pre-processing steps as in [6].
Revised: We apply the same pre-processing steps as in [6].

Original: Certification is performed on the default test sets.
Revised: Certification is performed on the default test sets.

Original: As our base classifiers, we use rotation and translation (i.e. SE ( D ) ) invariant versions of two well-established models that are used in prior work on robustness certification for point clouds:
Revised: As our base classifiers, we use rotation and translation (i.e. SE ( D ) ) invariant versions of two well-established models that are used in prior work on robustness certification for point clouds:

Original: PointNet [6] and DGCNN [128].
Revised: PointNet [6] and DGCNN [128].

Original: To implement the invariances, we center the input data, perform principal component analysis, apply the model to all possible poses (see discussion in [129, 130]) and average the output logits (EnsPointNet and EnsDGCNN).
Revised: To implement the invariances, we center the input data, perform principal component analysis, apply the model to all possible poses (see discussion in [129, 130]) and average the output logits (EnsPointNet and EnsDGCNN).

Original: In addition, we consider a more refined model [129] that combines canonical poses via a self-attention mechanism (AttnPointNet).
Revised: In addition, we consider a more refined model [129] that combines canonical poses via a self-attention mechanism (AttnPointNet).

Original: Throughout this section, we report both probabilistic upper and lower bounds for the Monte Carlo evaluation of the tight certificates (recall Section 6.3).
Revised: Throughout this section, we report both probabilistic upper and lower bounds for the Monte Carlo evaluation of the tight certificates (recall Section 6.3).

Original: θ / deg 0 .0 .0 .0 .0 .1 .
Revised: θ / deg 0 .0 .0 .0 .0 .1 .

Original: P r .
Revised: P r .

Original: C e r t .
Revised: C e r t .

Original: A cc .
Revised: A cc .

Original: SE (2) Tight
Revised: SE (2) Tight

Original: SE (2) Orbit
Revised: SE (2) Orbit

Original: Black-box (a) EnsPointNet on MNIST      θ / deg 0 .0 .0 .

Revised: Black-box (a) EnsPointNet on MNIST      θ / deg 0 .0 .0 .

Original: P r .
Revised: P r .

Original: C e r t .
Revised: C e r t .

Original: A cc .
Revised: A cc .

Original: SE (3) Tight
Revised: SE (3) Tight

Original: SE (3) Orbit
Revised: SE (3) Orbit

Original: Black-box (b) EnsDGCNN on ModelNet
Revised: Black-box (b) EnsDGCNN on ModelNet

Original: Practical smoothing parameters.
Revised: Practical smoothing parameters.

Original: Fig.
Revised: Fig.

Original: 5 shows the test set accuracies of randomly smoothed models under varying standard deviations σ .
Revised: 5 shows the test set accuracies of randomly smoothed models under varying standard deviations σ .

Original: Values of σ that preserve an accuracy above 50% are small, relative to the average norm of the test sets ( 10 .
Revised: Values of σ that preserve an accuracy above 50% are small, relative to the average norm of the test sets ( 10 .

Original: 67 for MNIST, 19 .
Revised: 67 for MNIST, 19 .

Original: 17 for ModelNet40).
Revised: 17 for ModelNet40).

Original: Going by our previous results, we expect the tight and orbit-based certificates to perform similarly well.
Revised: Going by our previous results, we expect the tight and orbit-based certificates to perform similarly well.

Original: Adversarial scaling.
Revised: Adversarial scaling.

Original: In Fig.
Revised: In Fig.

Original: 6 we again consider adversarial scaling, i.e. attacks without rotational components, but applied to the MNIST point cloud dataset with σ = 0 .
Revised: 6 we again consider adversarial scaling, i.e. attacks without rotational components, but applied to the MNIST point cloud dataset with σ = 0 .

Original: We report the certified accuracy, i.e. the percentage of correct and provably robust predictions, for certification with ( SE (2) ) and without ( SO (2) ) translation invariance.
Revised: We report the certified accuracy, i.e. the percentage of correct and provably robust predictions, for certification with ( SE (2) ) and without ( SO (2) ) translation invariance.

Original: The tight and orbit-based certificate yield similar results.
Revised: The tight and orbit-based certificate yield similar results.

Original: We further observe that enforcing translation invariance increases the certified accuracy and extends the range of || ∆ || for which robustness can be certified.
Revised: We further observe that enforcing translation invariance increases the certified accuracy and extends the range of || ∆ || for which robustness can be certified.

Original: Rotational components.
Revised: Rotational components.

Original: Finally, we study perturbations with rotational components.
Revised: Finally, we study perturbations with rotational components.

Original: We fix || ∆ || , randomly sample perturbations of the specified norm and then rotate Z = X + ∆ by a specified angle θ ∈ [0 , 10 ◦ ] (in the case of ModelNet40, around one randomly chosen axis).
Revised: We fix || ∆ || , randomly sample perturbations of the specified norm and then rotate Z = X + ∆ by a specified angle θ ∈ [0 , 10 ◦ ] (in the case of ModelNet40, around one randomly chosen axis).

Original: For each element of the test set and each θ , we generate 10 such samples.
Revised: For each element of the test set and each θ , we generate 10 such samples.

Original: We then compute the percentage of samples X ′ for which f ( X ) is correct and f ( X ′ )
Revised: We then compute the percentage of samples X ′ for which f ( X ) is correct and f ( X ′ )

Original: =
Revised: =

Original: f ( X ) is provably guaranteed (“probabilistic certified accuracy”).
Revised: f ( X ) is provably guaranteed (“probabilistic certified accuracy”).

Original: Fig.
Revised: Fig.

Original: 7 shows results for MNIST and ModelNet40 evaluated with || ∆ ||
Revised: 7 shows results for MNIST and ModelNet40 evaluated with || ∆ ||

Original: = σ = 0 .
Revised: = σ = 0 .

Original: The black-box baseline’s probabilistic certified accuracy drops close to 0 for θ = 2 ◦ .
Revised: The black-box baseline’s probabilistic certified accuracy drops close to 0 for θ = 2 ◦ .

Original: The gray-box certificates are almost constant in θ , i.e. effectively eliminate any induced rotation.
Revised: The gray-box certificates are almost constant in θ , i.e. effectively eliminate any induced rotation.

Original: However, the tight certificate did not offer any meaningful benefit beyond that.
Revised: However, the tight certificate did not offer any meaningful benefit beyond that.

Original: Using the lower bound for Monte Carlo evaluation, there was not a single sample for which only the tight certificate could guarantee robustness.
Revised: Using the lower bound for Monte Carlo evaluation, there was not a single sample for which only the tight certificate could guarantee robustness.

Original: In Appendix A we repeat the experiments from this and the previous section for various other combinations of parameter values.
Revised: In Appendix A we repeat the experiments from this and the previous section for various other combinations of parameter values.

Original: All results are consistent with the ones presented here, confirming that the orbit-based approach offers a good approximation of the tight certificates in practice.
Revised: All results are consistent with the ones presented here, confirming that the orbit-based approach offers a good approximation of the tight certificates in practice.

Original:  Conclusion
Revised:  Conclusion

Original: For the first time, we have studied the use of invariances for robustness certification.
Revised: For the first time, we have studied the use of invariances for robustness certification.

Original: We proposed a gray-box approach, combining white-box knowledge about invariances with black-box randomized smoothing.
Revised: We proposed a gray-box approach, combining white-box knowledge about invariances with black-box randomized smoothing.

Original: We have derived a orbit-based procedure for certification that can be applied to arbitrary models with invariance to permutations and Euclidean isometries.
Revised: We have derived a orbit-based procedure for certification that can be applied to arbitrary models with invariance to permutations and Euclidean isometries.

Original: We have proven that the orbit-based certificate for translation invariance is tight and derived strictly stronger certificates for rotation invariance.
Revised: We have proven that the orbit-based certificate for translation invariance is tight and derived strictly stronger certificates for rotation invariance.

Original: Our experiments are to be interpreted in two ways: Firstly, the fact that it is possible to derive tight invariance-aware certificates and that there exist scenarios in which they offer stronger guarantees for arbitrary perturbations should be an exciting inspiration for future work.
Revised: Our experiments are to be interpreted in two ways: Firstly, the fact that it is possible to derive tight invariance-aware certificates and that there exist scenarios in which they offer stronger guarantees for arbitrary perturbations should be an exciting inspiration for future work.

Original: Secondly, the fact that the orbit-based certificates are easily interpretable and offer good approximations of our tight certificates should invite their application to real-world tasks with inherent invariances.
Revised: Secondly, the fact that the orbit-based certificates are easily interpretable and offer good approximations of our tight certificates should invite their application to real-world tasks with inherent invariances.

Original: Acknowledgements.
Revised: Acknowledgements.

Original: The authors would like to thank Johannes Gasteiger for valuable discussions on invariant deep learning and Lukas Gosch for constructive criticism of the manuscript.
Revised: The authors would like to thank Johannes Gasteiger for valuable discussions on invariant deep learning and Lukas Gosch for constructive criticism of the manuscript.

Original: This research was supported by the German Research Foundation, grant GU 1409/4-1.
Revised: This research was supported by the German Research Foundation, grant GU 1409/4-1.