Original: Double Bubble, Toil and Trouble: Enhancing Certified Robustness through Transitivity
Revised: Double Bubble, Toil and Trouble: Enhancing Certified Robustness through Transitivity

Original: Abstract
Revised: Abstract

Original: In response to subtle adversarial examples flipping classifications of neural network models, recent research has promoted certified robustness as a solution.
Revised: In response to subtle adversarial examples flipping classifications of neural network models, recent research has promoted certified robustness as a solution.

Original: There, invariance of predictions to all norm-bounded attacks is achieved through randomised smoothing of network inputs.
Revised: There, invariance of predictions to all norm-bounded attacks is achieved through randomised smoothing of network inputs.

Original: Today’s state-of-the-art certifications make optimal use of the class output scores at the input instance under test: no better radius of certification (under the L 2 norm) is possible given only these score.
Revised: Today’s state-of-the-art certifications make optimal use of the class output scores at the input instance under test: no better radius of certification (under the L 2 norm) is possible given only these score.

Original: However, it is an open question as to whether such lower bounds can be improved using local information around the instance under test.
Revised: However, it is an open question as to whether such lower bounds can be improved using local information around the instance under test.

Original: In this work, we demonstrate how today’s “optimal” certificates can be improved by exploiting both the transitivity of certifications, and the geometry of the input space, giving rise to what we term
Revised: In this work, we demonstrate how today’s “optimal” certificates can be improved by exploiting both the transitivity of certifications, and the geometry of the input space, giving rise to what we term

Original: Geometrically-Informed Certified Robustness .
Revised: Geometrically-Informed Certified Robustness .

Original: By considering the smallest distance to points on the boundary of a set of certifications this approach improves certifications for more than 80% of Tiny-Imagenet instances, yielding an on average 5% increase in the associated certification.
Revised: By considering the smallest distance to points on the boundary of a set of certifications this approach improves certifications for more than 80% of Tiny-Imagenet instances, yielding an on average 5% increase in the associated certification.

Original: When incorporating training time processes that enhance the certified radius, our technique shows even more promising results, with a uniform 4 percentage point increase in the achieved certified radius.
Revised: When incorporating training time processes that enhance the certified radius, our technique shows even more promising results, with a uniform 4 percentage point increase in the achieved certified radius.

Original:  Introduction
Revised:  Introduction

Original: Learned models, including neural networks, are well known to be susceptible having the output changed by crafted perturbations to an input, that preserve the inputs semantic properties [2].
Revised: Learned models, including neural networks, are well known to be susceptible having the output changed by crafted perturbations to an input, that preserve the inputs semantic properties [2].

Original: Neural networks not only misclassify these perturbations—known as adversarial examples —but they also assign high confidence to these incorrect predictions.
Revised: Neural networks not only misclassify these perturbations—known as adversarial examples —but they also assign high confidence to these incorrect predictions.

Original: These behaviours have been observed across a wide range of models and datasets and appear to be a product of piecewise-linear interactions [13].
Revised: These behaviours have been observed across a wide range of models and datasets and appear to be a product of piecewise-linear interactions [13].

Original: Crafting these adversarial examples typically involves gradient-based optimisation to construct small perturbations.
Revised: Crafting these adversarial examples typically involves gradient-based optimisation to construct small perturbations.

Original: These attacks have been applied to both black- and white-box models [31], and can be used to target class changes, to attack all classes [10], or even introduce backdoors into model behaviour [5].
Revised: These attacks have been applied to both black- and white-box models [31], and can be used to target class changes, to attack all classes [10], or even introduce backdoors into model behaviour [5].

Original: To mitigate the influence of these attacks, defences have typically been designed to minimise the effect of a specific attack (or attacks).
Revised: To mitigate the influence of these attacks, defences have typically been designed to minimise the effect of a specific attack (or attacks).

Original: Such defences are known as best response strategies in a Stackelberg security game where the defender leads the attacker.
Revised: Such defences are known as best response strategies in a Stackelberg security game where the defender leads the attacker.

Original: Best response defences inherently favour the attacker, as deployed mitigations can be defeated by identifying undefended attack frameworks.
Revised: Best response defences inherently favour the attacker, as deployed mitigations can be defeated by identifying undefended attack frameworks.

Original: Moreover, the defender typically has to incorporate the defence at training time, and as such cannot response reactively to newly developed attacks.
Revised: Moreover, the defender typically has to incorporate the defence at training time, and as such cannot response reactively to newly developed attacks.

Original: To circumvent these limitations, certified guarantees of adversarial robustness can be constructed to identify class-constant regions around an input instance, that guarantee that all instances within a normbounded distance (typically L 2 ) are not adversarial examples.
Revised: To circumvent these limitations, certified guarantees of adversarial robustness can be constructed to identify class-constant regions around an input instance, that guarantee that all instances within a normbounded distance (typically L 2 ) are not adversarial examples.

Original: Certifications based on randomised smoothing of classifiers around an input point are in a sense optimal [8]: based only on the prediction class scores at the input point, no better radius is in general possible.
Revised: Certifications based on randomised smoothing of classifiers around an input point are in a sense optimal [8]: based only on the prediction class scores at the input point, no better radius is in general possible.

Original: Despite this, such certifications fail to use readily available—yet still local—information: the certifiability of points nearby to the input of interest.
Revised: Despite this, such certifications fail to use readily available—yet still local—information: the certifiability of points nearby to the input of interest.

Original: The key insight of this work is that these neighbourhood points may generate certified radius large enough to completely enclose that of a sample point, improving the radius of certification.
Revised: The key insight of this work is that these neighbourhood points may generate certified radius large enough to completely enclose that of a sample point, improving the radius of certification.

Original: This process can be extended to use the intersection of the regions of certification of multiple points, and the nature of the input domain itself to generate larger certifications.
Revised: This process can be extended to use the intersection of the regions of certification of multiple points, and the nature of the input domain itself to generate larger certifications.

Original: This leads to our main contribution— Geometrically-Informed Certified Robustness —that enjoys certifications exceeding those of the hitherto best-case guaranteed approach of Cohen et al.
Revised: This leads to our main contribution— Geometrically-Informed Certified Robustness —that produces certifications exceeding those of the hitherto best-case guaranteed approach of Cohen et al.

Original:  Background and literature review
Revised:  Background and literature review

Original: Bounding mechanisms
Revised: Bounding mechanisms

Original: Conservative bounds upon the impact of norm-bounded perturbations can be constructed by way of either Interval Bound Propagation (IBP) which propagates interval bounds through the model; or Convex Relaxation, which utilise linear relaxation to construct bounding output polytopes over input bounded perturbations [34, 26, 41, 45, 46, 37, 28], in a manner that generally provides tighter bounds than IBP [25].
Revised: Conservative bounds upon the impact of norm-bounded perturbations can be constructed by way of either Interval Bound Propagation (IBP) which propagates interval bounds through the model; or Convex Relaxation, which utilise linear relaxation to construct bounding output polytopes over input bounded perturbations [34, 26, 41, 45, 46, 37, 28], in a manner that generally provides tighter bounds than IBP [25].

Original: In contrast to Randomised Smoothing, bounding mechanisms employ augmented loss functions during training, which promote tight output bounds [42] at the cost of decreased applicability.
Revised: In contrast to Randomised Smoothing, bounding mechanisms employ augmented loss functions during training, which promote tight output bounds [42] at the cost of decreased applicability.

Original: Moreover they both exhibit a time and memory complexity that makes them infeasible for complex model architectures or high-dimensional data [40, 6, 21].
Revised: Moreover they both exhibit a time and memory complexity that makes them infeasible for complex model architectures or high-dimensional data [40, 6, 21].

Original: Randomised smoothing Outside of bounding mechanisms, another common framework for developing certifications leverages randomised smoothing [20], in which noise is applied to input instances to smooth model predictions, subject to a sampling distribution that is tied to the L P -norm of adversarial perturbations being certified against.
Revised: Randomised smoothing Outside of bounding mechanisms, another common framework for developing certifications leverages randomised smoothing [20], in which noise is applied to input instances to smooth model predictions, subject to a sampling distribution that is tied to the L P -norm of adversarial perturbations being certified against.

Original: In contrast to other robustness mechanisms, this application of the noise is the only architectural change that is required to achieve certification.
Revised: In contrast to other robustness mechanisms, this application of the noise is the only architectural change that is required to achieve certification.

Original: In the case of L 2 -norm bounded attacks, Gaussian sampling of the form
Revised: In the case of L 2 -norm bounded attacks, Gaussian sampling of the form

Original: [Equation]
Revised: [Equation]

Original: where y i i.i
Revised: where y i i.i

Original: [Equation]
Revised: [Equation]

Original: is employed for all test-time instances.
Revised: is employed for all test-time instances.

Original: These N samples are then used to estimate the expected output of the predicted class of x by way of the Monte-Carlo estimator
Revised: These N samples are then used to estimate the expected output of the predicted class of x by way of the Monte-Carlo estimator

Original: [Equation]
Revised: [Equation]

Original: While this Monte Carlo estimation of output expectations under randomised smoothing is a test-time process, model sensitivity to random perturbations may be decreased by performing adversarial training on such random perturbations.
Revised: While this Monte Carlo estimation of output expectations under randomised smoothing is a test-time process, model sensitivity to random perturbations may be decreased by performing adversarial training on such random perturbations.

Original: To mitigate the computational expense of large N sample sizes during each training update, training typically employs single draws from the noise distribution.
Revised: To mitigate the computational expense of large N sample sizes during each training update, training typically employs single draws from the noise distribution.

Original: Smoothing-based certifications Based on randomised smoothing, certified robustness can guarantee classification invariance for additive perturbations up to some L p -norm r , with recent work also considering rotational and/or translational semantic attacks [23, 7].
Revised: Smoothing-based certifications Based on randomised smoothing, certified robustness can guarantee classification invariance for additive perturbations up to some L p -norm r , with recent work also considering rotational and/or translational semantic attacks [23, 7].

Original: L p -norm certifications were first demonstrated by way of differential privacy [20, 11], with more recent approaches employing Rényi divergence [22], and parametrising worst-case behaviours [8, 33].
Revised: L p -norm certifications were first demonstrated by way of differential privacy [20, 11], with more recent approaches employing Rényi divergence [22], and parametrising worst-case behaviours [8, 33].

Original: By considering the worst-case
Revised: By considering the worst-case

Original: L 2 -perturbations, Cohen et al.
Revised: L 2 -perturbations, Cohen et al.

Original: (2019) purports that the largest achievable pointwise certification is
Revised: (2019) purports that the largest achievable pointwise certification is

Original: [Equation]
Revised: [Equation]

Original: Hhere { E 0 , E 1 } are the two largest class expectations (as per Equation (2)), σ is the noise, and Φ −is the inverse normal CDF, or Gaussian quantile function.
Revised: Here { E 0 , E 1 } are the two largest class expectations (as per Equation (2)), σ is the noise, and Φ − 1 is the inverse normal CDF, or Gaussian quantile function.

Original:  Geometrically-informed certified robustness
Revised:  Geometrically-informed certified robustness

Original: While the work contained within this paper can be applied generally, for this work we will focus upon certifications of robustness about L 2 -norm bounded adversarial perturbations, for which we assume that the difficulty of attacking a model is proportional to the size of the certification, based upon the need to evade both human and machine scrutiny [12].
Revised: While the work contained within this paper can be applied generally, for this work we will focus upon certifications of robustness about L 2 -norm bounded adversarial perturbations, for which we assume that the difficulty of attacking a model is proportional to the size of the certification, based upon the need to evade both human and machine scrutiny [12].

Original: Thus, constructing larger certifications in such a context is inherently valuable.
Revised: Thus, constructing larger certifications in such a context is inherently valuable.

Original: [Figure]
Revised: [Figure]

Original: This specific L P space is of interest due to both its viability as a defence model, and the provable guarantee that Cohen et al.
Revised: This specific L P space is of interest due to both its viability as a defence model, and the provable guarantee that Cohen et al.

Original: produces the largest possible certification for any instance [8].
Revised: produces the largest possible certification for any instance [8].

Original: Over the remainder of this section we will document how it is possible to improve upon this provably best-case guarantee by exploiting several properties of certified robustness.
Revised: Over the remainder of this section we will document how it is possible to improve upon this provably best-case guarantee by exploiting several properties of certified robustness.

Original: 3.1 Exploiting transitivity
Revised: 3.1 Exploiting transitivity

Original: While it is provably true that Equation (3) is the largest achievable certification for any point x , it is possible to exploit the behaviour of points in the neighbourhood of x in order to enhance certifiable radius.
Revised: While it is provably true that Equation (3) is the largest achievable certification for any point x , it is possible to exploit the behaviour of points in the neighbourhood of x in order to enhance certifiable radius.

Original: To achieve this, consider the case of a second point x ′ , that exists within the certifiable radius of x .
Revised: To achieve this, consider the case of a second point x ′ , that exists within the certifiable radius of x .

Original: As both points must correspond to the same class, it then follows that the union of their regions of certification can be also be considered as a region of certification, leading to Definition 3.1.
Revised: As both points must correspond to the same class, it then follows that the union of their regions of certification can be also be considered as a region of certification, leading to Definition 3.1.

Original: Definition 3.1 ( Overlap Properties of Certification ) .
Revised: Definition 3.1 ( Overlap Properties of Certification ) .

Original: A radius of certification r i about x
Revised: A radius of certification r i about x

Original: i can be calculated by evaluating Equation 3 at x i .
Revised: i can be calculated by evaluating Equation 3 at x i .

Original: This certification guarantees that no point x : ∥ x − x i ∥ P ≤ r i can induce a chance in the predicted class.
Revised: This certification guarantees that no point x : ∥ x − x i ∥ P ≤ r i can induce a chance in the predicted class.

Original: That this shape is a d -dimensional hypersphere for input data x ∈ R
Revised: That this shape is a d -dimensional hypersphere for input data x ∈ R

Original: d allows us to introduce the notational shorthand
Revised: d allows us to introduce the notational shorthand

Original: [Equation]
Revised: [Equation]

Original: to represent the region covered by the hypersphere and its surface.
Revised: to represent the region covered by the hypersphere and its surface.

Original: It follows from this definition that that if B P ( x 1 , r 1 ) ∩ B P ( x 2 , r 2 ) ̸
Revised: It follows from this definition that that if B P ( x 1 , r 1 ) ∩ B P ( x 2 , r 2 ) ̸

Original: = ∅ , which ensures that the class predictions at x 1 and x 2 match, then the region of certification about x 1 can be expressed as B P ( x 1 , r 1 ) ∪ B P ( x 2 , r 2 ) .
Revised: = ∅ , which ensures that the class predictions at x 1 and x 2 match, then the region of certification about x 1 can be expressed as B P ( x 1 , r 1 ) ∪ B P ( x 2 , r 2 ) .

Original: However typically we are concerned not with the size of the region of classification invariance, but rather the distance to the nearest adversarial example.
Revised: However typically we are concerned not with the size of the region of classification invariance, but rather the distance to the nearest adversarial example.

Original: If it is possible to find some x ′ such that its region of certification completely encircles that of the certification at x , the following definition demonstrates that the certification radius about x can be increased.
Revised: If it is possible to find some x ′ such that its region of certification completely encircles that of the certification at x , the following definition demonstrates that the certification radius about x can be increased.

Original: Lemma 3.2 ( Set Unions Certified Radius ) .
Revised: Lemma 3.2 ( Set Unions Certified Radius ) .

Original: If x 1 and x 2 have the same class associated with them and B P ( x 1 , r 1 ) ⊂ B P ( x 2 , r 2 ) , then the nearest possible adversarial example—and thus, the certifiable radius—exists at a distance r ′ ≥ r from x 1 , where
Revised: If x 1 and x 2 have the same class associated with them and B P ( x 1 , r 1 ) ⊂ B P ( x 2 , r 2 ) , then the nearest possible adversarial example—and thus, the certifiable radius—exists at a distance r ′ ≥ r from x 1 , where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The closest point on the surface of B P ( x 2 , r 2 ) to x 1 must exist on the vector between x 1 and x 2 .
Revised: The closest point on the surface of B P ( x 2 , r 2 ) to x 1 must exist on the vector between x 1 and x 2 .

Original: Thus r ′ =
Revised: Thus r ′ =

Original: min p r 2 ± ∥ x 2 − x 1 ∥ P q , which takes the form of Equation (5).
Revised: min p r 2 ± ∥ x 2 − x 1 ∥ P q , which takes the form of Equation (5).

Original: As such, we can recast the task of constructing a certification from being a strictly analytic function to the nonlinear optimisation problem in terms of a second ball with certified radius r 2 centred at x
Revised: As such, we can recast the task of constructing a certification from being a strictly analytic function to the nonlinear optimisation problem in terms of a second ball with certified radius r 2 centred at x

Original: [Equation]
Revised: [Equation]

Original: (6) with Figure 1a providing a two-dimensional exemplar.
Revised: (6) with Figure 1a providing a two-dimensional exemplar.

Original: Crucially, the above formalism does not require obtaining a global optima, as any r ′ > r 1 yields an improved certification at x 1 .
Revised: Crucially, the above formalism does not require obtaining a global optima, as any r ′ > r 1 yields an improved certification at x 1 .

Original: [Equation]
Revised: [Equation]

Original: 3.2 Multiple transitivity
Revised: 3.2 Multiple transitivity

Original: To further enhance our ability to certify, let us consider the set of points and their associated certifications { x 1 , r 1 , x 2 , r 2 , . . .
Revised: To further enhance our ability to certify, let us consider the set of points and their associated certifications { x 1 , r 1 , x 2 , r 2 , . . .

Original: x n , r n } .
Revised: x n , r n } .

Original: If the union of ˆ B P = ∪ i ∈{ 1 ,...,n } B P ( x i , r i ) is simply-connected, then the certification across this set can be expressed as r ( n ) ′ =
Revised: If the union of ˆ B P = ∪ i ∈{ 1 ,...,n } B P ( x i , r i ) is simply-connected, then the certification across this set can be expressed as r ( n ) ′ =

Original: min x ∈ ∂ ˆ B P ∥ x − x 1 ∥ P , where ∂ ˆ B P is the boundary of ˆ B P .
Revised: min x ∈ ∂ ˆ B P ∥ x − x 1 ∥ P , where ∂ ˆ B P is the boundary of ˆ B P .

Original: This can be further simplified by imposing that x j ∈ B P ( x 2 , r 2 ) ∀ j > 2 and that B P ( x j , r j ) ̸⊂ B P ( x k , r k ) ∀ ( j > 2 , k > 2) to ensure that hyperspheres exist near the boundary of S 2 and yielding a certification of
Revised: This can be further simplified by imposing that x j ∈ B P ( x 2 , r 2 ) ∀ j > 2 and that B P ( x j , r j ) ̸⊂ B P ( x k , r k ) ∀ ( j > 2 , k > 2) to ensure that hyperspheres exist near the boundary of S 2 and yielding a certification of

Original: [Equation]
Revised: [Equation]

Original: Here S n is a ( d − 1) -dimensional manifold embedded in R d .
Revised: Here S n is a ( d − 1) -dimensional manifold embedded in R d .

Original: Lemma 3.3 ( Optimal positioning of x 3 in the case of n = 3 ) .
Revised: Lemma 3.3 ( Optimal positioning of x 3 in the case of n = 3 ) .

Original: Consider the addition of a new hypersphere at some point x 3 with associated radius r 3 , which has an associated boundary S 3 .
Revised: Consider the addition of a new hypersphere at some point x 3 with associated radius r 3 , which has an associated boundary S 3 .

Original: If it is true that
Revised: If it is true that

Original: [Equation]
Revised: [Equation]

Original: then the largest possible certification r ′′ by Equation 7 is achieved at
Revised: then the largest possible certification r ′′ by Equation 7 is achieved at

Original: [Equation]
Revised: [Equation]

Original: for some s ∈ [
Revised: for some s ∈ [

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The closest point to x 1 upon S 2 is located at
Revised: The closest point to x 1 upon S 2 is located at

Original: [Equation]
Revised: [Equation]

Original: where r ′ is defined by Equation 6.
Revised: where r ′ is defined by Equation 6.

Original: Thus any improved radius of certification r ′′ > r ′ is only achievable if B P ( x 3 , r 3 ) satisfies ˘ x ∈ B P ( x 3 , r 3 ) and Equation 8.
Revised: Thus any improved radius of certification r ′′ > r ′ is only achievable if B P ( x 3 , r 3 ) satisfies ˘ x ∈ B P ( x 3 , r 3 ) and Equation 8.

Original: Then by symmetry, r ′′ is the maximally achievable radius of certification if Equation 9 hold and if x 3 is defined by Equation 10.
Revised: Then by symmetry, r ′′ is the maximally achievable radius of certification if Equation 9 hold and if x 3 is defined by Equation 10.

Original: While finding some x 3 satisfying Equations 8 and 10 is trivial, proving Equation 9 would require an exhaustive search of the input space [0 , 1] d .
Revised: While finding some x 3 satisfying Equations 8 and 10 is trivial, proving Equation 9 would require an exhaustive search of the input space [0 , 1] d .

Original: However, even in the absence of such a search,
Revised: However, even in the absence of such a search,

Original: Equation 10 still provides the framework for a simple search for x 3 , which follows Figure 1b.
Revised: Equation 10 still provides the framework for a simple search for x 3 , which follows Figure 1b.

Original: Lemma 3.4 ( Certification from two eccentric hyperspheres ) .
Revised: Lemma 3.4 ( Certification from two eccentric hyperspheres ) .

Original: If x 3 is defined by Equation 10 in a fashion that satisfies Equation 8 then an updated certification can be achieved in terms of some x 3 ( s ) defined by Equation 10 by way of
Revised: If x 3 is defined by Equation 10 in a fashion that satisfies Equation 8 then an updated certification can be achieved in terms of some x 3 ( s ) defined by Equation 10 by way of

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: If Equation 9 holds, then this is the largest achievable certification for n = 3 .
Revised: If Equation 9 holds, then this is the largest achievable certification for n = 3 .

Original: Proof.
Revised: Proof.

Original: By symmetry we can define the arbitrary rotational mapping f : R d → R d from x i (cid:55)→ y
Revised: By symmetry we can define the arbitrary rotational mapping f : R d → R d from x i (cid:55)→ y

Original: i by way of y i =
Revised: i by way of y i =

Original: f ( x i − x 1 ) , subject to the condition y i,j ̸ = 0 for i ∈ { 1 , 2 } and ∀ j ̸ = k , for some k such that j, k ∈ { 1 , 2 , . . .
Revised: f ( x i − x 1 ) , subject to the condition y i,j ̸ = 0 for i ∈ { 1 , 2 } and ∀ j ̸ = k , for some k such that j, k ∈ { 1 , 2 , . . .

Original: , d } (13) then the intersection of the hyperspheres centred about x 2 and x 3 occurs at
Revised: , d } (13) then the intersection of the hyperspheres centred about x 2 and x 3 occurs at

Original: [Equation]
Revised: [Equation]

Original: This is a consequence of our mapping y =
Revised: This is a consequence of our mapping y =

Original: f
Revised: f

Original: ( x − x 1 ) preserving distances under rotation, giving that y 2 ,k = ∥ x 2 − x 1 ∥
Revised: ( x − x 1 ) preserving distances under rotation, giving that y 2 ,k = ∥ x 2 − x 1 ∥

Original: = d 2 , and with the equivalent also holding for y 3 ,k .
Revised: = d 2 , and with the equivalent also holding for y 3 ,k .

Original: As a consequence of our choice of coordinate system, it follows that r ′′ = ∥
Revised: As a consequence of our choice of coordinate system, it follows that r ′′ = ∥

Original: y ∥ and
Revised: y ∥ and

Original: [Equation]
Revised: [Equation]

Original: which is an equivalence to Equation (12).
Revised: which is an equivalence to Equation (12).

Original: While Equation 7 holds for any n ≥ 2 , the certification radius beyond n = 3 cannot be enhanced by adding any one single additional hypersphere without contradicting Lemma 3.4.
Revised: While Equation 7 holds for any n ≥ 2 , the certification radius beyond n = 3 cannot be enhanced by adding any one single additional hypersphere without contradicting Lemma 3.4.

Original: This is a result of
Revised: This is a result of

Original: S 3 being a ( d − 1) -dimensional manifold in R d , the entirety of which must be enclosed to improve the certification.
Revised: S 3 being a ( d − 1) -dimensional manifold in R d , the entirety of which must be enclosed to improve the certification.

Original: An example of this can be seen with the two equidistant intersections between S(in Red) and S 3 (in Black) in Figure 1b.
Revised: An example of this can be seen with the two equidistant intersections between S(in Red) and S 3 (in Black) in Figure 1b.

Original: While multiple spheres could be constructed to completely enclose S 3 , the number required grows exponentially with d due to the sphere packing kissing number problem [9].
Revised: While multiple spheres could be constructed to completely enclose S 3 , the number required grows exponentially with d due to the sphere packing kissing number problem [9].

Original: This growth in complexity makes adding additional spheres beyond n = 3 infeasible.
Revised: This growth in complexity makes adding additional spheres beyond n = 3 infeasible.

Original: Further details of this are contained within Appendix A.2.
Revised: Further details of this are contained within Appendix A.2.

Original: 3.3 Boundary treatments
Revised: 3.3 Boundary treatments

Original: Without loss of performance or accuracy, we can freely scale the inputs of neural networks such that x 1 ∈ [0 , 1] d .
Revised: Without loss of performance or accuracy, we can freely scale the inputs of neural networks such that x 1 ∈ [0 , 1] d .

Original: However in the majority of cases a subset of B P ( x 1 , L 1 ) will exist outside [0 , 1] d .
Revised: However in the majority of cases a subset of B P ( x 1 , L 1 ) will exist outside [0 , 1] d .

Original: While this observation is trivially true, it has no influence on the radius of certification achieved by Equation 3 due to the symmetry of B P ( x 1 , r ) .
Revised: While this observation is trivially true, it has no influence on the radius of certification achieved by Equation 3 due to the symmetry of B P ( x 1 , r ) .

Original: However, the asymmetric nature of B P ( x 2 , r 2 ) about x 1 guarantees that if B P ( x 2 , r 2 ) exceeds [0 , 1] d , then the closest point to x 1 within the feasible domain must have an associated distance ˆ r > r ′ , as is demonstrated within Figure 1c.
Revised: However, the asymmetric nature of B P ( x 2 , r 2 ) about x 1 guarantees that if B P ( x 2 , r 2 ) exceeds [0 , 1] d , then the closest point to x 1 within the feasible domain must have an associated distance ˆ r > r ′ , as is demonstrated within Figure 1c.

Original: This allows us to make the following observation about improving the feasible radius of certification.
Revised: This allows us to make the following observation about improving the feasible radius of certification.

Original: Lemma 3.5 ( Boundary Certifications by way of Eccentric Circles ) .
Revised: Lemma 3.5 ( Boundary Certifications by way of Eccentric Circles ) .

Original: The eccentricity of B P ( x 2 , L 2 ) as a bounding region about x 1 , and the potential for a subset of B P ( x 2 , L 2 ) to exist outside the feasible space for instances allows us to construct an updated region of certification where
Revised: The eccentricity of B P ( x 2 , L 2 ) as a bounding region about x 1 , and the potential for a subset of B P ( x 2 , L 2 ) to exist outside the feasible space for instances allows us to construct an updated region of certification where

Original: [Equation]
Revised: [Equation]

Original: where z k =
Revised: where z k =

Original: [Equation]
Revised: [Equation]

Original: where 1 is an indicator function acting upon its operator.
Revised: where 1 is an indicator function acting upon its operator.

Original: Proof.
Revised: Proof.

Original: In contrast to the prior proof, for this problem we retain the coordinate system of the input space.
Revised: In contrast to the prior proof, for this problem we retain the coordinate system of the input space.

Original: To support this, we introduce the notation that x n = { x n, 1 , x n, 2 , . . .
Revised: To support this, we introduce the notation that x n = { x n, 1 , x n, 2 , . . .

Original: , x n,d } .
Revised: , x n,d } .

Original: If we let x k ∈ { 0 , 1 } , then the intersection between B P ( x 2 , L 2 ) and the bounding surface in dimension k creates a bounding hypersphere of the form
Revised: If we let x k ∈ { 0 , 1 } , then the intersection between B P ( x 2 , L 2 ) and the bounding surface in dimension k creates a bounding hypersphere of the form

Original: [Equation]
Revised: [Equation]

Original: which yields an effective radius r r 2 =
Revised: which yields an effective radius r r 2 =

Original: [Equation]
Revised: [Equation]

Original: By denoting the projection of x 1 and x 2 onto the bounding hyperplane in the k -th dimension as Ă xand Ă x 2 , then the distance from x 1 to Equation (17) must take the form
Revised: By denoting the projection of x 1 and x 2 onto the bounding hyperplane in the k -th dimension as Ă xand Ă x 2 , then the distance from x 1 to Equation (17) must take the form

Original: [Equation]
Revised: [Equation]

Original: By imposing that z k = 1 when the k -th component of x 1 is greater than 0 . 5 , and 0 otherwise, it follows that max k ∈{ 0 , 1 ,...,d } z k must be an improved radius of certification.
Revised: By imposing that z k = 1 when the k -th component of x 1 is greater than 0 . 5 , and 0 otherwise, it follows that max k ∈{ 0 , 1 ,...,d } z k must be an improved radius of certification.

Original: Algorithm 1 Single Bubble
Revised: Algorithm 1 Single Bubble

Original: 1: Input: data x , samples N , iterations M , true-label i 2: for 1 to M do 3: q E 0 , p E 1 , L 2 , j = Algorithm` x ′ + N ` 0 , σ 2 I N ˘ , N, σ ˘ 4: if j = i then 5: r ′ =
Revised: 1: Input: data x , samples N , iterations M , true-label i 2: for 1 to M do 3: q E 0 , p E 1 , L 2 , j = Algorithm` x ′ + N ` 0 , σ 2 I N ˘ , N, σ ˘ 4: if j = i then 5: r ′ =

Original: L 2 − ∥ x ′ − x ∥ 6: if r ′ >
Revised: L 2 − ∥ x ′ − x ∥ 6: if r ′ >

Original: r
Revised: r

Original: o
Revised: o

Original: then 7: r ′ o = r ′ 8: end if 9: x ′ = x ′ ±
Revised: then 7: r ′ o = r ′ 8: end if 9: x ′ = x ′ ±

Original: γ ∇ x ′ r ′
Revised: γ ∇ x ′ r ′

Original: ∥∇ x ′ r ′
Revised: ∥∇ x ′ r ′

Original: ∥ { γ calculated by Barzilai-Borwein [1].
Revised: ∥ { γ calculated by Barzilai-Borwein [1].

Original: Positive branch is selected if j = i , otherwise the negative branch brings x ′ towards the region in which j = i .} 10: end if 11: end for
Revised: Positive branch is selected if j = i , otherwise the negative branch brings x ′ towards the region in which j = i .} 10: end if 11: end for

Original: 3.4 Algorithms
Revised: 3.4 Algorithms

Original: To demonstrate how the above certification approaches can be applied in practice, Algorithmdemonstrates the application of Equation (5) through a simple, gradient based solver.
Revised: To demonstrate how the above certification approaches can be applied in practice, Algorithmdemonstrates the application of Equation (5) through a simple, gradient based solver.

Original: Such a solver is highly applicable for solving such problems, due to the inherent smoothing nature of randomised smoothing being applicable both to the function space and its derivatives.
Revised: Such a solver is highly applicable for solving such problems, due to the inherent smoothing nature of randomised smoothing being applicable both to the function space and its derivatives.

Original: To implement the multiple transitivity based approach of Section 3.2, Algorithm 1 can trivially be adapted to evaluate derivatives with respect to Equation (12).
Revised: To implement the multiple transitivity based approach of Section 3.2, Algorithm 1 can trivially be adapted to evaluate derivatives with respect to Equation (12).

Original: The boundary treatment of Section 3.3 does not require any additional calculations, but instead is simply the result of applying Equation (16) to the output of Algorithm 1.
Revised: The boundary treatment of Section 3.3 does not require any additional calculations, but instead is simply the result of applying Equation (16) to the output of Algorithm 1.

Original:  Extracting gradient information from non-differentiable functions
Revised:  Extracting gradient information from non-differentiable functions

Original: Implementing the aformentioned process requires the ability to evaluate the gradient of the class expectations.
Revised: Implementing the aformentioned process requires the ability to evaluate the gradient of the class expectations.

Original: This is problematic, as each class expectation is described in terms of a finite sum of non-differentiable indicator functions, as is seen in Equation (2).
Revised: This is problematic, as each class expectation is described in terms of a finite sum of non-differentiable indicator functions, as is seen in Equation (2).

Original: Within this work we have implemented two mechanisms to circumvent these conditions.
Revised: Within this work we have implemented two mechanisms to circumvent these conditions.

Original: The first substitutes the arg max operation with a Gumbel-Softmax [15].
Revised: The first substitutes the arg max operation with a Gumbel-Softmax [15].

Original: In doing so, the class expectations are rendered differentiable.
Revised: In doing so, the class expectations are rendered differentiable.

Original: The second approach involves recasting the Monte-Carlo estimators as integrals of the form
Revised: The second approach involves recasting the Monte-Carlo estimators as integrals of the form

Original: [Equation]
Revised: [Equation]

Original: where ω ( y , x ) is the multivariate-Normal probability distribution centred around x .
Revised: where ω ( y , x ) is the multivariate-Normal probability distribution centred around x .

Original: This formalism, and the symmetry of the underlying space allows for the construction of undifferentiable gradients by
Revised: This formalism, and the symmetry of the underlying space allows for the construction of undifferentiable gradients by

Original: [Equation]
Revised: [Equation]

Original: While this derivation is novel, the resultant gradient operator reflects those seen in prior works [33].
Revised: While this derivation is novel, the resultant gradient operator reflects those seen in prior works [33].

Original: It is important to note that such a sampling process is inherently noisy, and it has previously suggested that the underlying uncertainty scales with the input dimensionality [27].
Revised: It is important to note that such a sampling process is inherently noisy, and it has previously suggested that the underlying uncertainty scales with the input dimensionality [27].

Original: The relative performance of these two approaches—respectively labelled ‘Approximate’ and ‘Full’ for the above approach and the Gumbel-Softmax approaches—will be tested in the following section.
Revised: The relative performance of these two approaches—respectively labelled ‘Approximate’ and ‘Full’ for the above approach and the Gumbel-Softmax approaches—will be tested in the following section.

Original: For the case of the double transitivity of Section 3.2 our experiments suggest that uncertainty in the
Revised: For the case of the double transitivity of Section 3.2 our experiments suggest that uncertainty in the

Original: 5 R 0 .0 .0 .0 .0 .1 .
Revised: 5 R 0 .0 .0 .0 .0 .1 .

Original: C e rt
Revised: C e rt

Original: i ﬁ e d A cc u r a c y a) MNIST 0 .0 .1 .1 .
Revised: i ﬁ e d A cc u r a c y a) MNIST 0 .0 .1 .1 .

Original: 5 R 0 .0 .0 .0 .0 .0 .0 .b)
Revised: 5 R 0 .0 .0 .0 .0 .0 .0 .b)

Original: CIFAR0 .0 .1 .1 .
Revised: CIFAR0 .0 .1 .1 .

Original: 5 R 0 .0 .0 .0 .0 .0 .0 .0 .

Revised: 5 R 0 .0 .0 .0 .0 .0 .0 .0 .

Original: .c)
Revised: .c)

Original: Tiny-Imagenet
Revised: Tiny-Imagenet

Original: Cohen σ =0 .
Revised: Cohen σ =0 .

Original: 5 Best Approx σ =0 .
Revised: 5 Best Approx σ =0 .

Original: Best Full σ =0 .
Revised: Best Full σ =0 .

Original: 5 Cohen σ =1 .
Revised: 5 Cohen σ =1 .

Original: Best Approx σ =1 . 

Revised: Best Approx σ =1 .

Original: Best Full σ =1 .
Revised: Best Full σ =1 .

Original: analytic derivatives produces deleterious results.
Revised: analytic derivatives produces deleterious results.

Original: As such derivatives for the multiple transitivity approach are exclusively considered through autograd for both the Full and Approximate methods.
Revised: As such derivatives for the multiple transitivity approach are exclusively considered through autograd for both the Full and Approximate methods.

Original:  Experiments
Revised:  Experiments

Original: Configuration To evaluate the performance of our proposed certification improvements, we considered the certified radius produced for MNIST [19], CIFAR- 10 [18], and Tiny-Imagenet [16], the latter of these is a 200 -class variant of Imagenet [43] which downsamples images to 3 × 60 × 60 .
Revised: Configuration To evaluate the performance of our proposed certification improvements, we considered the certified radius produced for MNIST [19], CIFAR- 10 [18], and Tiny-Imagenet [16], the latter of these is a 200 -class variant of Imagenet [43] which downsamples images to 3 × 60 × 60 .

Original: All datasets were modelled using the Resnet 18 architecture in PyTorch [32], with Tiny-Imagenet also utilising 2 D adaptive average pooling.
Revised: All datasets were modelled using the Resnet 18 architecture in PyTorch [32], with Tiny-Imagenet also utilising 2 D adaptive average pooling.

Original: For both MNIST and CIFAR- 10 , our experimentation utilised a single NVIDIA P 100 GPU core with 12 GB of GPU RAM, with expectations estimated oversamples.
Revised: For both MNIST and CIFAR- 10 , our experimentation utilised a single NVIDIA P 100 GPU core with 12 GB of GPU RAM, with expectations estimated over 1500 samples. Training employed

Original: Training employed Cross Entropy loss with a batch size of 128 over 50 epochs.
Revised: Cross Entropy loss with a batch size of 128 over 50 epochs.

Original: Each epoch involved every training example was perturbed with a single perturbation drawn from N (0 , σ 2 ) , which was added prior to normalisation.
Revised: Each epoch involved every training example was perturbed with a single perturbation drawn from N (0 , σ 2 ) , which was added prior to normalisation.

Original: Parameter optimisation was performed with Adam [17], with the learning rate set as 0 .
Revised: Parameter optimisation was performed with Adam [17], with the learning rate set as 0 .

Original: Tiny-Imagenet training and evaluation utilised 3 P 100 GPU’s and utilised 1000 samples.
Revised: Tiny-Imagenet training and evaluation utilised 3 P 100 GPU’s and utilised 1000 samples.

Original: Training occurred using SGD over 80 epochs, with a starting learning rate of 0 .
Revised: Training occurred using SGD over 80 epochs, with a starting learning rate of 0 .

Original: 1 , decreasing by a factor of 10 after 30 and 60 epochs, and momentum set to 0 .
Revised: 1 , decreasing by a factor of 10 after 30 and 60 epochs, and momentum set to 0 .

Original: The full code to implement our experiments can be found at https://github.com/andrew-cullen/DoubleBubble .
Revised: The full code to implement our experiments can be found at https://github.com/andrew-cullen/DoubleBubble .

Original: Certified accuracy To explore the performance advantage provided by our technique, we begin by considering the performance of Cohen et al.
Revised: Certified accuracy To explore the performance advantage provided by our technique, we begin by considering the performance of Cohen et al.

Original: against the best of our approaches using both the approximate and full derivatives, as seen in Figure 2.
Revised: against the best of our approaches using both the approximate and full derivatives, as seen in Figure 2.

Original: While there are only minor differences between the two derivative approaches, there are clear regions of out performance relative to Cohen across all tested datasets.
Revised: While there are only minor differences between the two derivative approaches, there are clear regions of out performance relative to Cohen across all tested datasets.

Original: The proportion of this increase appears to be tied to the semantic complexity of the underlying dataset, with decreases in predictive accuracy (visible at R = 0 ) appearing to elicit greater percentage changes in the achieved certified radius, as evidenced by the results for Tiny-Imagenet.
Revised: The proportion of this increase appears to be tied to the semantic complexity of the underlying dataset, with decreases in predictive accuracy (visible at R = 0 ) appearing to elicit greater percentage changes in the achieved certified radius, as evidenced by the results for Tiny-Imagenet.

Original: Semantic complexity drives a decrease in the overall confidence of models, inducing a decrease in the separation between the highest class expectations E 0 and E 1 .
Revised: Semantic complexity drives a decrease in the overall confidence of models, inducing a decrease in the separation between the highest class expectations E 0 and E 1 .

Original: While this process shrinks the achievable certified radius, a higher E 1 provides more information to inform our gradient based search process, allowing for the identification of larger certifications.
Revised: While this process shrinks the achievable certified radius, a higher E 1 provides more information to inform our gradient based search process, allowing for the identification of larger certifications.

Original: This property would suggest that samples with a larger R would exhibit a decreased difference between our techniques and that of Cohen et al.
Revised: This property would suggest that samples with a larger R would exhibit a decreased difference between our techniques and that of Cohen et al.

Original: would decrease, as smaller values of E 1 provide less search information.
Revised: would decrease, as smaller values of E 1 provide less search information.

Original: However, it appears that singularities in the derivatives of Φ − 1 [ E ] as E → { 0 , 1 } counteract the decreased information provided by the second highest class, leading to the contradictory performance best observed in the MNIST experiments of Figure 2 at σ = 0 .
Revised: However, it appears that singularities in the derivatives of Φ − 1 [ E ] as E → { 0 , 1 } counteract the decreased information provided by the second highest class, leading to the contradictory performance best observed in the MNIST experiments of Figure 2 at σ = 0 .

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Rather than strictly considering the best performing of the Approximate and Full solvers, we can also delve into the relative performance of the underlying solvers.
Revised: Rather than strictly considering the best performing of the Approximate and Full solvers, we can also delve into the relative performance of the underlying solvers.

Original: Notably there is a moderate increase in the average percentage improvement of Figure 3 between the σ = 0 .
Revised: Notably there is a moderate increase in the average percentage improvement of Figure 3 between the σ = 0 .

Original: 5 and σ = 1 . 0 cases.
Revised: 5 and σ = 1 . 0 cases.

Original: This would appear to belie our previous statement regarding larger certifications yielding smaller improvements, due to the asymmetry of class information.
Revised: This would appear to belie our previous statement regarding larger certifications yielding smaller improvements, due to the asymmetry of class information.

Original: However, an equivalent certification for σ = 1 . 0 has significantly more information about the second class (due to the multiplicative influence of σ ) , allowing for greater improvements from our gradient based search.
Revised: However, an equivalent certification for σ = 1 . 0 has significantly more information about the second class (due to the multiplicative influence of σ ) , allowing for greater improvements from our gradient based search.

Original: That there is a clear demarcation between the Full and Approximate solver variants reflects the uncertainties introduced by Equation (20).
Revised: That there is a clear demarcation between the Full and Approximate solver variants reflects the uncertainties introduced by Equation (20).

Original: That the approximate technique is still applicable verifies the utility of our approach even when the final layer is a strict arg max function, rather than a Gumbel-Softmax.
Revised: That the approximate technique is still applicable verifies the utility of our approach even when the final layer is a strict arg max function, rather than a Gumbel-Softmax.

Original: The trends in performance across σ are further explored in Figures 4 and 6, the latter of which demonstrates that the median performance improvement increases quasi-linearly with σ .
Revised: The trends in performance across σ are further explored in Figures 4 and 6, the latter of which demonstrates that the median performance improvement increases quasi-linearly with σ .

Original: This is driven by both an increase in the performance of the certifications themselves, and in the number of instances able to be certified in an improved fashion.
Revised: This is driven by both an increase in the performance of the certifications themselves, and in the number of instances able to be certified in an improved fashion.

Original: This later property stems from the smoothing influence of σ , with larger levels of added noise inducing decreases in the difference between the highest class expectations, improving the ability for our search based mechanisms to identify performance improvements.
Revised: This later property stems from the smoothing influence of σ , with larger levels of added noise inducing decreases in the difference between the highest class expectations, improving the ability for our search based mechanisms to identify performance improvements.

Original: Here increases in the performance of the boundary treatment are correlated with larger radii of certification, due to the multiplicative influence of σ upon Equation (3).
Revised: Here increases in the performance of the boundary treatment are correlated with larger radii of certification, due to the multiplicative influence of σ upon Equation (3).

Original: Numerical performance
Revised: Numerical performance

Original: The numerical optimisation process at the core of our certification process inherently induces increases in the computational cost of finding these improved certifications, as shown in Table 1.
Revised: The numerical optimisation process at the core of our certification process inherently induces increases in the computational cost of finding these improved certifications, as shown in Table 1.

Original: While analytically approximating the derivatives for the first eccentric hypersphere yields a lower certified accuracy, the fact that the corresponding computational cost decreases by a factor of more than 4 emphasises the value of this approach.
Revised: While analytically approximating the derivatives for the first eccentric hypersphere yields a lower certified accuracy, the fact that the corresponding computational cost decreases by a factor of more than 4 emphasises the value of this approach.

Original: Interestingly, while the Approximate
Revised: Interestingly, while the Approximate

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: , as method does also utilise auto-differentiation for the Double variant, the increase in computational cost from the Single to Double variants is significantly higher than for the Actual approach.
Revised: , as method does also utilise auto-differentiation for the Double variant, the increase in computational cost from the Single to Double variants is significantly higher than for the Actual approach.

Original: This is surprising, as the Approx variant derives smaller values of r ′ , which should in turn lead to a smaller, easier to navigate search space for r ′′ .
Revised: This is surprising, as the Approx variant derives smaller values of r ′ , which should in turn lead to a smaller, easier to navigate search space for r ′′ .

Original: Instead it counter-intuitively appears that a smaller r ′ induces a search space for r ′′ which is less convex, and more difficult to converge upon.
Revised: Instead it counter-intuitively appears that a smaller r ′ induces a search space for r ′′ which is less convex, and more difficult to converge upon.

Original: Alternative training routines Recent work has considered the potential for enhancing certified robustness by modifying the underlying training regime to incentivise maximising the expectation gap between classes [33].
Revised: Alternative training routines Recent work has considered the potential for enhancing certified robustness by modifying the underlying training regime to incentivise maximising the expectation gap between classes [33].

Original: One such approach is MACER [44], which augments the training time behaviour by considering not just the classification loss, but also the ϵ -robustness loss, which reflects proportion of training samples with robustness above a threshold level.
Revised: One such approach is MACER [44], which augments the training time behaviour by considering not just the classification loss, but also the ϵ -robustness loss, which reflects proportion of training samples with robustness above a threshold level.

Original: Such a training time modification can increase the average certified radius by 10 − 20% , however doing so does increase the overall training cost by more than an order of magnitude.
Revised: Such a training time modification can increase the average certified radius by 10 − 20% , however doing so does increase the overall training cost by more than an order of magnitude.

Original: When applying Geometrically-Informed Certified Robustness to models trained with MACER,
Revised: When applying Geometrically-Informed Certified Robustness to models trained with MACER,

Original: Figure 5 demonstrates that our modifications yield an even more significant improvement than those observed in Figure 2.
Revised: Figure 5 demonstrates that our modifications yield an even more significant improvement than those observed in Figure 2.

Original: Under training with MACER, the best performing of our techniques yielded an approximately 4 percentage point increase in the average certification.
Revised: Under training with MACER, the best performing of our techniques yielded an approximately 4 percentage point increase in the average certification.

Original: From this it is clear that while
Revised: From this it is clear that while

Original: MACER does enhance the certified radii at a sample point, it also induces enough smoothing in the neighbourhood of the sample point to allow transitivity to yield even more significant improvements than are present without MACER.
Revised: MACER does enhance the certified radii at a sample point, it also induces enough smoothing in the neighbourhood of the sample point to allow transitivity to yield even more significant improvements than are present without MACER.

Original: However, we must emphasise that while such training time modifications do producer consistently larger certifications, doing so requires significantly more computational resources, both in terms of training time and GPU memory, as compared to the more traditional certification training regime.
Revised: However, we must emphasise that while such training time modifications do producer consistently larger certifications, doing so requires significantly more computational resources, both in terms of training time and GPU memory, as compared to the more traditional certification training regime.

Original: We also emphasise that training with MACER requires a degree of re-engineering.
Revised: We also emphasise that training with MACER requires a degree of re-engineering.

Original: In contrast the training mechanism used for the remainder of this work only requires the addition of noise to samples prior to being passed through the model, and thus imposes significantly fewer engineering considerations.
Revised: In contrast the training mechanism used for the remainder of this work only requires the addition of noise to samples prior to being passed through the model, and thus imposes significantly fewer engineering considerations.

Original: Limitations While the principles of our Geometrically Informed Certified Robustness are extensible to L p spaces, our experimental work has so far only considered L 2 -norm bounded perturbations due to the guarantee of best possible certification in this space provided by Cohen et al. .
Revised: Limitations While the principles of our Geometrically Informed Certified Robustness are extensible to L p spaces, our experimental work has so far only considered L 2 -norm bounded perturbations due to the guarantee of best possible certification in this space provided by Cohen et al. .

Original: Further
Revised: Further

Original: [Figure]
Revised: [Figure]

Original: experimentation could also consider both these general spaces and a broader range of training methods, which have been shown to be able to tighten the achievable radius of certification [24].
Revised: experimentation could also consider both these general spaces and a broader range of training methods, which have been shown to be able to tighten the achievable radius of certification [24].

Original: We note that enhanced robustness certification have the potential to counter beneficial applications of adversarial attacks, such as those used to promote stylometric privacy [3].
Revised: We note that enhanced robustness certification have the potential to counter beneficial applications of adversarial attacks, such as those used to promote stylometric privacy [3].

Original: However, we believe this drawbacks is significantly outweighed by the potential for enhanced confidence about models for which adversarial incentives exist.
Revised: However, we believe this drawbacks is significantly outweighed by the potential for enhanced confidence about models for which adversarial incentives exist.

Original: Finally, we also emphasise that our approach requires M evaluations of the certified robustness, which each require N Monte-Carlo draws, resulting in time- and memory-complexity of O ( MN ) and O ( NS ) respectively, where S is the size of the output logit vector.
Revised: Finally, we also emphasise that our approach requires M evaluations of the certified robustness, which each require N Monte-Carlo draws, resulting in time- and memory-complexity of O ( MN ) and O ( NS ) respectively, where S is the size of the output logit vector.

Original: With respect to the memorycomplexity, this is shared by any randomised smoothing based approach, and could be improved by implementing batching across the Monte-Carlo process.
Revised: With respect to the memorycomplexity, this is shared by any randomised smoothing based approach, and could be improved by implementing batching across the Monte-Carlo process.

Original: While the time cost can be problematic in some contexts, we emphasise that this framework is both requires both fewer adaptations to the training loop and significantly less training time relative to bound propagation approaches [21, 36].
Revised: While the time cost can be problematic in some contexts, we emphasise that this framework is both requires both fewer adaptations to the training loop and significantly less training time relative to bound propagation approaches [21, 36].

Original: We believe these costs can be reduced by performing the optimisation stages with N ′ < N model draws, and by potentially reusing model draws across the iterative process to approach the O ( N ) time-complexity of prior randomised smoothing based certifications.
Revised: We believe these costs can be reduced by performing the optimisation stages with N ′ < N model draws, and by potentially reusing model draws across the iterative process to approach the O ( N ) time-complexity of prior randomised smoothing based certifications.

Original: Even at present, we believe that the increased computational cost is not intractable, especially for human-in-the-loop certifications.
Revised: Even at present, we believe that the increased computational cost is not intractable, especially for human-in-the-loop certifications.

Original:  Acknowledgements
Revised:  Acknowledgements

Original: This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne.
Revised: This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne.

Original: This Facility was established with the assistance of LIEF Grant LE170100200.
Revised: This Facility was established with the assistance of LIEF Grant LE170100200.

Original: This work was also supported in part by the Australian Department of Defence Next Generation Technologies Fund, as part of the CSIRO/Data61 CRP AMLC project.
Revised: This work was also supported in part by the Australian Department of Defence Next Generation Technologies Fund, as part of the CSIRO/Data61 CRP AMLC project.

Original: Sarah Erfani is in part supported by Australian
Revised: Sarah Erfani is in part supported by Australian

Original: Research Council (ARC)
Revised: Research Council (ARC)

Original: Discovery Early Career Researcher Award (DECRA) DE220100680.
Revised: Discovery Early Career Researcher Award (DECRA) DE220100680.

Original:  Conclusions
Revised:  Conclusions

Original: This work has presented mechanisms that can be exploited to improve achievable levels of certified robustness, based upon exploiting underlying geometric properties of robust neural networks.
Revised: This work has presented mechanisms that can be exploited to improve achievable levels of certified robustness, based upon exploiting underlying geometric properties of robust neural networks.

Original: In doing so our Geometrically Informed Certified Robustness approach has been able to generate certifications that exceed prior guarantees by on average more than 5% at σ = 0 .
Revised: In doing so our Geometrically Informed Certified Robustness approach has been able to generate certifications that exceed prior guarantees by on average more than 5% at σ = 0 .

Original: 5 , with the percentage increase improving quasi-linearly with σ .
Revised: 5 , with the percentage increase improving quasi-linearly with σ .

Original: Incorporating training time modifications likes MACER yields more promising results, with the best performing of our approaches yielding a 4 percentage point increase in the certified proportion at a given radius.
Revised: Incorporating training time modifications likes MACER yields more promising results, with the best performing of our approaches yielding a 4 percentage point increase in the certified proportion at a given radius.

Original: Being able to improve upon the size of these guarantees inherently increases the cost of constructing adversarial attacks against systems leveraging machine learning, especially in the case where the attacker has no ability to observe the size of the robustness certificate.A Appendix A.1 Algorithmic details Algorithm 2 supports Algorithm 1 by demonstrating how the class prediction and expectations are calculated. Of note are two minor changes from prior implementations of this certification regime.
Revised: Being able to improve upon the size of these guarantees inherently increases the cost of constructing adversarial attacks against systems leveraging machine learning, especially in the case where the attacker has no ability to observe the size of the robustness certificate.

Original: The first is the addition of the Gumbel-Softmax on line 4 , although this step is only required for the ‘Full’ derivative approach.

Original: In contrast th ‘Approximate’ techniques able to circumvent this limitation and can be applied directly to the case where the class election is determined by an arg max .

Original: The second difference to prior works is the calculation of the lower and upper bounds on z 0 and z 1 on line 8 .

Original: Our initial testing revealed that when we employed either Sison-Glaz [38] or Goodman et al.

Original: [14] to estimate the multivariate class uncertainties, some Tiny-Imagenet samples devoted more than 95% of their computational time of the process to evaluating the confidence intervals, significantly outweighing even the costly process of model sampling.

Original: Further investigation revealed that this was occurring when there were a significant number of classes reporting counts of approximately 0 , the likelihood for which was higher in Tiny-Imagenet due to the increased class count relative to MNIST and CIFAR- 10 .

Original: To resolve this, we coalesced all classes where y j < 5 into one single meta-class with an associated class-count y ′ = max(5 , (cid:80)

Original: y j < 5 y j , which conforms with the requirements of

Original: Goodman et al.

Original: [14] that all class counts must be greater than 5 .

Original: Our testing demonstrated that while this process slightly decreased the resulting radius of certification (due to small changes in z 0 and z 1 ), the associated decrease in computational time was significant enough to justify this modification.

Original: We also note that all the experiments contained within this work have been conducted against publically releaed datasets with established licenses.

Original: MNIST exists under a GNU v 3 . 0 license;

Original: CIFAR-

Original: 10 employs a MIT license; and Imagenet employs a BSD 3 − Clause license.

Original: A.2 Ramifications of the dimensionality for n >

Original: To improve the achieved certification in the case n > 3 , the added set of hyperspheres must fully enclose the ( d − 1) -dimensional manifold that marks the intersection between S 2 and S 3 .

Original: In two- dimensions—as is used in the examplar Figure 1a—this intersection takes the form of two points.

Original: If

Original: Lemma’s 3.3 and 3.4 are to hold, then encompasing S 2 ∩ S 3 will require two additional certification hyperspheres to be identified.

Original: In the case where d = 3 , the intersection between these two hyperspheres is the boundary of the circle (equivalent to a d = 2 hypersphere) with radius

Original: [Equation]

Original: [Equation]

Original: Thus any set of spheres {S 2 , S 3 , . .

Original: . , S n } must uniformly cover all points on the boundary of this surface if we seek to improve the achieved certification.

Original: To provide an indicative example of how the complexity of the region that must be encircled grows with the underlying dimensionality, we now consider some properties of hyperspheres.

Original: In higher dimensions, prior work [39] has demonstrated that the volume contained within a d -dimensional hypersphere can be expressed as

Original: [Equation]

Original: with an associated surface area of

Original: [Equation]

Original: Thus if S 2 and S 3 are d -dimensional hyperspheres, then their region of intersection would in turn be a ( d − 1) -dimensional

Original: hypersphere, the exterior boundary of which scales with ˜ r d − 2 .

Original: While ˜ r may be less than 1 , it should also be true that any additional spheres would likely have an associated radii less than ˜ r .

Original: As such there would appear to be a power-law proportionality with respect to ( d − 1) between the area covered by the intersection manifold and the size of spheres over which we would seek to enclose said manifold.

Original: This underscores the complexity of finding a set of hyperspheres to encircle the boundary of S 2 ∩ S 3 .

Original: To give further evidence in the growth of complexity, let us consider a unit-hypersphere in R d that represents the intersection of two hyperspheres in R d +1 .

Original: The task of covering such a hypersphere is similar to that of the sphere packing kissing number k

Original: ( d ) [9], which describes the number of touching-but-not-overlapping unit-hyperspheres that can exist upon the surface of a d -dimensional hypersphere.

Original: To date, the kissing number has only been solved for the following dimensions outlined in Table 2, however it has been shown to exhibit exponential growth [4].

Original: [Table]

Original: Within the context of this work, the kissing number must be considered to be a significant underestimate of the number of boundary spheres that would be required to be found, as we must cover all the space around the central sphere (rather than just maximising the number of hyperspheres without intersection), and it is unlikely that the smallest of the set of encircling hyperspheres has the same radius as the region to be encircled.

Original: As such, we can be highly confident that the growth in complexity of the task of enclosing the boundary of intersection beyond the set of hyperspheres {S 1 , S 2 , S 3 } is exponential.

Original: We must also note that even if it were possible to perform such a bounding operation, the gains in certified radius would be exceedingly minor.

Original: [-If the region of intersection between S 2 and S 3 was a hypersphere of radius ˘ r , then going from the case where n = 3 to n > 3 would only increase the certified radius from a ( r ′ ) 
-]
Revised: [-If the region of intersection between S 2 and S 3 was a hypersphere of radius ˘ r , then going from the case where n = 3 to n > 3 would only increase the certified radius from a ( r ′ ) 
-]

Original: + (˘ r ) 2 to a ( r ′ ) 2 + 9(˘ r ) 2 , which is trivial relative to the increase in computational complexity.

Original: A.3 Relative performance for MNIST and CIFAR-

Original: While Figure 2 presents the best performing certified accuracy, it is important to understand the relative performance of the Single Transitivity, Double Transitivity, and Boundary treatments, in a similar fashion to Figures 3 and 4.

Original: In the case of MNIST, while the percentage increases exhibited in Figure 7 as r 1 → 1 are broadly similar to their Tiny-Imagenet counterpart for the Approximate solver, the difference between those results and the Full derivative treatment is significantly smaller, especially at σ = 1 .

Original: This may, in part, be driven by the 1500 samples employed when using MNIST and CIFAR- 10 , in contrast to 1000 for Tiny-Imagenet, which should decrease the uncertainty of the gradient estimation steps.

Original: However, the fact that this decreased difference in performance holds for CIFAR- 10 at σ = 1 .but not σ = 0 .

Original: 5 suggests that the performance difference between the techniques is also dependent upon the semantic complexity of the prediction task.

Original: While CIFAR- 10 is a more complex predictive

Original: [Figure]

Original: Equivalent figures for Tiny-Imagenet can be seen in Figure

Original: [Figure]

Original: Equivalent figures for Tiny-Imagenet can be seen in Figureenvironment than MNIST, which should increase the complexity of the gradient based search routine employed within this work, the increased level of noise at σ = 1 . 0 has a smoothing influence that decreases the complexity of the search task, and it would appear that this is the primary driver of the relative under performance of the Approximate derivatives in both Tiny-Imagenet and CIFARwhen σ = 0 .

Original: When considering the median percentage improvement (relative to Cohen et al. ) of these techniques,

Original: MNIST again reveals interesting properties when we consider Figure 11.

Original: When compared to CIFAR10 and Tiny-Imagenet (in Figures 12 and 6) it becomes apparent that the Approximate approach only produces consistently larger certifications in MNIST.

Original: Given the increased uncertainty in the derivatives calculated by the Approximate technique, this would suggest that the approximate solver may be improved by considering common improvements to gradient descent methods like momentum or the addition of calibrated noise.

Original: While MNIST may be the simplest of all the prediction tasks, Figure 9 demonstrates that at low σ < 0 .

Original: 5 the majority of samples cannot be improved upon by any of the certification enhancements developed within this paper.

Original: Given that this does not hold for CIFAR- 10 nor Tiny-Imagenet (in

Original: Figure 10 and 4 respectively) this would suggest that the potential for Cohen et al.

Original: to be improved upon in low semantic complexity datasets is smaller.

Original: That this behaviour is predominantly seen for small σ also suggests that our initial step size may be too large in these particular cases.

Original: [Figure]

Original: rt i

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: each of our approaches relative to Cohen et al.

Original: for MNIST across the level of additive noise σ .

Original: The median was chosen to provide a fair and representative comparison to Cohen et al. , that filters out outliers in the percentage improvement when r ≪ 1 .

Original: [Figure]

Original: [Figure]

Original: n

Original: [Figure]

Original: ce n t ag e I m p

Original: r

Original: o v e

Original: m e n t

Original: [Figure]

Original: [Figure]

Original: A.4 Influence of the starting step size

Original: The one heretofore un-considered feature is the influence of the initial step-size γ within Algorithm 1.

Original: As is shown in Figures 13 and

Original: [Figure]

Original: while the Full solver only exhibits sensitivity to γ when γ > 0 .

Original: [Figure]

Original: the approximate solvers are far more sensitive, with deleterious performance being observed for γ > 0 .in CIFAR- 10 , and even earlier for Tiny-Imagenet.

Original: This is likely due to the added uncertainty in the

Original: Approximate derivatives leading to convergence upon local sub-optima in the more semantically complex datasets.

Original: Based upon these results, the starting step size was uniformly set to 0 .

Original: 01 for all experiments.

Original: [Figure]