Original: A F RAMEWORK FOR R OBUSTNESS C ERTIFICATION OF S MOOTHED C LASSIFIERS USING F -D IVERGENCES
Revised: A F RAMEWORK FOR R OBUSTNESS C ERTIFICATION OF S MOOTHED C LASSIFIERS USING F -D IVERGENCES

Original: Anonymous authors
Revised: j.hayes

Original: Paper under double-blind review
Revised: @ucl.ac.uk { kaix } @mit.edu

Original: A BSTRACT
Revised: A BSTRACT

Original: Formal veriﬁcation techniques that compute provable guarantees on properties of machine learning models, like robustness to norm-bounded adversarial perturbations, have yielded impressive results.
Revised: Formal veriﬁcation techniques that compute provable guarantees on properties of machine learning models, like robustness to norm-bounded adversarial perturbations, have yielded impressive results.

Original: Although most techniques developed so far require knowledge of the architecture of the machine learning model and remains hard to scale to complex prediction pipelines, the method of randomized smoothing has been shown to overcome many of these obstacles.
Revised: Although most techniques developed so far require knowledge of the architecture of the machine learning model and remain hard to scale to complex prediction pipelines, the method of randomized smoothing has been shown to overcome many of these obstacles.

Original: By requiring only black-box access to the underlying model, randomized smoothing scales to large architectures and is agnostic to the internals of the network.
Revised: By requiring only black-box access to the underlying model, randomized smoothing scales to large architectures and is agnostic to the internals of the network.

Original: However, past work on randomized smoothing has focused on restricted classes of smoothing measures or perturbations (like Gaussian or discrete) and has only been able to prove robustness with respect to simple norm bounds.
Revised: However, past work on randomized smoothing has focused on restricted classes of smoothing measures or perturbations (like Gaussian or discrete) and has only been able to prove robustness with respect to simple norm bounds.

Original: In this paper we introduce a general framework for proving robustness properties of smoothed machine learning models in the black-box setting.
Revised: In this paper we introduce a general framework for proving robustness properties of smoothed machine learning models in the black-box setting.

Original: Speciﬁcally, we extend randomized smoothing procedures to handle arbitrary smoothing measures and prove robustness of the smoothed classiﬁer by using f -divergences.
Revised: Speciﬁcally, we extend randomized smoothing procedures to handle arbitrary smoothing measures and prove robustness of the smoothed classiﬁer by using f -divergences.

Original: Our methodology improves upon the state of the art in terms of computation time or certiﬁed robustness on several image classiﬁcation tasks and an audio classiﬁcation task, with respect to several classes of adversarial perturbations.
Revised: Our methodology improves upon the state of the art in terms of computation time or certiﬁed robustness on several image classiﬁcation tasks and an audio classiﬁcation task, with respect to several classes of adversarial perturbations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Predictors obtained from machine learning algorithms have been shown to be vulnerable to making errors when the inputs are perturbed by carefully chosen small but imperceptible amounts (Szegedy et al., 2014; Biggio et al., 2013).
Revised: Predictors obtained from machine learning algorithms have been shown to be vulnerable to making errors when the inputs are perturbed by carefully chosen small but imperceptible amounts (Szegedy et al., 2014; Biggio et al., 2013).

Original: This has motivated signiﬁcant amount of research in improving adversarial robustness of a machine learning model (see, e.g. Goodfellow et al., 2015; Madry et al., 2018).
Revised: This has motivated signiﬁcant amount of research in improving adversarial robustness of a machine learning model (see, e.g. Goodfellow et al., 2015; Madry et al., 2018).

Original: While signiﬁcant advances have been made, it has been shown that models that were estimated to be robust have later been broken by stronger attacks (Athalye et al., 2018; Uesato et al., 2018).
Revised: While signiﬁcant advances have been made, it has been shown that models that were estimated to be robust have later been broken by stronger attacks (Athalye et al., 2018; Uesato et al., 2018).

Original: This has led to the need for methods that offer provable guarantees that the predictor cannot be forced to misclassify an example by any attack algorithm restricted to produce perturbations within a certain set (for example, within an (cid:96) p norm ball).
Revised: This has led to the need for methods that offer provable guarantees that the predictor cannot be forced to misclassify an example by any attack algorithm restricted to produce perturbations within a certain set (for example, within an (cid:96) p norm ball).

Original: While progress has been made leading to methods that are able to compute provable guarantees for several image and text classiﬁcation tasks (Wong & Kolter, 2018; Wong et al., 2018; Raghunathan et al., 2018; Dvijotham et al., 2018; Katz et al., 2017; Huang et al., 2019; Jia et al., 2019), these methods require extensive knowledge of the architecture of the predictor and are not easy to extend to new models or architectures, requiring specialized algorithms for each new class of models.
Revised: While progress has been made leading to methods that are able to compute provable guarantees for several image and text classiﬁcation tasks (Wong & Kolter, 2018; Wong et al., 2018; Raghunathan et al., 2018; Dvijotham et al., 2018; Katz et al., 2017; Huang et al., 2019; Jia et al., 2019), these methods require extensive knowledge of the architecture of the predictor and are not easy to extend to new models or architectures, requiring specialized algorithms for each new class of models.

Original: Further, the computational complexity of these methods grows signiﬁcantly with input dimension and model size.
Revised: Furthermore, the computational complexity of these methods grows signiﬁcantly with input dimension and model size.

Original: Consequently, to deal with these obstacles, recent work has proposed the randomized smoothing strategy for verifying the robustness of classiﬁers.
Revised: To deal with these obstacles, recent work has proposed the randomized smoothing strategy for verifying the robustness of classiﬁers.

Original: Speciﬁcally, Lecuyer et al.
Revised: Speciﬁcally, Lecuyer et al.

Original: ; Cohen et al.
Revised: and Cohen et al.

Original: (2019) have shown that robustness properties can be more easily veriﬁed for the smoothed version of a base classiﬁer h  :
Revised: (2019) have shown that robustness properties can be more easily veriﬁed for the smoothed version of a base classiﬁer h producing labels in some set Y :

Original: [Equation]
Revised: [Equation]

Original: where the labels returned by the smoothed classiﬁer h s are obtained by taking a “majority vote” over the predictions of the original classiﬁer h on random inputs drawn from a probability distribution µ ( x ) , called the smoothing measure (here Y denotes the set of classes in the problem).
Revised: where the labels returned by the smoothed classiﬁer h s are obtained by taking a “majority vote” over the predictions of the original classiﬁer h on random inputs drawn from a probability distribution µ ( x ) , called the smoothing measure .

Original: Lecuyer et al.
Revised: Lecuyer et al.

Original: (2019) showed that verifying the robustness of this smoothed classiﬁer is signiﬁcantly simpler than verifying the original classiﬁer h and only requires estimating the distribution of outputs of the classiﬁer under random perturbations of the input, but does not require access to the internals of the classiﬁer h .
Revised: (2019) showed that verifying the robustness of this smoothed classiﬁer is signiﬁcantly simpler than verifying the original classiﬁer h and only requires estimating the distribution of outputs of the classiﬁer under random perturbations of the input, but does not require access to the internals of the classiﬁer h .

Original: We refer to this as black-box veriﬁcation.
Revised: We refer to this as black-box veriﬁcation.

Original: In this work, we develop a general framework for black-box veriﬁcation that recovers prior work as special cases, and improves upon previous results in various ways.
Revised: In this work, we develop a general framework for black-box veriﬁcation that recovers prior work as special cases, and improves upon previous results in various ways.

Original: Contributions Our contributions are summarized as follows:
Revised: Contributions Our contributions are summarized as follows:

Original: We formulate the general problem of black-box veriﬁcation via a generalized randomized smoothing procedure, which extends existing approaches to allow for arbitrary smoothing measures.
Revised: We formulate the general problem of black-box veriﬁcation via a generalized randomized smoothing procedure, which extends existing approaches to allow for arbitrary smoothing measures.

Original: Speciﬁcally, we show that robustness certiﬁcates for smoothed classiﬁers can be obtained by solving a small convex optimization problem when allowed adversarial perturbations can be characterized via divergence-based bounds on the smoothing measure.
Revised: Speciﬁcally, we show that robustness certiﬁcates for smoothed classiﬁers can be obtained by solving a small convex optimization problem when allowed adversarial perturbations can be characterized via divergence-based bounds on the smoothing measure.

Original: We prove that our certiﬁcates generalize previous results obtained in related work (Lecuyer et al., 2019; Cohen et al., 2019; Li et al., 2019), and vastly extend the class of perturbations and smoothing measures that can be used while still allowing certiﬁable guarantees.
Revised: We prove that our certiﬁcates generalize previous results obtained in related work (Lecuyer et al., 2019; Cohen et al., 2019; Li et al., 2019), and vastly extend the class of perturbations and smoothing measures that can be used while still allowing certiﬁable guarantees.

Original: We introduce the notion of full-information and information-limited settings, and show that the information-limited setting that has been the main focus of prior work leads to weaker certiﬁcates for smoothed probabilistic classiﬁers, and can be improved by using additional information (the distribution of label scores under randomized smoothing).
Revised: We introduce the notion of full-information and information-limited settings, and show that the information-limited setting that has been the main focus of prior work leads to weaker certiﬁcates for smoothed probabilistic classiﬁers, and can be improved by using additional information (the distribution of label scores under randomized smoothing).

Original: We evaluate our framework experimentally on image and classiﬁcation tasks, obtaining robustness certiﬁcates that improve upon other black-box methods either in terms of certiﬁcate tightness or computation time on robustness to (cid:96) 0 , (cid:96) 1 or (cid:96) 2 perturbations on MNIST, CIFARand ImageNet.
Revised: We evaluate our framework experimentally on image and classiﬁcation tasks, obtaining robustness certiﬁcates that improve upon other black-box methods either in terms of certiﬁcate tightness or computation time on robustness to (cid:96) 0 , (cid:96) 1 or (cid:96) 2 perturbations on MNIST, CIFARand ImageNet.

Original: (cid:96) 2 perturbations result from worst-case realizations of white noise that is common in many image, speech and video processing.
Revised: (cid:96) 2 perturbations result from worst-case realizations of white noise that is common in many image, speech and video processing.

Original: (cid:96) 

Revised: (cid:96)

Original: perturbations can model missing data (missing pixels in an image, or samples in a time-domain audio signal) while (cid:96) 1 perturbations can be used to model convex combinations of discrete perturbations in text classiﬁcation (Jia et al., 2019).
Revised: perturbations can model missing data (missing pixels in an image, or samples in a time-domain audio signal) while (cid:96) 1 perturbations can be used to model convex combinations of discrete perturbations in text classiﬁcation (Jia et al., 2019).

Original: We also obtain the ﬁrst, to the best of our knowledge, certiﬁably robust model for an audio classiﬁcation task, Librispeech (Panayotov et al., 2015), with variable-length inputs.
Revised: We also obtain the ﬁrst, to the best of our knowledge, certiﬁably robust model for an audio classiﬁcation task, Librispeech (Panayotov et al., 2015), with variable-length inputs

Original:  B LACK - BOX VERIFICATION FOR SMOOTHED CLASSIFIERS
Revised:  B LACK - BOX VERIFICATION FOR SMOOTHED CLASSIFIERS

Original: Consider a binary classiﬁer h :
Revised: Consider a binary classiﬁer h :

Original: X → {± 1 } given to us as a black box, so we can only access the inputs and outputs of h but not its internals.
Revised: X → {± 1 } given to us as a black box, so we can only access the inputs and outputs of h but not its internals.

Original: We are interested in investigating the robustness of the smoothed classiﬁer h s (deﬁned in Eq. 1) against adversarial perturbations of size at most (cid:15) with respect to a given norm (cid:107)·(cid:107) .
Revised: We are interested in investigating the robustness of the smoothed classiﬁer h s (deﬁned in Eq. 1) against adversarial perturbations of size at most (cid:15) with respect to a given norm (cid:107)·(cid:107) .

Original: To determine whether a norm-bounded adversarial attack on a ﬁxed input x ∈ X with h s ( x ) =
Revised: To determine whether a norm-bounded adversarial attack on a ﬁxed input x ∈ X with h s ( x ) =

Original: +1 could be successful, we can solve the optimization problem
Revised: +1 could be successful, we can solve the optimization problem

Original: [Equation]
Revised: [Equation]

Original: (2) and check whether the minimum value can be smaller than 12 .
Revised: and check whether the minimum value can be smaller than 12 .

Original: This is a non-convex optimization problem for which we may not even be able to compute gradients since we only have black-box access to h .
Revised: This is a non-convex optimization problem for which we may not even be able to compute gradients since we only have black-box access to h .

Original: While techniques have been developed to address this problem, obtaining provable guarantees on whether these algorithms actually ﬁnd the worst-case adversarial perturbation is difﬁcult since we do not know anything about the nature of h .
Revised: While techniques have been developed to address this problem, obtaining provable guarantees on whether these algorithms actually ﬁnd the worst-case adversarial perturbation is difﬁcult since we do not know anything about the nature of h .

Original: Motivated by this difﬁculty, we take a different approach:
Revised: Motivated by this difﬁculty, we take a different approach:

Original: Rather than studying the adversarial attack in the input space X , we study it in the space of probability measures over inputs, denoted by P ( X ) .
Revised: Rather than studying the adversarial attack in the input space X , we study it in the space of probability measures over inputs, denoted by P ( X ) .

Original: Formally, this amounts to rewriting Eq.
Revised: Formally, this amounts to rewriting Eq.

Original: 2 as
Revised: 2 as

Original: [Equation]
Revised: [Equation]

Original: This is an inﬁnite dimensional optimization problem over the space of probability measures ν ∈ P ( X ) subject to the constraint ν ∈ D = { µ ( x (cid:48) ) :
Revised: This is an inﬁnite dimensional optimization problem over the space of probability measures ν ∈ P ( X ) subject to the constraint ν ∈ D = { µ ( x (cid:48) ) :

Original: (cid:107) x (cid:48) − x (cid:107) ≤ (cid:15) } .
Revised: (cid:107) x (cid:48) − x (cid:107) ≤ (cid:15) } .

Original: While this set is still intractable to deal with, we can consider relaxations of this set deﬁned by divergence constraints between ν and ρ = µ ( x ) , i.e., D ⊆ { ν : D ( ν (cid:107) ρ ) ≤ (cid:15) D } where D denotes some divergence between probability distributions.
Revised: While this set is still intractable to deal with, we can consider relaxations of this set deﬁned by divergence constraints between ν and ρ = µ ( x ) , i.e., D ⊆ { ν : D ( ν (cid:107) ρ ) ≤ (cid:15) D } where D denotes some divergence between probability distributions.

Original: We will show in Section 3 that for several commonly used divergences (in fact, for any f -divergence; cf.
Revised: We will show in Section 3 that for several commonly used divergences (in fact, for any f -divergence; cf.

Original: Ali & Silvey, 1966), the relaxed problem can be solved efﬁciently.
Revised: Ali & Silvey, 1966), the relaxed problem can be solved efﬁciently.

Original: 2. A GENERAL FRAMEWORK FOR ROBUSTNESS CERTIFICATION
Revised: 2. A GENERAL FRAMEWORK FOR ROBUSTNESS CERTIFICATION

Original: To formulate the general veriﬁcation problem, consider a speciﬁcation φ : X → Z ⊆ R : a generic function over the input space (that typically is a function of the classiﬁer output) that we want to verify has certain properties.
Revised: To formulate the general veriﬁcation problem, consider a speciﬁcation φ : X → Z ⊆ R : a generic function over the input space (that typically is a function of the classiﬁer output) that we want to verify has certain properties.

Original: Unless otherwise speciﬁed, we will assume that X ⊆ R d (we work in a d dimensional input space).
Revised: Unless otherwise speciﬁed, we will assume that X ⊆ R d (we work in a d dimensional input space).

Original: Our framework also involves a reference measure ρ (in the above example we would take ρ = µ ( x ) ) and a collection of perturbed distributions D (in the above example we would take D = D x,(cid:15)
Revised: Our framework also involves a reference measure ρ (in the above example we would take ρ = µ ( x ) ) and a collection of perturbed distributions D (in the above example we would take D = D x,(cid:15)

Original: = { µ ( x (cid:48) ) : (cid:107) x (cid:48) − x (cid:107) ≤ (cid:15) } ).
Revised: = { µ ( x (cid:48) ) : (cid:107) x (cid:48) − x (cid:107) ≤ (cid:15) } ).

Original: Verifying that a given speciﬁcation φ is robustly certiﬁed is equivalent to checking whether the optimal value of the optimization problem
Revised: Verifying that a given speciﬁcation φ is robustly certiﬁed is equivalent to checking whether the optimal value of the optimization problem

Original: [Equation]
Revised: [Equation]

Original: is non-negative.
Revised: is non-negative.

Original: Solving problems of this form is the key workhorse of our general framework for black-box certiﬁcation of adversarial robustness for smoothed classiﬁers.
Revised: Solving problems of this form is the key workhorse of our general framework for black-box certiﬁcation of adversarial robustness for smoothed classiﬁers.

Original: Using these ingredients we introduce two closely related certiﬁcation problems: information-limited robust certiﬁcation and full-information robust certiﬁcation .
Revised: Using these ingredients we introduce two closely related certiﬁcation problems: information-limited robust certiﬁcation and full-information robust certiﬁcation .

Original: In the former case, we assume that we are given only given access to P X ∼ ρ
Revised: In the former case, we assume that we are given only given access to P X ∼ ρ

Original: [ φ ( X ) =
Revised: [ φ ( X ) =

Original: +1] , P X ∼ ν
Revised: +1] , P X ∼ ν

Original: [ φ ( X )
Revised: [ φ ( X )

Original: = +1] .
Revised: = +1] .

Original: In the latter case, we are given full-access to speciﬁcation φ .
Revised: In the latter case, we are given full-access to speciﬁcation φ .

Original: The deﬁnitions are below.
Revised: The deﬁnitions are below.

Original: Deﬁnition 2.1 (Information-limited robust certiﬁcation) .
Revised: Deﬁnition 2.1 (Information-limited robust certiﬁcation) .

Original: Given reference distribution ρ ∈ P ( X ) , probabilities θ a , θ b that satisfy θ a , θ b ≥ 0 , θ a + θ b ≤ 1 and collection of perturbed distributions
Revised: Given reference distribution ρ ∈ P ( X ) , probabilities θ a , θ b that satisfy θ a , θ b ≥ 0 , θ a + θ b ≤ 1 and collection of perturbed distributions

Original: D ⊂ P ( X ) containing ρ , deﬁne the class of speciﬁcations S as
Revised: D ⊂ P ( X ) containing ρ , deﬁne the class of speciﬁcations S as

Original: [Equation]
Revised: [Equation]

Original: We say that S is information-limited robustly certiﬁed at ρ with respect to D if the following condition holds: 
Revised: We say that S is information-limited robustly certiﬁed at ρ with respect to D if the following condition holds: E X ∼ ν

Revised: [ φ ( X )] ≥ 0 for all ν ∈ D , φ ∈ S

Original: [Equation]
Revised: [Equation]

Original: Note since we don’t have access to φ , we need to prove that E X ∼ ν
Revised: Note since we don’t have access to φ , we need to prove that E X ∼ ν

Original: [ φ ( X )] ≥ 0 ∀ ν ∈ D is satisﬁed for all speciﬁcations in set S .
Revised: [ φ ( X )] ≥ 0 ∀ ν ∈ D is satisﬁed for all speciﬁcations in set S .

Original: Although the information-limited case may seem challenging because we need to provide guarantees that hold simultaneously over a whole class of speciﬁcations, it turns out that, for perturbation sets D speciﬁed by an f -divergence bound, this certiﬁcation task can be solved efﬁciently using convex optimization.
Revised: Although the information-limited case may seem challenging because we need to provide guarantees that hold simultaneously over a whole class of speciﬁcations, it turns out that, for perturbation sets D speciﬁed by an f -divergence bound, this certiﬁcation task can be solved efﬁciently using convex optimization.

Original: Deﬁnition 2.2 (Full-information robust certiﬁcation) .
Revised: Deﬁnition 2.2 (Full-information robust certiﬁcation) .

Original: Given a reference distribution ρ ∈ P ( X ) , a speciﬁcation φ : X → Z ⊆ R and a collection of perturbed distributions D ⊂ P ( X ) containing ρ , we say that φ is full-information robustly certiﬁed at ρ with respect to D if the following condition holds:
Revised: Given a reference distribution ρ ∈ P ( X ) , a speciﬁcation φ : X → Z ⊆ R and a collection of perturbed distributions D ⊂ P ( X ) containing ρ , we say that φ is full-information robustly certiﬁed at ρ with respect to D if the following condition holds:

Original: [Equation]
Revised: E X ∼ ν

Revised: [ φ ( X )] ≥ 0 for all ν ∈ D

Original: Most often we are dealing with the case where we have full access to the speciﬁcation φ , thus we should be able to certify using full-information robust certiﬁcation .
Revised: Most often we are dealing with the case where we have full access to the speciﬁcation φ , thus we should be able to certify using full-information robust certiﬁcation .

Original: However, prior works, Cohen et al.
Revised: However, prior works, Cohen et al.

Original: and Lecuyer et al.
Revised: and Lecuyer et al.

Original: (2019), have only provided solutions to certify with respect to the information-limited case where we cannot use all of the information about φ .
Revised: (2019), have only provided solutions to certify with respect to the information-limited case where we cannot use all of the information about φ .

Original: The framework we develop is a more general method that can be used in both information-limited and full-information scenarios.
Revised: The framework we develop is a more general method that can be used in both information-limited and full-information scenarios.

Original: We will demonstrate that our framework recovers certiﬁcates provided by Cohen et al.
Revised: We will demonstrate that our framework recovers certiﬁcates provided by Cohen et al.

Original: , Li et al.
Revised: , Li et al.

Original: and dominates Lecuyer et al.
Revised: and dominates Lecuyer et al.

Original: (2019) in the information-limited setting (see section 5).
Revised: (2019) in the information-limited setting (see section 5).

Original: Further, it can utilize full-information about the speciﬁcation φ to provide tighter certiﬁcates for smoothed probabilistic classiﬁers (see section 6).
Revised: Further, it can utilize full-information about the speciﬁcation φ to provide tighter certiﬁcates for smoothed probabilistic classiﬁers (see section 6).

Original: 2.1. R OBUSTNESS SPECIFICATION FOR SMOOTHED HARD CLASSIFIERS
Revised: 2.1. R OBUSTNESS SPECIFICATION FOR SMOOTHED HARD CLASSIFIERS

Original: We ﬁrst note that the deﬁnitions above are sufﬁcient to capture the standard usage of randomized smoothing as it has been used in past work (e.g. Lecuyer et al., 2019;
Revised: We ﬁrst note that the deﬁnitions above are sufﬁcient to capture the standard usage of randomized smoothing as it has been used in past work (e.g. Lecuyer et al., 2019;

Original: Cohen et al., 2019) to verify the robustness of smoothed multi-class classiﬁers.
Revised: Cohen et al., 2019) to verify the robustness of smoothed multi-class classiﬁers.

Original: Speciﬁcally, consider smoothing a classiﬁer h : X → Y with a ﬁnite set of labels Y using a smoothing measure µ : X (cid:55)→ P ( X ) .
Revised: Speciﬁcally, consider smoothing a classiﬁer h : X → Y with a ﬁnite set of labels Y using a smoothing measure µ : X (cid:55)→ P ( X ) .

Original: The resulting randomly smoothed classiﬁer h s is deﬁned in Eq.
Revised: The resulting randomly smoothed classiﬁer h s is deﬁned in Eq.

Original: Our goal is to certify that the prediction h
Revised: Our goal is to certify that the prediction h

Original: s ( x ) is robust to perturbations of size at most (cid:15) measured by distance function 1 d : X × X
Revised: s ( x ) is robust to perturbations of size at most (cid:15) measured by distance function 1 d : X × X

Original: (cid:55)→ R + , i.e., h s ( x (cid:48) )
Revised: (cid:55)→ R + , i.e., h s ( x (cid:48) )

Original: =
Revised: =

Original: h s ( x ) ∀ x (cid:48) such that d ( x, x (cid:48) ) ≤ (cid:15) .
Revised: h s ( x ) ∀ x (cid:48) such that d ( x, x (cid:48) ) ≤ (cid:15) .

Original: To pose this question within our framework, we choose the reference distribution ρ = µ ( x ) , the set of perturbed distributions D x,(cid:15)
Revised: To pose this question within our framework, we choose the reference distribution ρ = µ ( x ) , the set of perturbed distributions D x,(cid:15)

Original: = { µ ( x (cid:48) )
Revised: = { µ ( x (cid:48) )

Original: : d ( x, x (cid:48) ) ≤ (cid:15) } , and the following speciﬁcations.
Revised: : d ( x, x (cid:48) ) ≤ (cid:15) } , and the following speciﬁcations.

Original: Let c = h s ( x ) .
Revised: Let c = h s ( x ) .

Original: For every c (cid:48) ∈
Revised: For every c (cid:48) ∈

Original: Y \ { c } , we deﬁne the speciﬁcation φ c,c (cid:48) :
Revised: Y \ { c } , we deﬁne the speciﬁcation φ c,c (cid:48) :

Original: X (cid:55)→ {− 1 , 0 , +1 } as follows: [Equation] [Equation]
Revised: X (cid:55)→ {− 1 , 0 , +1 } as follows: φ c,c (cid:48) ( x ) =

Original: =   + if h ( x ) =
Revised:   + if h ( x ) =

Original: c , −if h ( x ) =
Revised: c , −if h ( x ) =

Original: c (cid:48) , otherwise .
Revised: c (cid:48) , otherwise .

Original: Then, Eq. 5 holds if and only if every φ c,c (cid:48) , c (cid:48) (cid:54) =
Revised: Then, Eq. 5 holds if and only if every φ c,c (cid:48) , c (cid:48) (cid:54) =

Original: c , is robustly certiﬁed at µ ( x ) with respect to D x,(cid:15)
Revised: c , is robustly certiﬁed at µ ( x ) with respect to D x,(cid:15)

Original: (see Appendix A.1).
Revised: (see Appendix A.1).

Original: 2. C ONSTRAINT SETS FROM F - DIVERGENCES
Revised: 2. C ONSTRAINT SETS FROM F - DIVERGENCES

Original: Dealing with the set D x,(cid:15) directly is difﬁcult due to its possibly non-convex geometry.
Revised: Dealing with the set D x,(cid:15) directly is difﬁcult due to its possibly non-convex geometry.

Original: In this section, we discuss speciﬁc relaxations of this set, i.e., choices for sets D such that D x,(cid:15) ⊆
Revised: In this section, we discuss speciﬁc relaxations of this set, i.e., choices for sets D such that D x,(cid:15) ⊆

Original: D that are easier to optimize over.
Revised: D that are easier to optimize over.

Original: In particular, we focus on a general family of constraint sets deﬁned in terms of f -divergences.
Revised: In particular, we focus on a general family of constraint sets deﬁned in terms of f -divergences.

Original: These divergences satisfy a number of useful properties and include many well-known instances (e.g. relative entropy, total variation); see Appendix A.2 for details.
Revised: These divergences satisfy a number of useful properties and include many well-known instances (e.g. relative entropy, total variation); see Appendix A.2 for details.

Original: Deﬁnition 2.3.
Revised: Deﬁnition 2.3.

Original: ( f -divergence constraint set).
Revised: ( f -divergence constraint set).

Original: Given ρ, ν ∈ P ( X ) , their f -divergence is deﬁned as
Revised: Given ρ, ν ∈ P ( X ) , their f -divergence is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where f : R + (cid:55)→ R is a convex function with f (1) = 0 .
Revised: where f : R + (cid:55)→ R is a convex function with f (1) = 0 .

Original: Given a reference distribution ρ , an f -divergence D f and a bound (cid:15) f ≥ 0 , we deﬁne the f -divergence constraint set to be:
Revised: Given a reference distribution ρ , an f -divergence D f and a bound (cid:15) f ≥ 0 , we deﬁne the f -divergence constraint set to be:

Original: [Equation]
Revised: [Equation]

Original: Technically, this deﬁnition depends on the Radon-Nikodym derivative of ν with respect to ρ , but we ignore measure-theoretic issues in this paper for simplicity of exposition.
Revised: Technically, this deﬁnition depends on the Radon-Nikodym derivative of ν with respect to ρ , but we ignore measure-theoretic issues in this paper for simplicity of exposition.

Original: For continuous distributions, ν and ρ should be treated as densities, and for discrete distributions as probability mass functions.
Revised: For continuous distributions, ν and ρ should be treated as densities, and for discrete distributions as probability mass functions.

Original: Relaxations using f -divergence This construction immediately allows us to obtain relaxations of
Revised: Relaxations using f -divergence This construction immediately allows us to obtain relaxations of

Original: D x,(cid:15) .
Revised: D x,(cid:15) .

Original: For example, by choosing f ( u ) =
Revised: For example, by choosing f ( u ) =

Original: u log( u ) , we have the KL -divergence.
Revised: u log( u ) , we have the KL -divergence.

Original: Using KL -divergence yields the following relaxation between norm-based and divergence-based constraint sets for Gaussian smoothing measures, i.e. µ ( x ) =
Revised: Using KL -divergence yields the following relaxation between norm-based and divergence-based constraint sets for Gaussian smoothing measures, i [Equation] e. µ ( x ) =

Original: N ( x, σ 2 I ) :
Revised: N ( x, σ 2 I ) :

Original: [Equation]
Revised: [Equation]

Original: Tighter relaxations can be constructed by combining multiple divergence-based constraints.
Revised: Tighter relaxations can be constructed by combining multiple divergence-based constraints.

Original: In particular, suppose
Revised: In particular, suppose

Original: F is a collection of convex functions each deﬁning an f -divergence, and assume each f ∈ F has a bound (cid:15) f associated with it.
Revised: F is a collection of convex functions each deﬁning an f -divergence, and assume each f ∈ F has a bound (cid:15) f associated with it.

Original: Then we can deﬁne the constraint set containing perturbed distributions where all the bounds hold simultaneously (Fig. 1):
Revised: Then we can deﬁne the constraint set containing perturbed distributions where all the bounds hold simultaneously (Fig. 1):

Original: [Equation]
Revised: [Equation]

Original: In this paper, we work with the following divergences:
Revised: In this paper, we work with the following divergences:

Revised: (1) R´enyi:

Revised: R α ( ν (cid:107) ρ )

Original: [Equation]{+= log(
 + D f ( ν (cid:107) ρ )) / ( α − 1)+}
Revised: {+= log(
 + D f ( ν (cid:107) ρ )) / ( α − 1)+}

Original: where f ( x ) = x
Revised: where f ( x ) = x

Original: α − 1 (for α ≥1 ) and
Revised: α − 1 (for α ≥ and

Revised: R α ( ρ (cid:107) ν ) =

Revised: log(1 −

Original: R α ( ρ (cid:107) 
Revised: D f ( ν (cid:107) ρ

Original: [Equation]
Revised: ( α − ) with f x ) =

Original: with f ( x ) = 1 − x α (for 0 ≤ α ≤ 1 ).
Revised: 1 − x α (for 0 ≤ α ≤ 1 ).

Original: The limit α → ∞ yields the inﬁnite order R´enyi divergence 
Revised: The limit α → ∞ yields the inﬁnite order R´enyi divergence R ∞ ( ν (cid:107) ρ

Original: [Equation]
Revised: ) = sup x ν ( x ) /ρ ( x )) .

Original: (2) KL( ν (cid:107) ρ ) =
Revised: (2) KL( ν (cid:107) ρ ) =

Original: D f ( ν (cid:107) ρ ) with f ( x ) = x log( x ) .
Revised: D f ( ν (cid:107) ρ ) with f ( x ) = x log( x ) .

Original: (3) Hockey-Stick: D HS ,β ( ν (cid:107) ρ ) =
Revised: (3) Hockey-Stick: D HS ,β ( ν (cid:107) ρ ) =

Original: D f ( ν (cid:107) ρ ) with f ( x ) = max( x − β, 0) − max(1 − β, 0) .
Revised: D f ( ν (cid:107) ρ ) with f ( x ) = max( x − β, 0) − max(1 − β, 0) .

Original: It turns out that the R´enyi and KL divergences are computationally attractive for a broad class of smoothing measures, while the Hockey-Stick divergences are theoretically attractive as they lead to optimal certiﬁcates in the information-limited setting.
Revised: It turns out that the R´enyi and KL divergences are computationally attractive for a broad class of smoothing measures, while the Hockey-Stick divergences are theoretically attractive as they lead to optimal certiﬁcates in the information-limited setting.

Original: However, Hockey-Stick divergences are harder to estimate in general, so we only use them for Gaussian smoothing measures.
Revised: However, Hockey-Stick divergences are harder to estimate in general, so we only use them for Gaussian smoothing measures.

Original: 2. C OMPUTING F - DIVERGENCE BOUNDS
Revised: 2. C OMPUTING F - DIVERGENCE BOUNDS

Original: In general, our framework can be used with any family of smoothing measures and any family of f divergences such that an upper bound on max ν ∈D x,(cid:15)
Revised: In general, our framework can be used with any family of smoothing measures and any family of f divergences such that an upper bound on max ν ∈D x,(cid:15)

Original: D f ( ν (cid:107) ρ ) can be estimated efﬁciently.
Revised: D f ( ν (cid:107) ρ ) can be estimated efﬁciently.

Original: We describe how f -divergence bounds can be obtained for several classes of smoothing measures:
Revised: We describe how f -divergence bounds can be obtained for several classes of smoothing measures:

Original: Product measures Product measures are of the form µ ( x ) =
Revised: Product measures Product measures are of the form µ ( x ) =

Original: ⊗ di =1 µ i ( x i )
Revised: ⊗ di =1 µ i ( x i )

Original: where X = (cid:81) di
Revised: where X = (cid:81) di

Original: =1 X i and µ i is a smoothing measure on X i .
Revised: =1 X i and µ i is a smoothing measure on X i .

Original: We note that the discrete smoothing measure used in (Lee et al., 2019), the Gaussian measure used in (Cohen et al., 2019) and the Laplacian measure used in (Li et al., 2019) are all of this form.
Revised: We note that the discrete smoothing measure used in (Lee et al., 2019), the Gaussian measure used in (Cohen et al., 2019) and the Laplacian measure used in (Li et al., 2019) are all of this form.

Original: For such measures, one can construct bounds on R ´ enyidivergences subject to any (cid:96) p norm constraint using a Lagrangian relaxation of the optimization problem max x (cid:48) :
Revised: For such measures, one can construct bounds on R ´ enyidivergences subject to any (cid:96) p norm constraint using a Lagrangian relaxation of the optimization problem max x (cid:48) :

Original: (cid:107) x − x (cid:48) (cid:107) p ≤ (cid:15) R α ( µ ( x (cid:48) )
Revised: (cid:107) x − x (cid:48) (cid:107) p ≤ (cid:15) R α ( µ ( x (cid:48) )

Original: (cid:107) µ ( x ))
Revised: (cid:107) µ ( x ))

Original: (see Appendix A.3 for details).
Revised: (see Appendix A.3 for details).

Original: Norm-based smoothing measures Appendix A.9.1 also shows how we can obtain bounds on the inﬁnite-order R ´ enyi divergence R ∞ , as well as on several classes of f -divergences, for norm-based smoothing measures of the form µ ( x )[ X ] ∝ exp( −(cid:107)
Revised: Norm-based smoothing measures Appendix A.9.1 also shows how we can obtain bounds on the inﬁnite-order R ´ enyi divergence R ∞ , as well as on several classes of f -divergences, for norm-based smoothing measures of the form µ ( x )[ X ] ∝ exp( −(cid:107)

Original: X − x (cid:107) ) .
Revised: X − x (cid:107) ) .

Original:  O UR CERTIFICATION PROCEDURES
Revised:  O UR CERTIFICATION PROCEDURES

Original: We now show how to reduce the problems of full-information and information-limited robust blackbox certiﬁcation to simple convex optimization problems for general constraint sets D deﬁned in terms of f -divergences.
Revised: We now show how to reduce the problems of full-information and information-limited robust blackbox certiﬁcation to simple convex optimization problems for general constraint sets D deﬁned in terms of f -divergences.

Original: This allows us, by extension, to solve the problem for related divergences like R´enyi divergences.
Revised: This allows us, by extension, to solve the problem for related divergences like R ´ enyi divergences.

Original: The following two theorems provide the main foundation for the veriﬁcation procedures in the paper.
Revised: The following two theorems provide the main foundation for the veriﬁcation procedures in the paper.

Original: Theorem 1 (Verifying full-information robust certiﬁcation) .
Revised: Theorem 1 (Verifying full-information robust certiﬁcation) .

Original: Let D F be the constraint set deﬁned by
Revised: Let D F be the constraint set deﬁned by

Original: F = { f 1 , . .
Revised: F = { f 1 , . .

Original: , f M } and (cid:15) f
Revised: , f M } and (cid:15) f

Original: i = (cid:15) i .
Revised: i = (cid:15) i .

Original: Deﬁne f λ ( u ) =
Revised: Deﬁne f λ ( u ) =

Original: (cid:80) Mi
Revised: (cid:80) Mi

Original: =1 λ
Revised: =1 λ

Original: i f i ( u ) and denote its convex conjugateby f ∗ λ .
Revised: i f i ( u ) and denote its convex conjugateby f ∗ λ .

Original: The speciﬁcation φ is robustly certiﬁed at ρ with respect to D F (cf.
Revised: The speciﬁcation φ is robustly certiﬁed at ρ with respect to D F (cf.

Original: Deﬁnition 2.2)
Revised: Deﬁnition 2.2)

Original: if and only if the optimal value of the following convex optimization problem is non-negative:
Revised: if and only if the optimal value of the following convex optimization problem is non-negative:

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 1, given in Appendix A.4, uses standard duality results to show that the dual of the veriﬁcation optimization problem has the desired form.
Revised: The proof of Theorem 1, given in Appendix A.4, uses standard duality results to show that the dual of the veriﬁcation optimization problem has the desired form.

Original: We note that the special case where
Revised: We note that the special case where

Original: M = 1 reduces to Proposition 1 of Duchi & Namkoong (2018), although the result is used in a completely different context in that work.
Revised: M = 1 reduces to Proposition 1 of Duchi & Namkoong (2018), although the result is used in a completely different context in that work.

Original: To build a practical certiﬁcation algorithm from Theorem 1, we must do two things: 1) compute the optimal values of λ and κ ; and 2) estimate the expectation in Eq.
Revised: To build a practical certiﬁcation algorithm from Theorem 1, we must do two things: 1) compute the optimal values of λ and κ ; and 2) estimate the expectation in Eq.

Original: Since the estimation of the expectation cannot be done in closed form (due to the black-box nature of φ ), we must rely on sampling.
Revised: Since the estimation of the expectation cannot be done in closed form (due to the black-box nature of φ ), we must rely on sampling.

Original: In step 1 of Algorithm 1, we use N samples taken independently from ρ to estimate the expectation and solve the “sampled” optimization problem using an off-the-shelf solver (Diamond & Boyd, 2016).
Revised: In step 1 of Algorithm 1, we use N samples taken independently from ρ to estimate the expectation and solve the “sampled” optimization problem using an off-the-shelf solver (Diamond 2 For any function f : R + (cid:55)→ R , its convex conjugate is deﬁned as f ∗ ( u ) = max v ≥ 0 ( uv − f ( v )) . [Table] [Table] [Table]

Original: This gives us κ ∗ , λ ∗ , the estimated optimal values of κ and λ , respectively.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Boyd, 2016).

Revised: This gives us κ ∗ , λ ∗ , the estimated optimal values of κ and λ , respectively.

Original: Then we take these values and compute a high-conﬁdence lower bound on the objective function of Eq. 6, which is then used to verify robustness.
Revised: Then we take these values and compute a high-conﬁdence lower bound on the objective function of Eq. 6, which is then used to verify robustness.

Original: In particular, in step 2, we compute a high-conﬁdence upper bound E ub on the expectation term in the objective such that E ub ≥ E X ∼ ρ
Revised: In particular, in step 2, we compute a high-conﬁdence upper bound E ub on the expectation term in the objective such that E ub ≥ E X ∼ ρ

Original: [ f ∗ λ ( κ ∗ − φ ( X ))] with probability at least ζ ; this computation involves taking ˜ N independent samples from ρ and ﬁnding a conﬁdence interval around the resulting empirical estimate of the expectation (for details, see Eq.in Appendix A.9.1).
Revised: [ f ∗ λ ( κ ∗ − φ ( X ))] with probability at least ζ ; this computation involves taking ˜ N independent samples from ρ and ﬁnding a conﬁdence interval around the resulting empirical estimate of the expectation (for details, see Eq.in Appendix A.9.1).

Original: Plugging in this estimate back into Eq.
Revised: Plugging in this estimate back into Eq.

Original: 6 gives the desired high-conﬁdence lower bound in step 3.
Revised: 6 gives the desired high-conﬁdence lower bound in step 3.

Original: Details of both subroutines E STIMATE O PT and U PPER C ONFIDENCE B OUND used in Algorithm 1 are given in Algorithm 3 in Appendix A.9.2.
Revised: Details of both subroutines E STIMATE O PT and U PPER C ONFIDENCE B OUND used in Algorithm 1 are given in Algorithm 3 in Appendix A.9.2.

Original: Our next theorem concerns the specialization of this veriﬁcation procedure to the information-limited setting.
Revised: Our next theorem concerns the specialization of this veriﬁcation procedure to the information-limited setting.

Original: Theorem 2 (Verifying information-limited robust certiﬁcation) .
Revised: Theorem 2 (Verifying information-limited robust certiﬁcation) .

Original: Let D F be as in Theorem 1, and
Revised: Let D F be as in Theorem 1, and

Original: S and θ a , θ b be as in Deﬁnition 2.1.
Revised: S and θ a , θ b be as in Deﬁnition 2.1.

Original: The class of speciﬁcations S is information-limited robustly certiﬁed at ρ with respect to D F (cf.
Revised: The class of speciﬁcations S is information-limited robustly certiﬁed at ρ with respect to D F (cf.

Original: Deﬁnition 2.1)
Revised: Deﬁnition 2.1)

Original: if and only if the optimal value of the following convex optimization problem is non-negative:
Revised: if and only if the optimal value of the following convex optimization problem is non-negative:

Original: [Equation]
Revised: [Equation]

Original: to
Revised: to

Original: [Equation]
Revised: [Equation]

Original: where θ =
Revised: where θ =

Original: ( θ a , θ b , 1 − θ a − θ b ) and ζ = ( ζ a , ζ b , ζ c ) are interpreted as probability distributions.
Revised: ( θ a , θ b , 1 − θ a − θ b ) and ζ = ( ζ a , ζ b , ζ c ) are interpreted as probability distributions.

Original: The proof of Theorem 2 is presented in Appendix A.5.
Revised: The proof of Theorem 2 is presented in Appendix A.5.

Original: It is based on the fact that in the informationlimited setting, it is possible to directly compute the expectation in Eq. 6, and in fact this expectation only depends on φ via the probabilities θ a and θ b .
Revised: It is based on the fact that in the informationlimited setting, it is possible to directly compute the expectation in Eq. 6, and in fact this expectation only depends on φ via the probabilities θ a and θ b .

Original: Theorem 2 naturally leads to a certiﬁcation algorithm, presented in Algorithm 2.
Revised: Theorem 2 naturally leads to a certiﬁcation algorithm, presented in Algorithm 2.

Original: It simply uses the same procedure as Cohen et al.
Revised: It simply uses the same procedure as Cohen et al.

Original: (2019) to compute a high-conﬁdence lower bound θ a on the probability of the correct class under randomized smoothing and then solves the convex optimization problem Eq. 7.
Revised: (2019, Section 3.2) to compute a high-conﬁdence lower bound θ a on the probability of the correct class under randomized smoothing and then solves the convex optimization problem Eq. 7.

Original: Again, we can use an off-the-shelf solver CVXPY (Diamond & Boyd, 2016) in stepfor the general M > 1 case, but closed-form solutions are also available for M = 1 ; these are given in Table 4 in Appendix A.6. 
Revised: Again, we can use an off-the-shelf solver CVXPY (Diamond & Boyd, 2016) in step 2 for the general M > 1 case, but closed-form solutions are also available for M = 1 ; these are given in Table 4 in Appendix A.6. Algorithm

Original: Inputs: Query access to classiﬁer h , correct label y , sampling access to reference distribution ρ , divergences f i and bounds (cid:15) i , sample sizes N , ˜ N , conﬁdence level ζ . 
Revised: Inputs: Query access to classiﬁer h , correct label y , sampling access to reference distribution ρ , divergences f i and bounds (cid:15) i , sample sizes N , ˜ N , conﬁdence level ζ . 1: Sample X 1 , . . .

Original: 1: Use the method in Section 3.2 of Cohen et al.

Original: (2019) to determine whether y is the most likely label produced for h ( X ) with X  ∼ ρ(using N samples) and if so, obtain alower bound θ a on the probability that h outputs the correct class with conﬁdence ζ (using ˜ N samples).
Revised: , X N ∼ ρ and test 3 whether y = arg max y (cid:48) ∈Y P X ∼ ρ [ h ( X ) = y (cid:48) ] . 2: Sample X 1 , . . . , X ˜ N ∼ ρ and compute 4 a bound P X ∼ ρ [ h ( X ) =

Original: 2: Obtain o ∗ by solving Eq.
Revised: y (cid:48) ] ≥ θ a with conﬁdence ζ . 3: Obtain o ∗ by solving Eq.

Original: 7 with θ a ← θ a and θ b ← 1 − θ a . 3: If o ∗ ≥ 0 return CERTIFIED else return NOT CERTFIED
Revised: 7 with θ a ← θ a and θ b ← 1 − θ a . 4: If o ∗ ≥ 0 return CERTIFIED else return NOT CERTIFIED.

Original:  T HEORETICAL ANALYSIS OF CERTIFICATION METHODS
Revised:  T HEORETICAL ANALYSIS OF CERTIFICATION METHODS

Original: We now present theoretical results characterizing our certiﬁcation methods and show the following:
Revised: We now present theoretical results characterizing our certiﬁcation methods and show the following:

Revised: 3 Using the algorithm from Hung & Fithian (2019).Using the algorithm from Clopper & Pearson (1934).

Original: 1 For smoothed probabilistic classiﬁers, the full-information certiﬁcate dominates the information-limited one. 
Revised: For smoothed probabilistic classiﬁers, the full-information certiﬁcate dominates the informationlimited one. 2.

Original: In the information-limited setting, if we deﬁne the f -divergence relaxation D F using HockeyStick divergences with speciﬁc parameters, then the computed certiﬁcate is provably tight.
Revised: In the information-limited setting, if we deﬁne the f -divergence relaxation D F using HockeyStick divergences with speciﬁc parameters, then the computed certiﬁcate is provably tight.

Original: 4. A DVANTAGE OF FULL - INFORMATION CERTIFICATION
Revised: 4. A DVANTAGE OF FULL - INFORMATION CERTIFICATION

Original: Consider a soft binary classiﬁer H :
Revised: Consider a soft binary classiﬁer H :

Original: X → [0 , 1] that outputs the probability of label +1 and consider a point x ∈ X with H ( x ) > 1 / 2 .
Revised: X → [0 , 1] that outputs the probability of label +1 and consider a point x ∈ X with H ( x ) > 1 / 2 .

Original: We deﬁne the speciﬁcation φ ( x ) =
Revised: We deﬁne the speciﬁcation φ ( x ) =

Original: H ( x ) − 12 .
Revised: H ( x ) − 12 .

Original: Then, the smoothed classiﬁer H s ( x ) =
Revised: Then, the smoothed classiﬁer H s ( x ) =

Original: E X ∼ µ ( x ) [ H ( X )] predicts label +1 for all x (cid:48) with (cid:107) x (cid:48)
Revised: E X ∼ µ ( x ) [ H ( X )] predicts label +1 for all x (cid:48) with (cid:107) x (cid:48)

Original: − x (cid:107) ≤ (cid:15) if and only if φ is full-information robustly certiﬁed at µ ( x ) with respect to D x,(cid:15)
Revised: − x (cid:107) ≤ (cid:15) if and only if φ is full-information robustly certiﬁed at µ ( x ) with respect to D x,(cid:15)

Original: = { µ ( x (cid:48) ) : (cid:107) x (cid:48) − x (cid:107) ≤ (cid:15) } .
Revised: = { µ ( x (cid:48) ) : (cid:107) x (cid:48) − x (cid:107) ≤ (cid:15) } .

Original: Note that the optimization in Theorem 1 depends on the full distribution of φ ( X ) ∈ [ − 1 / 2 , 1 / 2] , X ∼ µ ( x ) .
Revised: Note that the optimization in Theorem 1 depends on the full distribution of φ ( X ) ∈ [ − 1 / 2 , 1 / 2] , X ∼ µ ( x ) .

Original: On the other hand, to certify this robustness in the information-limited setting is equivalent to taking the speciﬁcation φ ( x ) = 

Revised: On the other hand, to certify this robustness in the information-limited setting is equivalent to taking the speciﬁcation φ ( x ) =

Original: [ H ( x ) > 1 / 2] (the indicator function of the event H ( x ) > 1 / 2 ), in which case the only information available is θ a
Revised: [ H ( x ) > 1 / 2] (the indicator function of the event H ( x ) > 1 / 2 ), in which case the only information available is θ a

Original: = H s ( x ) =
Revised: = H s ( x ) =

Original: E X ∼ µ ( x ) [ H ( X )] .
Revised: E X ∼ µ ( x ) [ H ( X )] .

Original: To compare the two approaches, consider the objective of Eq.
Revised: To compare the two approaches, consider the objective of Eq.

Original: 6 with a single f -divergence constraint
Revised: 6 with a single f -divergence constraint

Original: D f ( ν (cid:107) ρ ) ≤ (cid:15) .
Revised: D f ( ν (cid:107) ρ ) ≤ (cid:15) .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where the third line follows from Jensen’s inequality.
Revised: where the third line follows from Jensen’s inequality.

Original: The proof of Theorem 2 shows that maximizing the ﬁnal expression above with respect to κ, λ is equivalent to the dual of the information-limited certiﬁcation problem Eq.
Revised: The proof of Theorem 2 shows that maximizing the ﬁnal expression above with respect to κ, λ is equivalent to the dual of the information-limited certiﬁcation problem Eq.

Original: Thus, the information-limited setting computes a weaker certiﬁcate than the full-information setting for soft classiﬁers:
Revised: Thus, the information-limited setting computes a weaker certiﬁcate than the full-information setting for soft classiﬁers:

Original: Corollary 3.
Revised: Corollary 3.

Original: The optimization problem of Eq.
Revised: The optimization problem of Eq.

Original: 6 with the speciﬁcation φ deﬁned above has an optimal value that is greater than or equal to that of the optimization problem deﬁned in Eq.
Revised: 6 with the speciﬁcation φ deﬁned above has an optimal value that is greater than or equal to that of the optimization problem deﬁned in Eq.

Original: . 4. T IGHT RELAXATIONS FOR I NFORMATION - LIMITED ROBUST CERTIFICATION
Revised: . 4. T IGHT RELAXATIONS FOR I NFORMATION - LIMITED ROBUST CERTIFICATION

Original: Ideally, we would like to certify robustness of speciﬁcations with respect to sets of the form D x,(cid:15)
Revised: Ideally, we would like to certify robustness of speciﬁcations with respect to sets of the form D x,(cid:15)

Original: = { µ ( x (cid:48) )
Revised: = { µ ( x (cid:48) )

Original: : d ( x, x (cid:48) ) ≤ (cid:15) } .
Revised: : d ( x, x (cid:48) ) ≤ (cid:15) } .

Original: The following result shows that the gap between the ideal D x,(cid:15) and the tractable constraint sets D F can be closed in the context of information-limited robust certiﬁcation provided that we can measure hockey-stick divergences of every non-negative order β ≥ 0 .
Revised: The following result shows that the gap between the ideal D x,(cid:15) and the tractable constraint sets D F can be closed in the context of information-limited robust certiﬁcation provided that we can measure hockey-stick divergences of every non-negative order β ≥ 0 .

Original: The proof is given in Appendix A.7.
Revised: The proof is given in Appendix A.7.

Original: Theorem 4.
Revised: Theorem 4.

Original: Let θ a , θ b , ρ, D , S be as in Deﬁnition 2.1.
Revised: Let θ a , θ b , ρ, D , S be as in Deﬁnition 2.1.

Original: Deﬁne (cid:15) β = max ν ∈D D HS ,β ( ν (cid:107) ρ ) for all β ≥ 0 and let β ∗ a , β ∗ b be chosen as follows:
Revised: Deﬁne (cid:15) β = max ν ∈D D HS ,β ( ν (cid:107) ρ ) for all β ≥ 0 and let β ∗ a , β ∗ b be chosen as follows:

Original: [Equation]
Revised: [Equation]

Original: Deﬁne the constraint set
Revised: Deﬁne the constraint set

Original: [Equation]
Revised: [Equation]

Original: Then, S is information-limited robustly certiﬁed at ρ with respect to D if and only if S is informationlimited robustly certiﬁed at ρ with respect to D HS .
Revised: Then, S is information-limited robustly certiﬁed at ρ with respect to D if and only if S is informationlimited robustly certiﬁed at ρ with respect to D HS .

Original: Thus, the optimal information-limited certiﬁcate in this case can be obtained by applying theorem 2 to D HS .
Revised: Thus, the optimal information-limited certiﬁcate in this case can be obtained by applying theorem 2 to D HS .

Original:  C ONNECTIONS WITH PRIOR WORK
Revised:  C ONNECTIONS WITH PRIOR WORK

Original: Table 1 summarizes the differences between our work and prior work in terms of the set of smoothing measures admitted, the ofﬂine computation cost of the certiﬁcation procedure (which needs to be performed once for every possible perturbation size and choice of smoothing measure), the perturbations considered, whether they can use information beyond θ a , θ b to improve the certiﬁcates and whether they compute optimal certiﬁcates for a given smoothing measure in the informationlimited setting.
Revised: Table 1 summarizes the differences between our work and prior work in terms of the set of smoothing measures admitted, the ofﬂine computation cost of the certiﬁcation procedure (which needs to be performed once for every possible perturbation size and choice of smoothing measure), the perturbations considered, whether they can use information beyond θ a , θ b to improve the certiﬁcates and whether they compute optimal certiﬁcates for a given smoothing measure in the informationlimited setting.

Original: µ ( x )
Revised: µ ( x )

Original: Computation (cid:107)·(cid:107)
Revised: Computation (cid:107)·(cid:107)

Original: Use additional
Revised: Use additional

Original: Optimal information?
Revised: Optimal information?

Original: Cohen et al.
Revised: Cohen et al.

Original: Gaussian
Revised: Gaussian

Original: O (1) (cid:96)
Revised: O (1) (cid:96)

Original: X
Revised: X

Original: Yes
Revised: Yes

Original: Lee et al.
Revised: Lee et al.

Original: Finite
Revised: Finite

Original: O ( d 3 ) (cid:96)
Revised: O ( d 3 ) (cid:96)

Original: Only decision trees
Revised: Only decision trees

Original: Yes
Revised: Yes

Original: Support*
Revised: Support *

Original: Li et al.
Revised: Li et al.

Original: Gaussian
Revised: Gaussian

Original: O (1) (cid:96)
Revised: O (1) (cid:96)

Original: X
Revised: X

Original: No
Revised: No

Original: Laplacian (cid:96)
Revised: Laplacian (cid:96)

Original: Lecuyer et al.
Revised: Lecuyer et al.

Original: Arbitrary
Revised: Arbitrary

Original: O (1)
Revised: O (1)

Original: All (cid:96) p
Revised: All (cid:96) p

Original: X
Revised: X

Original: No
Revised: No

Original: Our work
Revised: Our work

Original: Arbitrary
Revised: Arbitrary

Original: O (1)
Revised: O (1)

Original: All (cid:96) p
Revised: All (cid:96) p

Original: Arbitrary classiﬁers
Revised: Arbitrary classiﬁers

Original: Yes
Revised: Yes

Original: Cohen et al.
Revised: Cohen et al.

Original: (2019) study the problem of verifying hard classiﬁers smoothed by Gaussian noise, and derive optimal certiﬁcates with respect to (cid:96) 2 perturbations of the input.
Revised: (2019) study the problem of verifying hard classiﬁers smoothed by Gaussian noise, and derive optimal certiﬁcates with respect to (cid:96) 2 perturbations of the input.

Original: Their results can be recovered as a special case of our framework when applied to sets deﬁned via constraints on hockey-stick divergences.
Revised: Their results can be recovered as a special case of our framework when applied to sets deﬁned via constraints on hockey-stick divergences.

Original: Theorem 4 shows that the optimal certiﬁcate in the information-limited setting can be computed by applying theorem 2 to a constraint set with two hockey-stick divergences.
Revised: Theorem 4 shows that the optimal certiﬁcate in the information-limited setting can be computed by applying theorem 2 to a constraint set with two hockey-stick divergences.

Original: For the Gaussian measure µ ( x ) =
Revised: For the Gaussian measure µ ( x ) =

Original: N (cid:0) x, σ 2 I (cid:1) , the HS divergence D HS ,β ( µ ( x ) (cid:107) µ ( x (cid:48) )) can be computed in closed form and is purely a function of the (cid:96) 2 distance (cid:107) x − x (cid:48)
Revised: N (cid:0) x, σ 2 I (cid:1) , the HS divergence D HS ,β ( µ ( x ) (cid:107) µ ( x (cid:48) )) can be computed in closed form and is purely a function of the (cid:96) 2 distance (cid:107) x − x (cid:48)

Original: (cid:107) 2 .
Revised: (cid:107) 2 .

Original: This enables us to efﬁcienctly compute the β ∗ a , β ∗ b in theorem 4.
Revised: This enables us to efﬁcienctly compute the β ∗ a , β ∗ b in theorem 4.

Original: Thus, we obtain the following result (see Appendix
Revised: Thus, we obtain the following result (see Appendix

Original: A.7.2 for a proof):
Revised: A.7.2 for a proof):

Original: Corollary 5.
Revised: Corollary 5.

Original: Let ρ = N (cid:0) x, σ [Equation] [Equation] (cid:107)
Revised: Let ρ = N (cid:0) x, σ 2 I (cid:1) , D x,(cid:15)

Original: [Equation]
Revised: =

Original:  ≤ [Equation] } and 1 ≥ θ a ≥ θ b ≥ 0 .
Revised: {N (cid:0) x (cid:48) , σ 2 I (cid:1) : (cid:107) x − x (cid:48) (cid:107) 2 ≤ (cid:15) } and 1 ≥ θ a ≥ θ b ≥ 0 .

Original: Let D HS be deﬁned as in theorem 4.
Revised: Let D HS be deﬁned as in theorem 4.

Original: Then, applying theorem 2 to the constraint set D HS gives the following condition for robust certiﬁcation:
Revised: Then, applying theorem 2 to the constraint set D HS gives the following condition for robust certiﬁcation:

Original: [Equation]
Revised: [Equation]

Original: where Ψ g is the CDF of a standard normal random variable N (0 , 1) .
Revised: where Ψ g is the CDF of a standard normal random variable N (0 , 1) .

Original: With straightforward algebra (worked out in appendix A.7.2) , this can be shown to be equivalent to
Revised: With straightforward algebra (worked out in appendix A.7.2) , this can be shown to be equivalent to

Original: [Equation]
Revised: [Equation]

Original: which is the certiﬁcate from Theorem 1 of Cohen et al.
Revised: which is the certiﬁcate from Theorem 1 of Cohen et al.

Original: Lee et al.
Revised: Lee et al.

Original: (2019) derive optimal certiﬁcates in the information-limited setting under the assumption that the likelihood ratio between measures ν ( X ) ρ ( X ) (where ν = µ ( x (cid:48) ) , ρ = µ ( x ) ) can only take values from a ﬁnite set.
Revised: (2019) derive optimal certiﬁcates in the information-limited setting under the assumption that the likelihood ratio between measures ν ( X ) ρ ( X ) (where ν = µ ( x (cid:48) ) , ρ = µ ( x ) ) can only take values from a ﬁnite set.

Original: This is a restrictive assumption that prevents the authors from accommodating natural smoothing measures like Gaussian or Laplacian measures.
Revised: This is a restrictive assumption that prevents the authors from accommodating natural smoothing measures like Gaussian or Laplacian measures.

Original: Further, the complexity of computing the certiﬁcates in their framework is signiﬁcant: O ( d 3 ) computation (where d is the input dimension) is needed to certify smoothness to (cid:96) 0 perturbations.
Revised: Further, the complexity of computing the certiﬁcates in their framework is signiﬁcant: O ( d 3 ) computation (where d is the input dimension) is needed to certify smoothness to (cid:96) 0 perturbations.

Original: The authors also derive tighter certiﬁcates for the special case of certain classes of decision trees by exploiting the tree structure.
Revised: The authors also derive tighter certiﬁcates for the special case of certain classes of decision trees by exploiting the tree structure.

Original: In contrast, our framework can derive tighter certiﬁcates in the full-information setting for arbitrary classiﬁers.
Revised: In contrast, our framework can derive tighter certiﬁcates in the full-information setting for arbitrary classiﬁers.

Original: Li et al.
Revised: Li et al.

Original: (2019) use properties of R´enyi divergences to derive robustness certiﬁcates for classiﬁers smoothed by Gaussian (resp.
Revised: (2019) use properties of R´enyi divergences to derive robustness certiﬁcates for classiﬁers smoothed by Gaussian (resp.

Original: Laplacian) noise under (cid:96) 2 (resp.
Revised: Laplacian) noise under (cid:96) 2 (resp.

Original: (cid:96) 1 ) perturbations.
Revised: (cid:96) 1 ) perturbations.

Original: Their results can be obtained as special cases of ours; in particular, the R ´ enyi divergence certiﬁcates in Table 4 (in
Revised: Their results can be obtained as special cases of ours; in particular, the R ´ enyi divergence certiﬁcates in Table 4 (in

Original: Appendix A.6) recover the results of Lemma 1 of Li et al.
Revised: Appendix A.6) recover the results of Lemma 1 of Li et al.

Original: , but the latter are only applicable for Gaussian and Laplacian smoothing measures.
Revised: , but the latter are only applicable for Gaussian and Laplacian smoothing measures.

Original: Lecuyer et al.
Revised: Lecuyer et al.

Original: (2019) introduce the notion of pixel differential privacy (pixelDP) and show that smoothing measures µ satisfying pixelDP with respect to a certain type of perturbations lead to adversarially robust classiﬁers.
Revised: (2019) introduce the notion of pixel differential privacy (pixelDP) and show that smoothing measures µ satisfying pixelDP with respect to a certain type of perturbations lead to adversarially robust classiﬁers.

Original: We can show that pixelDP can be viewed as a special instance of our certiﬁcation framework with two speciﬁc hockey-stick divergences, and that the certiﬁcates derived from the pixelDP are provably dominated by the certiﬁcates from our framework (Theorem 1) with the same choice of divergences (see Corollary 7 in Appendix A.7.3).
Revised: We can show that pixelDP can be viewed as a special instance of our certiﬁcation framework with two speciﬁc hockey-stick divergences, and that the certiﬁcates derived from the pixelDP are provably dominated by the certiﬁcates from our framework (Theorem 1) with the same choice of divergences (see Corollary 7 in Appendix A.7.3)

Original:  E XPERIMENTS 6. F ULL -I NFORMATION IMPROVES UPON I NFORMATION - LIMITED CERTIFICATION
Revised:  E XPERIMENTS 6. F ULL -I NFORMATION IMPROVES UPON I NFORMATION - LIMITED CERTIFICATION

Revised: [Figure]

Revised: t

Original:  To compare full-information certiﬁcates with limited-information certiﬁcates, we trained a ResNet152 model on ImageNet with data augmentation by adding noise via sampling from a zero-mean Gaussian with variance 0.5 for each coordinate; during certiﬁcation we sample from the same distribution to estimate lower bounds on the probability of the top predicted class.
Revised: i [Figure] To compare full-information certiﬁcates with limitedinformation certiﬁcates we trained a ResNet-152 model on ImageNet with data augmentation by adding noise via sampling from a zero-mean Gaussian with variance 0.5 for each coordinate; during certiﬁcation we sample from the same distribution to estimate lower bounds on the probability of the top predicted class.

Original: For the fullinformation certiﬁcate, we use two hockey-stick divergences for the certiﬁcate and tune the parameters β to obtain the highest value in the optimization problem in step 2 of Algorithm 1.
Revised: For the fullinformation certiﬁcate, we use two hockey-stick divergences for the certiﬁcate and tune the parameters β to obtain the highest value in the optimization problem in step 2 of Algorithm 1.

Original: For the infromationlimited certiﬁcate, our approach reduces to that of (Cohen et al., 2019) and we follow the same certiﬁcation procedure.
Revised: For the infromation-limited certiﬁcate, our approach reduces to that of (Cohen et al., 2019) and we follow the same certiﬁcation procedure.

Original: We use N = 1000 , ˜ N = 1000000 , ζ = .
Revised: We use N = 1000 , ˜ N = 1000000 , ζ = .

Original: 99 for both certiﬁcation procedures.
Revised: 99 for both certiﬁcation procedures.

Original: [Figure]

Original: The dashed line represents equal certiﬁcates and every point below the dashed line has a stronger certiﬁcate from the full information veriﬁcation setting.

Original: We run the comparison on 50 randomly selected examples from the validation set.

Original: Each blue dot in Figure 2 corresponds to one test point, with its x coordinate representing the radius for full information certiﬁcate (from Algorithm 1) and y coordinate the information-limited certiﬁcate (which is equivalent to the certiﬁcation procedure of Cohen et al., 2019).

Original: The running time of the full-information certiﬁcation procedure is .2s

Original: per example (excluding the sampling cost) while the limited-information certiﬁcation takes .002s

Original: per example.

Original: Both procedures incur the same sampling cost as they use the same number of samples.

Original: Figure 2 shows the difference between the two certiﬁcates.
Revised: Figure 2 shows the difference between the two certiﬁcates.

Original: The certiﬁcate provided by the fullinformation method is always stronger than the one given by the information-limited method.
Revised: The certiﬁcate provided by the full-information method is always stronger than the one given by the informationlimited method.

Original: The difference is often substantia - for one of the test samples, the full-information setting can certify robustness to (cid:96) 2 perturbations of radius (cid:15)
Revised: The difference is often substantial – for one of the test samples, the full-information setting can certify robustness to (cid:96) 2 perturbations of radius (cid:15)

Original: = 9 .
Revised: = 9 .in

Original: 42 in the full-information case while the limitedinformation certiﬁcate can only be provided for perturbation radius (cid:15)
Revised: the full-information case while the limited-information certiﬁcate can only be provided for perturbation radius (cid:15)

Original: = 2 .
Revised: = 2 .

Original: 6. S CALABILITY AND T IGHTNESS
Revised: 6. S CALABILITY AND T IGHTNESS

Original: In this section we consider (cid:96) 0 perturbations for both ImageNet and Binary MNIST (that is, we consider the number of pixels that can be perturbed without changing the prediction).
Revised: In this section we consider (cid:96) 0 perturbations for both ImageNet and Binary MNIST (that is, we consider the number of pixels that can be perturbed without changing the prediction).

Original: To test for scalability and tightness trade-offs of our framework, we compare our methodology to that of Lee et al.
Revised: To test for scalability and tightness trade-offs of our framework, we compare our methodology to that of Lee et al.

Original: (2019), as their work obtains the optimal bound for (cid:96) 0 .
Revised: (2019), as their work obtains the optimal bound for (cid:96) 0 .

Original: We computed certiﬁcates for a single model for each classiﬁcation task; for Binary MNIST we used the same model and training procedure as Lee et al.
Revised: We computed certiﬁcates for a single model for each classiﬁcation task; for Binary MNIST we used the same model and training procedure as Lee et al.

Original: (2019) and
Revised: (2019) and

Original: for ImageNet, we used the model released in the Github code accompanying the paper of Lee et al.
Revised: for ImageNet, we used the model released in the Github code accompanying the paper of Lee et al.

Original: We use the discrete smoothing measure (appendix A.10) with parameter p = 0 .
Revised: We use the discrete smoothing measure (appendix A.10) with parameter p = 0 .

Original: 8 for Binary MNIST certiﬁcation, and p = 0 .
Revised: 8 for Binary MNIST certiﬁcation, and p = 0 .

Original: 2 for ImageNet certiﬁcation.
Revised: 2 for ImageNet certiﬁcation.

Original: In our experiments we ran the certiﬁcation procedures on all test examples from the Binary MNIST dataset, while for ImageNet, following prior work (Lee et al., 2019; Cohen et al., 2019), on every 100th example from validation set.
Revised: In our experiments we ran the certiﬁcation procedures on all test examples from the Binary MNIST dataset, while for ImageNet, following prior work (Lee et al., 2019; Cohen et al., 2019), on every 100th example from validation set.

Original: The proportion of the examples for which (cid:15) accuracy can be certiﬁed are reported in Table 2 for various values of (cid:15) .
Revised: The proportion of the examples for which (cid:15) accuracy can be certiﬁed are reported in Table 2 for various values of (cid:15) .

Original: Comparing with the optimal certiﬁcates obtained by the certiﬁcation method of Lee et al.
Revised: Comparing with the optimal certiﬁcates obtained by the certiﬁcation method of Lee et al.

Original: , the table shows that our bounds remains tight for Binary MNIST up until (cid:15) = 3 , and we do so with 130 times speed-up.
Revised: , the table shows that our bounds remains tight for Binary MNIST up until (cid:15) = 3 , and we do so with 130 times speed-up.

Original: For ImageNet, our bounds differ only about 7–15% in terms of accuracy, and we obtain this with approximatelymillion times speed-up.
Revised: For ImageNet, our bounds differ only about 7–15% in terms of accuracy, and we obtain this with approximatelymillion times speed-up.

Original: [Table]
Revised: [Table]

Original: C ERTIFICATION FOR A UDIO C LASSIFICATION : L IBRISPEECH
Revised: C ERTIFICATION FOR A UDIO C LASSIFICATION : L IBRISPEECH

Original: Audio classiﬁcation systems have been shown to be susceptible to adversarial attacks (Qin et al., 2019).
Revised: Audio classiﬁcation systems have been shown to be susceptible to adversarial attacks (Qin et al., 2019).

Original: However, building audio classiﬁers that are provably robust to adversarial attacks has been hard due to the complexity of audio processing architectures.
Revised: However, building audio classiﬁers that are provably robust to adversarial attacks has been hard due to the complexity of audio processing architectures.

Original: We take a step towards provably robust audio classiﬁers by showing that our approach can certify robustness of a classiﬁer trained for speaker recognition on a state-of-the-art model for this task.
Revised: We take a step towards provably robust audio classiﬁers by showing that our approach can certify robustness of a classiﬁer trained for speaker recognition on a state-of-the-art model for this task.

Original: We focus on (cid:96) 0 perturbations that zero out a fraction of the audio sample, as they correspond to missing data in an audio signal.
Revised: We focus on (cid:96) 0 perturbations that zero out a fraction of the audio sample, as they correspond to missing data in an audio signal.

Original: Missing data can occur due to errors in recording audio or packets dropped while transmitting an audio signal over a network and is a common issue (Turner, 2010;
Revised: Missing data can occur due to errors in recording audio or packets dropped while transmitting an audio signal over a network and is a common issue (Turner, 2010;

Original: Smaragdis et al., 2009).
Revised: Smaragdis et al., 2009).

Original: In principle, the method of Lee et al.
Revised: In principle [Table] the method of Lee et al.

Original: (2019) is applicable to compute robustness certiﬁcates, but at an impractically large computational cost, since the computation needs to be repeated whenever an input of a new length (for which a certiﬁcate has not previously been computed) arrives.
Revised: (2019) is applicable to compute robustness certiﬁcates [Table] but at an impractically large computational cost, since the computation needs to be repeated whenever an input of a new length (for which a certiﬁcate has not previously been computed) arrives.

Original: Concretely, this constitutes an O ( d 3 ) computation for the length d ranging from 38 to 522,320 (the set of audio sequence lengths observed in the Librispeech test dataset (Panayotov et al., 2015)).
Revised: Concretely, this constitutes an O ( d 3 ) computation for the length d ranging from 38 to 522,320 (the set of audio sequence lengths observed in the Librispeech test dataset (Panayotov et al., 2015)).

Original: The results are shown in Table 3.
Revised: The results are shown in Table 3.

Original: To the best of our knowledge, these are the ﬁrst results showing certiﬁed robustness of an audio classiﬁer.
Revised: To the best of our knowledge, these are the ﬁrst results showing certiﬁed robustness of an audio classiﬁer.

Original: We believe this is a signiﬁcant advance towards certiﬁcation of classiﬁers in audio and classiﬁers operating on variable-length inputs more generally.
Revised: We believe this is a signiﬁcant advance towards certiﬁcation of classiﬁers in audio and classiﬁers operating on variable-length inputs more generally

Original: [Table]
Revised: [Table]

Original: we created a corpus of sentence utterances from ten different speakers.

Original: The classiﬁcation task is given an audio sample, to predict whom is speaking.

Original: The test set consisted of 30 audio samples for each of the ten speakers.

Original: We use a DeepSpeaker architecture (Li et al., 2017), trained with the Adam optimizer ( β 1 = 0 .

Original: 9 , β 2 = 0 .

Original: for 50,000 steps with a learning rate of 0.0001.

Original: The architecture is the same as that of Li et al.

Original: (2017), except for changing the number of neurons in the ﬁnal layer for speaker identiﬁcation with ten classes.

Original: Three models were trained with smoothing values of p = 0 .

Original: 5 , p = 0 .

Original: 7 , and p = 0 .

Original: 9 , respectively, and we used the same values for certiﬁcation.

Original: Certiﬁcation was performed using N = 1000 , ˜ N = 1000000 , ζ = .

Original: 99 using M = 1 R´enyi divergence, with α tuned to obtain the best certiﬁcate.

Original: The proportion of samples with certiﬁed robustness for different accuracy values are reported, computed on 300 test set samples.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have introduced a general framework for black-box veriﬁcation using f -divergence constraints.
Revised: We have introduced a general framework for black-box veriﬁcation using f -divergence constraints.

Original: The framework improves upon state-of-the-art results on both image classiﬁcation and audio tasks by a signiﬁcant margin in terms of robustness certiﬁcates or computation time.
Revised: The framework improves upon state-of-the-art results on both image classiﬁcation and audio tasks by a signiﬁcant margin in terms of robustness certiﬁcates or computation time.

Original: We believe that our framework can potentially enable scalable computation of robustness veriﬁcation for more complex predictors and structured perturbations that can be modeled using f-divergence constraints.
Revised: We believe that our framework can potentially enable scalable computation of robustness veriﬁcation for more complex predictors and structured perturbations that can be modeled using f-divergence constraints.

Original: A A PPENDIX A. A DVERSARIAL SPECIFICATION FOR SMOOTHED CLASSIFIERS
Revised: A A PPENDIX A. A DVERSARIAL SPECIFICATION FOR SMOOTHED CLASSIFIERS

Original: Note that for any ν ∈ D x,(cid:15) we have
Revised: Note that for any ν ∈ D x, [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, E X ∼ ν
Revised: Therefore, E X ∼ ν

Original: [ φ c,c (cid:48)
Revised: [ φ c,c (cid:48)

Original: ( X )] ≥ 0 for all c (cid:48) ∈
Revised: ( X )] ≥ 0 for all c (cid:48) ∈

Original: Y \ { c } is equivalent to c ∈ arg max y
Revised: Y \ { c } is equivalent to c ∈ arg max y

Original: ∈Y P X ∼ ν
Revised: ∈Y P X ∼ ν

Original: [ h ( X ) =
Revised: [ h ( X ) =

Original: y ] .
Revised: y ] .

Original: For ν = µ ( x (cid:48) ) , this means that h s ( x (cid:48) )
Revised: For ν = µ ( x (cid:48) ) , this means that h s ( x (cid:48) )

Original: =
Revised: =

Original: c (assuming the argmax is unique).
Revised: c (assuming the argmax is unique).

Original: In other words, E X ∼ ν
Revised: In other words, E X ∼ ν

Original: [ φ c,c (cid:48)
Revised: [ φ c,c (cid:48)

Original: ( X )] ≥ 0 for all c (cid:48)
Revised: ( X )] ≥ 0 for all c (cid:48)

Original: ∈
Revised: ∈

Original: Y \ { c } and all µ ( x (cid:48) )
Revised: Y \ { c } and all µ ( x (cid:48) )

Original: ∈ D x,(cid:15)
Revised: ∈ D x,(cid:15)

Original: if and only if h s ( x (cid:48) ) =
Revised: if and only if h s ( x (cid:48) ) =

Original: c for all x (cid:48)
Revised: c for all x (cid:48)

Original: such that d ( x, x (cid:48) ) ≤ (cid:15) , proving the required robustness certiﬁcate.
Revised: such that d ( x, x (cid:48) ) ≤ (cid:15) , proving the required robustness certiﬁcate.

Original: A.1. R OBUSTNESS SPECIFICATION FOR SMOOTHED SOFT CLASSIFIERS
Revised: A.1. R OBUSTNESS SPECIFICATION FOR SMOOTHED SOFT CLASSIFIERS

Original: Consider a soft classiﬁer H :
Revised: Consider a soft classiﬁer H :

Original: X → P ( Y ) that for each input x returns a probability distribution H ( x ) over the set of potential labels
Revised: X → P ( Y ) that for each input x returns a probability distribution H ( x ) over the set of potential labels

Original: Y
Revised: Y

Original: (e.g. H might represent the outputs of the soft-max layer of a neural network).
Revised: (e.g. H might represent the outputs of the soft-max layer of a neural network).

Original: As in the case of hard classiﬁers, our methodology can be used to provide robustness guarantees for smoothed soft classiﬁers obtained by applying a smoothing measure µ ( x ) to the input.
Revised: As in the case of hard classiﬁers, our methodology can be used to provide robustness guarantees for smoothed soft classiﬁers obtained by applying a smoothing measure µ ( x ) to the input.

Original: In this case, the smoothed classiﬁer is again a soft classiﬁer given by H s ( x ) =
Revised: In this case, the smoothed classiﬁer is again a soft classiﬁer given by H s ( x ) =

Original: E X ∼ µ ( x ) [ H ( X )] .
Revised: E X ∼ µ ( x ) [ H ( X )] .

Original: Let x be a ﬁxed input point and write p
Revised: Let x be a ﬁxed input point and write p

Original: = H s ( x ) ∈ P ( Y ) to denote the distribution over labels.
Revised: = H s ( x ) ∈ P ( Y ) to denote the distribution over labels.

Original: A number of robustness properties about the soft classiﬁer H s at x can be phrased in terms of
Revised: A number of robustness properties about the soft classiﬁer H s at x can be phrased in terms of

Original: Deﬁnition 2.2.
Revised: Deﬁnition 2.2.

Original: For example, let Y = { 1 , . . .
Revised: For example, let Y = { 1 , . . .

Original: , K } and suppose that p 1 ≥ p 2 ≥ · · · ≥ p K so that { 1 , . . .
Revised: , K } and suppose that p 1 ≥ p 2 ≥ · · · ≥ p K so that { 1 , . . .

Original: , k } are the top k labels at x .
Revised: , k } are the top k labels at x .

Original: Then we can verify that the set of top k labels will not change when moving the input from x to x (cid:48) with (cid:107) x − x (cid:48) (cid:107) ≤ (cid:15) by deﬁning the speciﬁcations
Revised: Then we can verify that the set of top k labels will not change when moving the input from x to x (cid:48) with (cid:107) x − x (cid:48) (cid:107) ≤ (cid:15) by deﬁning the speciﬁcations

Original: φ i,j ( z ) =
Revised: φ i,j ( z ) =

Original: H ( z ) i − H
Revised: H ( z ) i − H

Original: ( z ) j for i ∈ [1 , k ] and j ∈ [ k + 1 , K ] , and showing that all of these φ i,j are robustly certiﬁed at µ ( x ) with respect to the set D x,(cid:15) deﬁned above.
Revised: ( z ) j for i ∈ [1 , k ] and j ∈ [ k + 1 , K ] , and showing that all of these φ i,j are robustly certiﬁed at µ ( x ) with respect to the set D x,(cid:15) deﬁned above.

Original: The case k = 1 corresponds to robustness of the standard classiﬁcation rule outputting the label with the largest score.
Revised: The case k = 1 corresponds to robustness of the standard classiﬁcation rule outputting the label with the largest score.

Original: Another example is robustness of classiﬁers which are allowed to abstain.
Revised: Another example is robustness of classiﬁers which are allowed to abstain.

Original: For example, suppose we build a hard classiﬁer ˜ h out of H s which returns the label with the maximum score as long as the gap between this score and the score of any other label is at least γ ; otherwise it produces no output.
Revised: For example, suppose we build a hard classiﬁer ˜ h out of H s which returns the label with the maximum score as long as the gap between this score and the score of any other label is at least γ ; otherwise it produces no output.

Original: Then we can certify that ˜ h will not abstain and return the label c = arg max y ∈Y p
Revised: Then we can certify that ˜ h will not abstain and return the label c = arg max y ∈Y p

Original: y at any point close to x by showing that every φ c (cid:48) ( z ) =
Revised: y at any point close to x by showing that every φ c (cid:48) ( z ) =

Original: H ( z ) c − H ( z ) c (cid:48) − γ , c (cid:48) (cid:54)
Revised: H ( z ) c − H ( z ) c (cid:48) − γ , c (cid:48) (cid:54)

Original: =
Revised: =

Original: c , is robustly certiﬁed at µ ( x ) with respect to D x,(cid:15) .
Revised: c , is robustly certiﬁed at µ ( x ) with respect to D x,(cid:15) .

Original: A. B ACKGROUND ON F - DIVERGENCES
Revised: A. B ACKGROUND ON F - DIVERGENCES

Original: A number of well-known properties about f -divergences are used throughout the paper, both explicitly and implicitly.
Revised: A number of well-known properties about f -divergences are used throughout the paper, both explicitly and implicitly.

Original: Here we review such properties for the readers’ convenience.
Revised: Here we review such properties for the readers’ convenience.

Original: Proofs and further details can be found in, e.g., (Csisz ´ ar et al., 2004; Liese & Vajda, 2006).
Revised: Proofs and further details can be found in, e.g., (Csisz ´ ar et al., 2004; Liese & Vajda, 2006).

Original: Recall that the f -divergences can be deﬁned for any convex function f : R + → R such that f (1) = 0 .
Revised: Recall that the f -divergences can be deﬁned for any convex function f : R + → R such that f (1) = 0 .

Original: We note that this requirement holds without loss of generality as the map x
Revised: We note that this requirement holds without loss of generality as the map x

Original: (cid:55)→ f ( x ) − f (1) is convex whenever f is convex.
Revised: (cid:55)→ f ( x ) − f (1) is convex whenever f is convex.

Original: Any f -divergence D f satisﬁes the following:
Revised: Any f -divergence D f satisﬁes the following:

Original: D f ( ν (cid:107) ρ ) ≥ 0 .
Revised: D f ( ν (cid:107) ρ ) ≥ 0 .

Original: D f
Revised: D f

Original: ( ρ (cid:107) ρ ) = 0 , and D f ( ν (cid:107) ρ ) = 0 implies ν
Revised: ( ρ (cid:107) ρ ) = 0 , and D f ( ν (cid:107) ρ ) = 0 implies ν

Original: = ρ whenever f is strictly convex at 1 . 3.
Revised: = ρ whenever f is strictly convex at 1 . 3.

Original: D f ( F ∗ ( ν ) (cid:107) F ∗ ( ρ )) ≤ D
Revised: D f ( F ∗ ( ν ) (cid:107) F ∗ ( ρ )) ≤ D

Original: f ( ν (cid:107) ρ ) for any function F , where F ∗ ( ρ ) is the push-forward of ρ . 4.
Revised: f ( ν (cid:107) ρ ) for any function F , where F ∗ ( ρ ) is the push-forward of ρ . 4.

Original: D f
Revised: D f

Original: ( ν (cid:107) ρ ) =
Revised: ( ν (cid:107) ρ ) =

Original: D ¯ f ( ρ (cid:107) ν ) where ¯ f ( u ) =
Revised: D ¯ f ( ρ (cid:107) ν ) where ¯ f ( u ) =

Original: uf (cid:0) 1 u (cid:1) is again convex with ¯ f (1)
Revised: uf (cid:0) 1 u (cid:1) is again convex with ¯ f (1)

Original: = 0 .
Revised: = 0 .

Original: A. B OUNDING R ENYI DIVERGENCES
Revised: A. B OUNDING R ENYI DIVERGENCES

Original: We being with the optimization problem
Revised: We being with the optimization problem

Original: [Equation]
Revised: [Equation]

Original: subject to (cid:107)
Revised: subject to (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:15) (10)
Revised: (cid:15) (10)

Original: Since we have R α ( µ ( x (cid:48) )
Revised: Since we have R α ( µ ( x (cid:48) )

Original: (cid:107) µ ( x ))
Revised: (cid:107) µ ( x ))

Original: = (cid:80) i R α ( µ i ( x (cid:48) i ) (cid:107) µ i ( x i )) .
Revised: = (cid:80) i R α ( µ i ( x (cid:48) i ) (cid:107) µ i ( x i )) .

Original: The constraint can be rewritten as
Revised: The constraint can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Forming the Lagrangian relaxation, we obtain
Revised: Forming the Lagrangian relaxation, we obtain

Original: [Equation]
Revised: [Equation]

Original: where the constraint | x (cid:48)
Revised: where the constraint | x (cid:48)

Original: i − x i | ≤ (cid:15) is implied by (cid:107) x (cid:48) − x (cid:107) p ≤ (cid:15) .
Revised: i − x i | ≤ (cid:15) is implied by (cid:107) x (cid:48) − x (cid:107) p ≤ (cid:15) .

Original: We can maximize separately over each x (cid:48)
Revised: We can maximize separately over each x (cid:48)

Original: i to obtain
Revised: i to obtain

Original: [Equation]
Revised: [Equation]

Original: By weak duality, for any γ ≥ 0 , this is an upper bound on Eq.
Revised: By weak duality, for any γ ≥ 0 , this is an upper bound on Eq.

Original: We can minimize this bound over γ ≥ 0 to obtain the tightest bound.
Revised: We can minimize this bound over γ ≥ 0 to obtain the tightest bound.

Original: The minimization over x (cid:48)
Revised: The minimization over x (cid:48)

Original: i for each i can be solved in closed-form or via as simple 1-dimensional minimization problem for most smoothing measures.
Revised: i for each i can be solved in closed-form or via as simple 1-dimensional minimization problem for most smoothing measures.

Original: A. P ROOF OF THEOREM
Revised: A. P ROOF OF THEOREM

Original: For simplicity of exposition (and to avoid measure theoretic issues), we focus on the case where ν, ρ have well deﬁned densities ν ( x ) , ρ ( x ) such that ρ ( x ) > 0 whenever ν ( x ) > 0 .
Revised: For simplicity of exposition (and to avoid measure theoretic issues), we focus on the case where ν, ρ have well deﬁned densities ν ( x ) , ρ ( x ) such that ρ ( x ) > 0 whenever ν ( x ) > 0 .

Original: We begin by rewriting the optimization problem in terms of the likelihood ratio r ( X ) =
Revised: We begin by rewriting the optimization problem in terms of the likelihood ratio r ( X ) =

Original: ν ( X ) ρ [Equation] X ) : We have
Revised: ν ( X ) ρ ( X ) : We have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst two equalities follow directly by plugging in ν ( X )
Revised: where the ﬁrst two equalities follow directly by plugging in ν ( X )

Original: =
Revised: =

Original: ρ ( X ) r
Revised: ρ ( X ) r

Original: ( X ) and the third is obtained using the fact that ν is a probability measure.
Revised: ( X ) and the third is obtained using the fact that ν is a probability measure.

Original: Using these relations, the optimization over ν can be rewritten as
Revised: Using these relations, the optimization over ν can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: where r ≥ 0 denotes that r ( x ) ≥ 0 ∀ x ∈ X .
Revised: where r ≥ 0 denotes that r ( x ) ≥ 0 ∀ x ∈ X .

Original: The optimization over r is a convex optimization problem and can be solved using Lagrangian duality as follows – we ﬁrst dualize the constraints on r to obtain
Revised: The optimization over r is a convex optimization problem and can be solved using Lagrangian duality as follows – we ﬁrst dualize the constraints on r to obtain

Original: [Equation]
Revised: [Equation]

Original: By strong duality, it holds that maximizing the ﬁnal expression with respect to λ ≥ 0 , κ achieves the optimal value in Eq.
Revised: By strong duality, it holds that maximizing the ﬁnal expression with respect to λ ≥ 0 , κ achieves the optimal value in Eq.

Original: Thus, if the optimal value is smaller than 0 , the speciﬁcation is not robustly certiﬁed and if it is larger than 0 , the speciﬁcation is robustly certiﬁed.
Revised: Thus, if the optimal value is smaller than 0 , the speciﬁcation is not robustly certiﬁed and if it is larger than 0 , the speciﬁcation is robustly certiﬁed.

Original: Finally, since we are ultimately interested in proving that the objective is non-negative, we can restrict ourselves to λ ≥such that (cid:80) i λ i = 1 (since if the optimal λ added up to something larger, we could simply rescale the values to add up to 1 and multiply κ by the same scaling factor without changing the sign of the objective function).
Revised: Finally, since we are ultimately interested in proving that the objective is non-negative, we can restrict ourselves to λ ≥such that (cid:80) i λ i = 1 (since if the optimal λ added up to something larger, we could simply rescale the values to add up to 1 and multiply κ by the same scaling factor without changing the sign of the objective function).

Original: This concludes the proof of correctness of the certiﬁcate Eq. 6.
Revised: This concludes the proof of correctness of the certiﬁcate Eq. 6.

Original: A. P ROOF OF THEOREM
Revised: A. P ROOF OF THEOREM

Original: For the next result, we observe that when φ is ternary valued, the optimization over κ, λ above can be written as
Revised: For the next result, we observe that when φ is ternary valued, the optimization over κ, λ above can be written as

Original: [Equation]
Revised: [Equation]

Original: where θ a = P X ∼ ρ [ φ ( X ) =
Revised: where θ a = P X ∼ ρ [ φ ( X ) =

Original: +1] , θ b =
Revised: +1] , θ b =

Original: P X ∼ ρ [ φ ( X ) = − 1] , θ c = P X ∼ ρ
Revised: P X ∼ ρ [ φ ( X ) = − 1] , θ c = P X ∼ ρ

Original: [ φ ( X ) = 0] .
Revised: [ φ ( X ) = 0] .

Original: Writing out the expression for f ∗ , we obtain
Revised: Writing out the expression for f ∗ , we obtain

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from strong duality.
Revised: where the second inequality follows from strong duality.

Original: The inner maximization is unbounded unless
Revised: The inner maximization is unbounded unless

Original: [Equation]
Revised: [Equation]

Original: One thing to note is that, we can rewrite these constraints in terms of ζ = θ (cid:12) γ , i.e. ζ y = θ y γ y for y ∈ { a, b, c } .
Revised: One thing to note is that, we can rewrite these constraints in terms of ζ = θ (cid:12) γ , i.e. ζ y = θ y γ y for y ∈ { a, b, c } .

Original: These constraints ensure that ζ is a probability distribution over { +1 , 0 , − 1 } and furthermore (cid:88)
Revised: These constraints ensure that ζ is a probability distribution over { +1 , 0 , − 1 } and furthermore (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Thus, the second constraint above is equivalent to D f i ( ζ (cid:107) θ ) ≤ (cid:15) i .
Revised: Thus, the second constraint above is equivalent to D f i ( ζ (cid:107) θ ) ≤ (cid:15) i .

Original: Writing the optimization problem in terms of ζ , we obtain
Revised: Writing the optimization problem in terms of ζ , we obtain

Original: [Equation]
Revised: [Equation]

Original: A. C LOSED - FORM CERTIFICATES FOR THE INFORMATION - LIMITED SETTING
Revised: A. C LOSED - FORM CERTIFICATES FOR THE INFORMATION - LIMITED SETTING

Original: In this section we present closed-form certiﬁcates for the information-limited setting which can be derived from Theorem 2 for M = 1 .
Revised: In this section we present closed-form certiﬁcates for the information-limited setting which can be derived from Theorem 2 for M = 1 .

Original: The results are summarized in Table 4.
Revised: The results are summarized in Table 4.

Original: In the next subsections we present the derivation of the certiﬁcates for Hockey-Stick and R´enyi divergences.
Revised: In the next subsections we present the derivation of the certiﬁcates for Hockey-Stick and R´enyi divergences.

Original: The certiﬁcates for the KL and inﬁnite R´enyi divergence can be derived by taking limits of the R´enyi certiﬁcate (as α → 1 , ∞ respectively).
Revised: The certiﬁcates for the KL and inﬁnite R´enyi divergence can be derived by taking limits of the R´enyi certiﬁcate (as α → 1 , ∞ respectively).

Original: A.6. C ALCULATION OF CERTIFICATE FOR HOCKEY - STICK DIVERGENCE
Revised: A.6. C ALCULATION OF CERTIFICATE FOR HOCKEY - STICK DIVERGENCE

Original: The function f ( u ) = max( u − β, 0) − max(1 − β, 0) is a convex function with f (1) = 0 .
Revised: The function f ( u ) = max( u − β, 0) − max(1 − β, 0) is a convex function with f (1) = 0 .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: KL divergence
Revised: KL divergence

Original: [Table]
Revised: [Table]

Original: Inﬁnite R´enyi divergence
Revised: Inﬁnite R´enyi divergence

Original: [Table]
Revised: [Table]

Original: Hockey

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: The inﬁnite R´enyi divergence, deﬁned as sup x log( ν ( x ) /ρ
Revised: The inﬁnite R´enyi divergence, deﬁned as sup x log( ν ( x ) /ρ

Original: ( x )) , is obtained by taking the limit α → ∞ .
Revised: ( x )) , is obtained by taking the limit α → ∞ .

Original: All certiﬁcates depend on the gap between  a and θ b 
Revised: All certiﬁcates depend on the gap between θ a and θ b .

Original: Notation: [ u ] + = max( u, 0) .
Revised: Notation: [ u ] + = max( u, 0) .

Original: The certiﬁcate given by Eq.
Revised: The certiﬁcate given by Eq.

Original: 6 in Theorem 1 for this divergence in the case of a smoothed hard classiﬁer takes the form
Revised: 6 in Theorem 1 for this divergence in the case of a smoothed hard classiﬁer takes the form

Original: [Equation]
Revised: [Equation]

Original: where the speciﬁcation takes the values
Revised: where the speciﬁcation takes the values

Original: [Equation]
Revised: [Equation]

Original: Plugging in the expression for f ∗ the objective function above takes the form
Revised: Plugging in the expression for f ∗ the objective function above takes the form

Original: [Equation]
Revised: [Equation]

Original: where we use the notation [ u ] + = max( u, 0) and assumed the constraints κ ≤ λ − 1 since the objective is −∞ otherwise.
Revised: where we use the notation [ u ] + = max( u, 0) and assumed the constraints κ ≤ λ − 1 since the objective is −∞ otherwise.

Original: If β ≤ 1 , the objective is increasing monotonically in κ , so the optimal value is to set κ to its upper bound λ − 1 .
Revised: If β ≤ 1 , the objective is increasing monotonically in κ , so the optimal value is to set κ to its upper bound λ − 1 .

Original: Plugging this in, the possible values of the derivative with respect to λ are
Revised: Plugging this in, the possible values of the derivative with respect to λ are

Original: [Equation]
Revised: [Equation]

Original: Thus, if (cid:15) ≤ βθ a , the maximum is attained at 2 , if βθ a ≤ (cid:15) ≤ β (1 − θ b ) , the maximum is attained at 1 , else the maximum is attained at 0 , leading to the certiﬁcate:
Revised: Thus, if (cid:15) ≤ βθ a , the maximum is attained at 2 , if βθ a ≤ (cid:15) ≤ β (1 − θ b ) , the maximum is attained at 1 , else the maximum is attained at 0 , leading to the certiﬁcate:

Original: [Equation]
Revised: [Equation]

Revised: ≤ βθ a .

Original: Thus, the certiﬁcate is non-negative only if
Revised: Thus, the certiﬁcate is non-negative only if

Original: [Equation]
Revised: [Equation]

Original: The case β ≥ 1 can be worked out similarly, leading to
Revised: The case β ≥ 1 can be worked out similarly, leading to

Original: [Equation]
Revised: [Equation]

Original: The two cases can be combined as
Revised: The two cases can be combined as

Original: [Equation]
Revised: [Equation]

Original: A.6. C ALCULATION OF CERTIFICATE FOR R´ ENYI DIVERGENCE
Revised: A.6. C ALCULATION OF CERTIFICATE FOR R´ ENYI DIVERGENCE

Original: We consider the cases α ≥ 1 and α ≤ 1 separately.
Revised: We consider the cases α ≥ 1 and α ≤ 1 separately.

Original: Case 1 ( α ≥ 1 )
Revised: Case 1 ( α ≥ 1 )

Original: If α ≥ 1 , the function f ( u ) =
Revised: If α ≥ 1 , the function f ( u ) =

Original: ( u α − 1) is a convex function with f (1) =
Revised: ( u α − 1) is a convex function with f (1) =

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Suppose we have a bound on the R´enyi divergence R α ( ν (cid:107) ρ ) ≤ (cid:15) .
Revised: Suppose we have a bound on the R´enyi divergence R α ( ν (cid:107) ρ ) ≤ (cid:15) .

Original: Then we know D f ( ν (cid:107) ρ ) ≤ exp(( α − 1) (cid:15) ) − 1 .
Revised: Then we know D f ( ν (cid:107) ρ ) ≤ exp(( α − 1) (cid:15) ) − 1 .

Original: Let β = αα − 1 and
Revised: Let β = αα − 1 and

Original: [Equation]
Revised: [Equation]

Original: Then the certiﬁcate Eq.
Revised: Then the certiﬁcate Eq.

Original: 6 simpliﬁes to (after some algebra)
Revised: 6 simpliﬁes to (after some algebra)

Original: [Equation]
Revised: [Equation]

Original: Setting the derivative with respect to λ to 0 and solving for λ
Revised: Setting the derivative with respect to λ to 0 and solving for λ

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: and the optimal certiﬁcate reduces to
Revised: and the optimal certiﬁcate reduces to

Original: [Equation]
Revised: [Equation]

Original: For this number to be positive, we need that κ ≥ 0 and
Revised: For this number to be positive [Equation] we need that κ ≥ 0 and

Original: [Equation]
Revised: [Equation]

Original: The LHS above evaluates to
Revised: The LHS above evaluates to

Original: [Equation]
Revised: [Equation]

Original: where γ = 1 κ ≥ 0 .
Revised: where γ = 1 κ ≥ 0 .

Original: Maximizing this expression with respect to γ , we obtain
Revised: Maximizing this expression with respect to γ , we obtain

Original: [Equation]
Revised: [Equation]

Original: so that the certiﬁcate reduces to
Revised: so that the certiﬁcate reduces to

Original: [Equation]
Revised: [Equation]

Original: Taking logarithms now gives the result.
Revised: Taking logarithms now gives the result.

Original: Case 2 ( 0 ≤ α ≤ 1 )
Revised: Case 2 ( 0 ≤ α ≤ 1 )

Original: When 0 ≤ α ≤ 1 , the function f ( u ) =
Revised: When 0 ≤ α ≤ 1 , the function f ( u ) =

Original: (1 − u α ) is a convex function with f (1) = 0 .
Revised: (1 − u α ) is a convex function with f (1) = 0 .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: if u ≤ 0 , ∞ otherwise .
Revised: if u ≤ 0 , ∞ otherwise .

Original: Further, a bound R α ( ν (cid:107) ρ ) ≤ (cid:15) implies
Revised: Further, a bound R α ( ν (cid:107) ρ ) ≤ (cid:15) implies

Original: D f ( ν (cid:107) ρ ) ≤ 1 − exp(( α − 1) (cid:15) ) .
Revised: D f ( ν (cid:107) ρ ) ≤ 1 − exp(( α − 1) (cid:15) ) .

Original: Then the certiﬁcate from Eq.
Revised: Then the certiﬁcate from Eq.

Original: 6 reduces to
Revised: 6 reduces to

Original: [Equation]
Revised: [Equation]

Original: with the constraint κ ≤ − 1 (otherwise the certiﬁcate is −∞ ).
Revised: with the constraint κ ≤ − 1 (otherwise the certiﬁcate is −∞ ).

Original: Setting the derivative with respect to λ to 0 and solving for λ
Revised: Setting the derivative with respect to λ to 0 and solving for λ

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Plugging this back into the certiﬁcate and setting β = α 1 − α , we obtain
Revised: Plugging this back into the certiﬁcate and setting β = α 1 − α , we obtain

Original: [Equation]
Revised: [Equation]

Original: For this number to be positive
Revised: For this number to be positive

Original: [Equation]
Revised: [Equation]

Original: we require that
Revised: we require that

Original: [Equation]
Revised: [Equation]

Original: The LHS of the above expression evaluates to
Revised: The LHS of the above expression evaluates to

Original: [Equation]
Revised: [Equation]

Original: where γ = − 1 κ .
Revised: where γ = − 1 κ .

Original: Maximizing this expression over γ ∈ [0 , 1]
Revised: Maximizing this expression over γ ∈ [ [Equation]

Original: [Equation]
Revised: [Equation]

Original: we obtain the ﬁnal certiﬁcate to be
Revised: we obtain the ﬁnal certiﬁcate to be

Original: [Equation]
Revised: [Equation]

Original: Taking logarithms, we obtain
Revised: Taking logarithms, we obtain

Original: [Equation]
Revised: [Equation]

Original: A. I NFORMATION - LIMITED ROBUST CERTIFICATION AND TIGHT RELAXATIONS A.7. P ROOF OF THEOREM
Revised: A. I NFORMATION - LIMITED ROBUST CERTIFICATION AND TIGHT RELAXATIONS A.7. P ROOF OF THEOREM

Original: At a high level, the proof shows that, in the information-limited case, to achieve robust certiﬁcation under an arbitrary set of constraints D it sufﬁces to know the “envelope” of D with respect to all hockey-stick divergences of order β ≥ 0 , i.e. the function β (cid:55)→ max ν ∈D D HS ,β ( ν (cid:107) ρ ) captures all the necessary information to provide information-limited robust certiﬁcation with respect to D .
Revised: At a high level, the proof shows that, in the information-limited case, to achieve robust certiﬁcation under an arbitrary set of constraints D it sufﬁces to know the “envelope” of D with respect to all hockey-stick divergences of order β ≥ 0 , i.e. the function β (cid:55)→ max ν ∈D D HS ,β ( ν (cid:107) ρ ) captures all the necessary information to provide information-limited robust certiﬁcation with respect to D .

Original: We start by considering the following optimization problem:
Revised: We start by considering the following optimization problem:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the information-limited setting, this problem attains the minimum expected value over φ ∈ S .
Revised: In the information-limited setting, this problem attains the minimum expected value over φ ∈ S .

Original: Here 1[ φ ( X ) = 1] denotes the indicator function.
Revised: Here 1[ φ ( X ) = 1] denotes the indicator function.

Original: It will be convenient to write this in a slightly different form: Rather than looking at the outputs of Ψ as the +1 , 0 , − 1 , we look at them as vectors in R 3 :
Revised: It will be convenient to write this in a slightly different form: Rather than looking at the outputs of Ψ as the +1 , 0 , − 1 , we look at them as vectors in R 3 :

Original: [Equation]
Revised: [Equation]

Original: and deﬁne
Revised: and deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then [Equation] we can write the optimization problem Eq.
Revised: Then, we can write the optimization problem Eq.

Original: 12 equivalently as
Revised: 12 equivalently as

Original: [Equation]
Revised: [Equation]

Original: subject to[Equation] ∼
Revised: to

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst consider the minimization over Ψ for a ﬁxed value of ν .
Revised: We ﬁrst consider the minimization over Ψ for a ﬁxed value of ν .

Original: We begin by observing that since the objective is linear, the optimization over Ψ can be replaced with the optimization over the convex hull of the set of Ψ that satisfy the constraints (Bubeck, 2013).
Revised: We begin by observing that since the objective is linear, the optimization over Ψ can be replaced with the optimization over the convex hull of the set of Ψ that satisfy the constraints (Bubeck, 2013).

Original: Since each input x ∈ X can be mapped independently of the rest, the convex hull is simply the cross product of the convex hull at every x , to obtain the constraint set (cid:26)
Revised: Since each input x ∈ X can be mapped independently of the rest, the convex hull is simply the cross product of the convex hull at every x , to obtain the constraint set (cid:26)

Original: [Equation]
Revised: [Equation]

Original: such that 
Revised: such that E

Original: [Equation]
Revised: [Equation]

Original: Therefore, the optimization problem reduces to
Revised: Therefore, the optimization problem reduces to

Revised: [Equation]

Revised: to

Original: [Equation]
Revised: [Equation]

Original: This is a convex optimization problem in Ψ .
Revised: This is a convex optimization problem in Ψ .

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: Considering the dual of this optimization problem with respect to the optimization variable Ψ
Revised: Considering the dual of this optimization problem with respect to the optimization variable Ψ

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Since we can choose Ψ( x ) independently for each x ∈ X
Revised: Since we can choose Ψ( x ) independently for each x ∈ X

Original: [Equation]
Revised: [Equation]

Original: we can minimize each term in the expectation independently to obtain
Revised: we can minimize each term in the expectation independently to obtain

Original: [Equation]
Revised: [Equation]

Original: This implies that the Lagrangian evaluates to
Revised: This implies that the Lagrangian evaluates to

Original: [Equation]
Revised: [Equation]

Original: We now consider two cases:
Revised: We now consider two cases:

Original: Case 1 ( λ a ≥ λ b ≥ 0 )
Revised: Case 1 ( λ a ≥ λ b ≥ 0 )

Original: In this case, we can see that
Revised: In this case, we can see that

Original: [Equation]
Revised: [Equation]

Original: Then, the Lagrangian reduces to λ
Revised: Then, the Lagrangian reduces to λ

Original: [Equation]
Revised: [Equation]

Original: Case 2 ( λ b ≥ λ a ≥ 0 )
Revised: Case 2 ( λ b ≥ λ a ≥ 0 )

Original: In this case, we can see that
Revised: In this case, we can see that

Original: [Equation]
Revised: [Equation]

Original: Then, the Lagrangian reduces to
Revised: Then, the Lagrangian reduces to

Original: [Equation]
Revised: [Equation]

Original: We know that 1 − θ a ≥ θ b and λ b ≥
Revised: We know that 1 − θ a ≥ θ b and λ b ≥

Original: λ a .
Revised: λ a .

Original: If λ b > λ a , by choosing λ (cid:48) a = λ a + κ and λ (cid:48) b =
Revised: If λ b > λ a , by choosing λ (cid:48) a = λ a + κ and λ (cid:48) b =

Original: λ b − κ for some small κ > 0 , we know that the the sum of the ﬁrst three terms would reduce while the ﬁnal term would remain unchanged.
Revised: λ b − κ for some small κ > 0 , we know that the the sum of the ﬁrst three terms would reduce while the ﬁnal term would remain unchanged.

Original: Thus, at the the optimum in this case, we can assume λ a = λ b
Revised: Thus, at the the optimum in this case [Equation] we can assume λ a = λ b

Original: and we obtain
Revised: and we obtain

Original: [Equation]
Revised: [Equation]

Original: Final analysis of the Lagrangian Combining the two cases we can write the dual problem as
Revised: Final analysis of the Lagrangian Combining the two cases we can write the dual problem as

Original: [Equation]
Revised: [Equation]

Original: By strong duality
Revised: By strong duality

Original: [Equation]
Revised: [Equation]

Original: the optimal value of the above problem precisely matches the optimal value of
Revised: the optimal value of the above problem precisely matches the optimal value of

Original: Eq.
Revised: Eq.

Original: 14 (and hence Eq. 12).
Revised: 14 (and hence Eq. 12).

Original: Thus, information limited robust certiﬁcation with respect to D holds if and only if Eq. 15 has a non-negative optimal value for each ν ∈ D .
Revised: Thus, information limited robust certiﬁcation with respect to D holds if and only if Eq. 15 has a non-negative optimal value for each ν ∈ D .

Original: Since we have that
Revised: Since we have that

Original: [Equation]
Revised: [Equation]

Original: information-limited robust certiﬁcation holds if and only if the optimal value of
Revised: information-limited robust certiﬁcation holds if and only if the optimal value of

Original: [Equation]
Revised: [Equation]

Original: [Equation] is non-negative.
Revised: 16) is non-negative.

Original: Further, since the optimal value only depended on the value of D HS ,β ( ν (cid:107) ρ ) for β ≥ 0 , it is equivalent to information-limited robust certiﬁcation with respect to D HS .
Revised: Further, since the optimal value only depended on the value of D HS ,β ( ν (cid:107) ρ ) for β ≥ 0 , it is equivalent to information-limited robust certiﬁcation with respect to D HS .

Original: The above argument also shows that in this case, information-limited robust certiﬁcation with respect to D is equivalent to requiring that the following convex optimization problem has a non-negative optimal value:
Revised: The above argument also shows that in this case, information-limited robust certiﬁcation with respect to D is equivalent to requiring that the following convex optimization problem has a non-negative optimal value:

Original: [Equation]
Revised: [Equation]

Original: Let λ ∗ a , λ ∗ b be the optimal values attained.
Revised: Let λ ∗ a , λ ∗ b be the optimal values attained.

Original: Since this certiﬁcate depends only on the value of two
Revised: Since this certiﬁcate depends only on the value of two

Original: Hockey-stick divergences at λ ∗ a , λ ∗ b , it must coincide with the application of theorem 2 to the constraint set D HS deﬁned by constraints on these hockey-stick divergences (as we know that 2 computes the optimal certiﬁcate for any constraint set deﬁned only by a set of f-divergences).
Revised: Hockey-stick divergences at λ ∗ a , λ ∗ b , it must coincide with the application of theorem 2 to the constraint set D HS deﬁned by constraints on these hockey-stick divergences (as we know that 2 computes the optimal certiﬁcate for any constraint set deﬁned only by a set of f-divergences).

Original: This observation completes the proof.
Revised: This observation completes the proof.

Original: A.7. G AUSSIAN SMOOTHING MEASURES
Revised: A.7. G AUSSIAN SMOOTHING MEASURES

Original: Theorem 4 gives us the optimal limited-information certiﬁcate problem provided that we can compute
Revised: Theorem 4 gives us the optimal limited-information certiﬁcate problem provided that we can compute

Original: [Equation]
Revised: [Equation]

Original: for each β ≥ 0 .
Revised: for each β ≥ 0 .

Original: In particular, when µ is a Gaussian measure µ ( x ) = N (cid:0) x, σ 2 I (cid:1) , we can leverage the following result from Balle & Wang (2018).
Revised: In particular, when µ is a Gaussian measure µ ( x ) = N (cid:0) x, σ 2 I (cid:1) , we can leverage the following result from Balle & Wang (2018).

Original: Lemma 6.
Revised: Lemma 6.

Original: Let Ψ g be the CDF of a standard normal random variable N (0 , 1) .
Revised: Let Ψ g be the CDF of a standard normal random variable N (0 , 1) .

Original: For any β ≥ 0 and x ∈ R
Revised: For any β ≥ 0 and x ∈ R

Original: d
Revised: d

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Applying Eq.
Revised: Applying Eq.

Original: 17 to the expression in Lemma 6 proves Corollary 5.
Revised: 17 to the expression in Lemma 6 proves Corollary 5.

Original: Proof of Corollary 5.
Revised: Proof of Corollary 5.

Original: With the notation from Theorem 4 we have, for β ≥ 0 ,
Revised: With the notation from Theorem 4 we have, for β ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Plugging this expression into Eq.
Revised: Plugging this expression into Eq.

Original: 17 allows us to verify information-limited robust certiﬁcation of
Revised: 17 allows us to verify information-limited robust certiﬁcation of

Original: N (cid:0) x [Equation] , σ
Revised: N (cid:0) x (cid:48) , σ

Original: [Equation]
Revised: [Equation]

Original: (cid:1) with respect to D x,(cid:15)
Revised: (cid:1) with respect to D x,(cid:15)

Original: =
Revised: =

Original: {N (cid:0) x (cid:48) , σ [Equation] (cid:1) :
Revised: {N (cid:0) x (cid:48) , σ 2 I (cid:1) :

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: } by solving
Revised: } by solving

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 9 then follows from setting the derivatives of this expression to 0 with respect to λ a , λ b and imposing the condition that the optimal solution is non-negative.
Revised: 9 then follows from setting the derivatives of this expression to 0 with respect to λ a , λ b and imposing the condition that the optimal solution is non-negative.

Original: To check that Corollary 5 is equivalent to the optimal certiﬁcation in (Cohen et al., 2019, Theorem 1) we ﬁrst recall that, in our notation, their result can be stated as: the class of speciﬁcations
Revised: To check that Corollary 5 is equivalent to the optimal certiﬁcation in (Cohen et al., 2019, Theorem 1) we ﬁrst recall that, in our notation, their result can be stated as: the class of speciﬁcations

Original: S in Deﬁnition 2.1 is information-limited robustly certiﬁed at ρ = N (cid:0) x, σ 2 I (cid:1) with respect to
Revised: S in Deﬁnition 2.1 is information-limited robustly certiﬁed at ρ = N (cid:0) x, σ 2 I (cid:1) with respect to

Original: D x,(cid:15) =
Revised: D x,(cid:15) =

Original: {N (cid:0) x (cid:48) , σ 2 I (cid:1) : (cid:107) x − x (cid:48) (cid:107) 2 ≤ (cid:15) } if and only if[Equation]
Revised: {N (cid:0) x (cid:48) , σ 2 I (cid:1) : (cid:107) x − x (cid:48) (cid:107) 2 ≤ (cid:15) } if and only if

Original: [Equation]
Revised: [Equation]

Original: The equivalence between Eq.
Revised: The equivalence between Eq.

Original: 9 and Eq.
Revised: 9 and Eq.

Original: 18 now follows from the identity 1 − Ψ g
Revised: 18 now follows from the identity 1 − Ψ g

Original: ( θ ) =
Revised: ( θ ) =

Original: Ψ g ( − θ ) and the monotonicity of Ψ g :
Revised: Ψ g ( − θ ) and the monotonicity of Ψ g :

Original: [Equation]
Revised: [Equation]

Original: A.7. R ELATION TO P IXEL DP
Revised: A.7. R ELATION TO P IXEL DP

Original: Pixel differential privacy (pixelDP) was introduced in Lecuyer et al.
Revised: Pixel differential privacy (pixelDP) was introduced in Lecuyer et al.

Original: (2019) using the same similarity measure between distributions used in differential privacy: a distribution-valued function G : R d →
Revised: (2019) using the same similarity measure between distributions used in differential privacy: a distribution-valued function G : R d →

Original: P ( Z ) satisﬁes ( ε, τ ) -pixelDP with respect to (cid:96) p perturbations if for any (cid:107) x − x (cid:48)
Revised: P ( Z ) satisﬁes ( ε, τ ) -pixelDP with respect to (cid:96) p perturbations if for any (cid:107) x − x (cid:48)

Original: (cid:107) p ≤ 1 it holds that
Revised: (cid:107) p ≤ 1 it holds that

Original: D DP ,e ε ( G ( x ) (cid:107) G
Revised: D DP ,e ε ( G ( x ) (cid:107) G

Original: ( x (cid:48) ))
Revised: ( x (cid:48) ))

Original: ≤ τ , where
Revised: ≤ τ , where

Original: [Equation]
Revised: [Equation]

Original: and the supremum is over all (measurable) subsets E of Z .
Revised: and the supremum is over all (measurable) subsets E of Z .

Original: In particular, Lecuyer et al. show that using a smoothing measure µ satisfying pixelDP with respect to (cid:96) p leads to adversarially robust classiﬁers against (cid:96) p perturbations.
Revised: In particular, Lecuyer et al. show that using a smoothing measure µ satisfying pixelDP with respect to (cid:96) p leads to adversarially robust classiﬁers against (cid:96) p perturbations.

Original: To show that their result ﬁts as a particular instance of our framework, take ρ = µ ( x ) and ﬁx ε ≥and τ ∈ [0 , 1] .
Revised: To show that their result ﬁts as a particular instance of our framework, take ρ = µ ( x ) and ﬁx ε ≥and τ ∈ [0 , 1] .

Original: Due to the symmetry of the constraint (cid:107) x − x(cid:48) (cid:107) p ≤ 1 , if µ satisﬁes ( ε, τ ) -pixelDP with respect to (cid:96) p perturbations, then we have the relaxation condition { µ ( x (cid:48) ) : (cid:107) x
Revised: Due to the symmetry of the constraint (cid:107) x − x

Original: [Equation]
Revised: [Equation]

Original:  x
Revised: (cid:107) p ≤ 1 , if µ satisﬁes ( ε, τ ) -pixelDP with respect to (cid:96) p perturbations, then we have the relaxation condition { µ ( x (cid:48) ) : (cid:107) x

Original: [Equation]
Revised: [Equation]

Original: (cid:107) 
Revised: (cid:107) p

Original: [Equation]
Revised: [Equation]

Original: } ⊆ D ε
Revised: } ⊆ D ε

Original: [Equation]
Revised: [Equation]

Original: τ , where
Revised: τ , where

Original: [Equation]
Revised: [Equation]

Original: Now we recall that Barthe & Olmedo (2013) noticed that D DP ,e ε is equivalent to the hockey-stick divergence D HS ,β of order β = e ε .
Revised: Now we recall that Barthe & Olmedo (2013) noticed that D DP ,e ε is equivalent to the hockey-stick divergence D HS ,β of order β = e ε .

Original: Thus, since f -divergences are closed under reversal (property 4 in Appendix A.2), we see that the constraint set D ε,τ can be directly written in the form D F (cf.
Revised: Thus, since f -divergences are closed under reversal (property 4 in Appendix A.2), we see that the constraint set D ε,τ can be directly written in the form D F (cf.

Original: Section 2.2).
Revised: Section 2.2).

Original: The main result in Lecuyer et al.
Revised: The main result in Lecuyer et al.

Original: (2019) is a limited-information black-box certiﬁcation method for smoothed classiﬁers.
Revised: (2019) is a limited-information black-box certiﬁcation method for smoothed classiﬁers.

Original: The resulting certiﬁcate for, which provides certiﬁcation with respect to D ε,τ , is given by
Revised: The resulting certiﬁcate for, which provides certiﬁcation with respect to D ε,τ , is given by

Original: [Equation]
Revised: [Equation]

Original: For comparison, the certiﬁcate we obtain for the relaxation { ν : D DP ,e ε ( ν (cid:107) ρ ) ≤ τ } of D ε,τ (HS certiﬁcate in Table 4) already improves on the certiﬁcate by Lecuyer et al.
Revised: For comparison, the certiﬁcate we obtain for the relaxation { ν : D DP ,e ε ( ν (cid:107) ρ ) ≤ τ } of D ε,τ (HS certiﬁcate in Table 4) already improves on the certiﬁcate by Lecuyer et al.

Original: whenever θ a − θ b ≥ ( β − 1)(1 − θ a − θ b ) , which, e.g., always holds in the binary classiﬁcation case.
Revised: whenever θ a − θ b ≥ ( β − 1)(1 − θ a − θ b ) , which, e.g., always holds in the binary classiﬁcation case.

Original: Furthermore, since
Revised: Furthermore, since

Original: Theorem 2 provides optimal certiﬁcates for D , we have the following result.
Revised: Theorem 2 provides optimal certiﬁcates for D , we have the following result.

Original: Corollary 7.
Revised: Corollary 7.

Original: The optimal certiﬁcates for the constraint set D (cf.
Revised: The optimal certiﬁcates for the constraint set D (cf.

Original: Eq. 20) obtained from Theoremare stronger than those obtained from Eq. 21.
Revised: Eq. 20) obtained from Theoremare stronger than those obtained from Eq. 21.

Original: A. E FFICIENT SAMPLING AND F - DIVERGENCE COMPUTATION FOR NORM - BASED SMOOTHING MEASURES
Revised: A. E FFICIENT SAMPLING AND F - DIVERGENCE COMPUTATION FOR NORM - BASED SMOOTHING MEASURES

Original: Lemma 8.
Revised: Lemma 8.

Original: The smoothing measure µ : X (cid:55)→ P ( X ) with density µ ( x )[ z ] ∝ exp( −(cid:107) z − x (cid:107) ) satisﬁes
Revised: The smoothing measure µ : X (cid:55)→ P ( X ) with density µ ( x )[ z ] ∝ exp( −(cid:107) z − x (cid:107) ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: if (cid:107) x (cid:107) is any norm.
Revised: if (cid:107) x (cid:107) is any norm.

Original: Further, if f is convex function with f (1) = 

Revised: Further, if f is convex function with f (1) =

Original: such that f (cid:0) 1 u (cid:1) is convex and monotonically increasing in u , then
Revised: such that f (cid:0) 1 u (cid:1) is convex and monotonically increasing in u , then

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: By the triangle inequality, we have
Revised: By the triangle inequality, we have

Original: [Equation]
Revised: [Equation]

Original: so that
Revised: so that

Original: [Equation]
Revised: [Equation]

Original: Similarly, for f that satisfy the conditions of the theorem, it can be shown
Revised: Similarly, for f that satisfy the conditions of the theorem, it can be shown

Original: that D f ( µ ( x (cid:48) )
Revised: that D f ( µ ( x (cid:48) )

Original: (cid:107) µ ( x )) is convex in x (cid:48)
Revised: (cid:107) µ ( x )) is convex in x (cid:48)

Original: so that its maximum over the convex set (cid:107) x (cid:48) − x (cid:107) ≤ (cid:15) is attained on the boundary.
Revised: so that its maximum over the convex set (cid:107) x (cid:48) − x (cid:107) ≤ (cid:15) is attained on the boundary.

Original: For several norms, the optimization problem in Eq.
Revised: For several norms, the optimization problem in Eq.

Original: 22 can be solved in closed form.
Revised: 22 can be solved in closed form.

Original: These include (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ norms and the matrix spectral norm and nuclear norm (the ﬁnal two are relevant when X is a space of matrices).
Revised: These include (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ norms and the matrix spectral norm and nuclear norm (the ﬁnal two are relevant when X is a space of matrices).

Original: The results are documented in Table 5.
Revised: The results are documented in Table 5.

Original: Thus, every f -divergence that meets the conditions of Lemma 8 can be estimated efﬁciently for these norms.
Revised: Thus, every f -divergence that meets the conditions of Lemma 8 can be estimated efﬁciently for these norms.

Original: In particular, the divergences that are induced by the functions ˜ f
Revised: In particular, the divergences that are induced by the functions ˜ f

Original: ( u − α ) for any monotonic convex function ˜ f and α ≥ 0 satisfy this constraint.
Revised: ( u − α ) for any monotonic convex function ˜ f and α ≥ 0 satisfy this constraint.

Original: This gives us a very ﬂexible class of f -divergences that can be efﬁciently estimated for these norm-based smoothing measures.
Revised: This gives us a very ﬂexible class of f -divergences that can be efﬁciently estimated for these norm-based smoothing measures.

Original: Constraint on δ Bound on Eq.
Revised: Constraint on δ Bound on Eq.

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: X

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Efﬁcient sampling The only other requirement for obtaining a certiﬁcate computationally is to be able to sample from µ ( x ) to estimate θ a , θ b .
Revised: Efﬁcient sampling The only other requirement for obtaining a certiﬁcate computationally is to be able to sample from µ ( x ) to estimate θ a , θ b .

Original: Since µ ( x ) is log-concave, there are general purpose polynomial time algorithms for sampling from this measure.
Revised: Since µ ( x ) is log-concave, there are general purpose polynomial time algorithms for sampling from this measure.

Original: However, for most norms, more efﬁcient methods exist, as outlined below.
Revised: However, for most norms, more efﬁcient methods exist, as outlined below.

Original: The random variable X ∼ µ ( x ) can be obtained as X =
Revised: The random variable X ∼ µ ( x ) can be obtained as X =

Original: x + Z with Z ∼ µ (0) .
Revised: x + Z with Z ∼ µ (0) .

Original: Thus, to sample from µ ( x ) for any x it is enough to be able to sample from µ (0) .
Revised: Thus, to sample from µ ( x ) for any x it is enough to be able to sample from µ (0) .

Original: For (cid:107)·(cid:107) 1 , this reduces to sampling from a Laplace distribution which can be done easily.
Revised: For (cid:107)·(cid:107) 1 , this reduces to sampling from a Laplace distribution which can be done easily.

Original: For (cid:107)·(cid:107) ∞ , (Steinke & Ullman, 2016) give the following efﬁcient sampling procedure: ﬁrst sample r from a Gamma distribution with shape d + 1 and mean d + 1 , i.e. r ∼ Γ( d + 1 , 1) , and then sample each Z i ,
Revised: For (cid:107)·(cid:107) ∞ , (Steinke & Ullman, 2016) give the following efﬁcient sampling procedure: ﬁrst sample r from a Gamma distribution with shape d + 1 and mean d + 1 , i.e. r ∼ Γ( d + 1 , 1) , and then sample each Z i ,

Original: i ∈
Revised: i ∈

Original: [ d ] , uniformly from [ − r, r ] .
Revised: [ d ] , uniformly from [ − r, r ] .

Original: Theorem 9 gives a short proof of correctness for this procedure.
Revised: Theorem 9 gives a short proof of correctness for this procedure.

Original: Theorem 10 also has a similar result for the case of (cid:107)·(cid:107) 2 and Table 5 lists the sampling procedures for several norms.
Revised: Theorem 10 also has a similar result for the case of (cid:107)·(cid:107) 2 and Table 5 lists the sampling procedures for several norms.

Original: Theorem 9.
Revised: Theorem 9.

Original: The random variable Z ∈ R d obtained by ﬁrst sampling R ∼ Γ( d + 1 , 1) and then sampling each Z i , i ∈
Revised: The random variable Z ∈ R d obtained by ﬁrst sampling R ∼ Γ( d + 1 , 1) and then sampling each Z i , i ∈

Original: [ d ] , uniformly from [ − R, R ] has density ∝ e −(cid:107) z (cid:107) ∞ .
Revised: [ d ] , uniformly from [ − R, R ] has density ∝ e −(cid:107) z (cid:107) ∞ .

Original: Proof.
Revised: Proof.

Original: We ﬁrst compute the normalization constant for a density of the form ∝ e −(cid:107) z (cid:107) ∞ as follows:
Revised: We ﬁrst compute the normalization constant for a density of the form ∝ e −(cid:107) z (cid:107) ∞ as follows:

Original: [Equation]
Revised: [Equation]

Original: Next we show the density of Z satisﬁes p Z ( z ) =
Revised: Next we show the density of Z satisﬁes p Z ( z ) =

Original: e −(cid:107)
Revised: e −(cid:107)

Original: z (cid:107) ∞ / ( d !2 d ) by noting that conditioned on
Revised: z (cid:107) ∞ / ( d !2 d ) by noting that conditioned on

Original: R = r
Revised: R = r

Original: we have p Z | R = r ( z ) = 

Revised: we have p Z | R = r ( z ) =

Original: [ (cid:107) z (cid:107) ∞ ≤ r ] / (2 r ) d because of the uniform sampling used in each 
Revised: [ (cid:107) z (cid:107) ∞ ≤ r ] / (2 r ) d because of the uniform sampling used in each yields

Original: [Equation]
Revised: [Equation]

Original: Theorem 10.
Revised: Theorem 10.

Original: The random variable Z ∈ R d obtained by ﬁrst sampling Z (cid:48) ∼ N (0 , I ) and R ∼ Γ p ( d, 1) and then taking Z =
Revised: The random variable Z ∈ R d obtained by ﬁrst sampling Z (cid:48) ∼ N (0 , I ) and R ∼ Γ p ( d, 1) and then taking Z =

Original: R Z
Revised: R Z

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation]
Revised: [Equation]

Original: 2 has density ∝ e −(cid:107)
Revised: 2 has density ∝ e −(cid:107)

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: p 2 .
Revised: p 2 .

Original: Here Γ p ( d, a ) denotes the generalized
Revised: Here Γ p ( d, a ) denotes the generalized

Original: Gamma distribution of order p > 0 with shape d and scale a .
Revised: Gamma distribution of order p > 0 with shape d and scale a .

Original: Proof.
Revised: Proof.

Original: First note that W =
Revised: First note that W =

Original: [Equation]
Revised: [Equation]

Original: ∼ U

Original: [Equation] B [Equation] ) ; i.e. it is uniform on the (cid:96) 2 ball of radius 1 .
Revised: ∼ U ( B 2 ) ; i.e. it is uniform on the (cid:96) 2 ball of radius 1 .

Original: Therefore,
Revised: Therefore,

Original: RW is uniform on the (cid:96) 2 ball of radius R and the conditional density of Z given R is given by 
Revised: RW is uniform on the (cid:96) 2 ball of radius R and the conditional density of Z given R is given by p

Original: [Equation]
Revised: [Equation]

Original: Since R has density p R ( r ) ∝ r d − 1 e − r p , we get
Revised: Since R has density p R ( r ) ∝ r d − 1 e − r p , we get

Original: [Equation]
Revised: [Equation]

Original: d − 1 e − r p dr ∝ e

Original: [Table]
Revised: [Table]

Original: z (cid:107) p 2 .
Revised: [Equation]

Original: A. A LGORITHM FOR FULL - INFORMATION CERTIFICATION
Revised: A. A LGORITHM FOR FULL - INFORMATION CERTIFICATION

Original: In this section we describe the subroutines used in Algorithm 1.
Revised: In this section we describe the subroutines used in Algorithm 1.

Original: First we describe a generic procedure to provide high-probability conﬁdence intervals for estimating expectations, then we give a detailed description of the subroutines.
Revised: First we describe a generic procedure to provide high-probability conﬁdence intervals for estimating expectations, then we give a detailed description of the subroutines.

Original: A.9. H IGH - CONFIDENCE ESTIMATES OF EXPECTED VALUES
Revised: A.9. H IGH - CONFIDENCE ESTIMATES OF EXPECTED VALUES

Original: Let Z 1 , . . .
Revised: Let Z 1 , . . .

Original: , Z N be independent, identically distributed random variables with range R and mean m .
Revised: , Z N be independent, identically distributed random variables with range R and mean m .

Original: Let the empirical mean be ¯ Z = 1 N (cid:80) Ni =1 Z i and the empirical variance be ¯
Revised: Let the empirical mean be ¯ Z = 1 N (cid:80) Ni =1 Z i and the empirical variance be ¯

Original: σ 2 = 1 N (cid:80) Ni =

Revised: σ 2 = 1 N (cid:80) Ni =

Original: ( Z i − ¯ Z ) 2 .
Revised: ( Z i − ¯ Z ) 2 .

Original: Applying Bernstein’s inequality to the sum and the sum of the squares of these random variables, we get the empirical Bernstein bound (Audibert et al., 2009), which states that with probability at least 1 − ζ ,
Revised: Applying Bernstein’s inequality to the sum and the sum of the squares of these random variables, we get the empirical Bernstein bound (Audibert et al., 2009), which states that with probability at least 1 − ζ ,

Original: [Equation]
Revised: [Equation]

Original: The main beneﬁt of the above inequality is that as long as the variance of the sample Z 1 , . . .
Revised: The main beneﬁt of the above inequality is that as long as the variance of the sample Z 1 , . . .

Original: , Z N is small, the convergence rate becomes essentially O (1 /N ) instead of the standard O (1 / √ N ) .
Revised: , Z N is small, the convergence rate becomes essentially O (1 /N ) instead of the standard O (1 / √ N ) .

Original: Also, since Eq. 23 only contains empirical quantities apart from the range R , it can be used to obtain computable bounds for the expectation µ : with probability at least 1 − ζ , ¯ Z−
Revised: Also, since Eq. 23 only contains empirical quantities apart from the range R , it can be used to obtain computable bounds for the expectation µ : with probability at least 1 − ζ , ¯ Z

Original: [Equation]
Revised: [Equation]

Original: A.9. S UBROUTINES FOR A LGORITHM
Revised: A.9. S UBROUTINES FOR A LGORITHM

Original: The bound in Eq.
Revised: The bound in Eq.

Original: 24 can be applied to approximate the expectation in Eq.
Revised: 24 can be applied to approximate the expectation in Eq.

Original: 6 with high probability for given values of λ and κ .
Revised: 6 with high probability for given values of λ and κ .

Original: More speciﬁcally, if the function f ∗ λ ( κ − φ ( · )) is bounded with range R , then taking N samples X 1 , . . .
Revised: More speciﬁcally, if the function f ∗ λ ( κ − φ ( · )) is bounded with range R , then taking N samples X 1 , . . .

Original: , X N independently from ρ , and deﬁning Z i =
Revised: , X N independently from ρ , and deﬁning Z i =

Original: f ∗ λ ( κ − φ ( X i )) , and ¯
Revised: f ∗ λ ( κ − φ ( X i )) , and ¯

Original: Z and ¯ σ 2 as above, Eq. 24 implies that with probability at least 1 − ζ ,
Revised: Z and ¯ σ 2 as above, Eq. 24 implies that with probability at least 1 − ζ ,

Original: [Equation]
Revised: [Equation]

Original: Plugging in this bound to Eq.
Revised: Plugging in this bound to Eq.

Original: 6 gives a high-probability lower bound for the function to be maximized for any given λ and κ .
Revised: 6 gives a high-probability lower bound for the function to be maximized for any given λ and κ .

Original: Details of the above procedures are given in Algorithm 3.
Revised: Details of the above procedures are given in Algorithm 3.

Original: We use an off-the-shelf convex optimization solver (Diamond & Boyd, 2016) in the ESTIMETEOPT subroutine.
Revised: We use an off-the-shelf convex optimization solver (Diamond & Boyd, 2016) in the ESTIMETEOPT subroutine.

Original: Algorithm 3 Subroutines for Algorithmfunction E STIMATE O PT ( ρ, φ, N, { f i } , { (cid:15) i } )
Revised: Algorithm 3 Subroutines for Algorithmfunction E STIMATE O PT ( ρ, φ, N, { f i } , { (cid:15) i } )

Original: Sample X 1 , . . .
Revised: Sample X 1 , . . .

Original: , X N ∼ ρ and obtain κ ∗ and λ ∗ by solving Eq.
Revised: , X N ∼ ρ and obtain κ ∗ and λ ∗ by solving Eq.

Original: 6 with E X ∼ ρ
Revised: 6 with E X ∼ ρ

Original: [ f ∗ λ ( κ − φ ( X ))] replaced by 1 N (cid:80)
Revised: [ f ∗ λ ( κ − φ ( X ))] replaced by 1 N (cid:80)

Original: Ni =1 f ∗ λ (cid:0) κ − φ (cid:0) X
Revised: Ni =1 f ∗ λ (cid:0) κ − φ (cid:0) X

Original: i (cid:1)(cid:1) and the additional constraints f ∗ λ ∗ ( κ − a ) < ∞ , f ∗ λ ∗ ( κ − b ) < ∞ .
Revised: i (cid:1)(cid:1) and the additional constraints f ∗ λ ∗ ( κ − a ) < ∞ , f ∗ λ ∗ ( κ − b ) < ∞ .

Original: return κ ∗ , λ ∗ .
Revised: return κ ∗ , λ ∗ .

Original: end function function U PPER C ONFIDENCE B OUND ( ρ, φ, ˜ N, { f i } Mi =1 , { (cid:15) i } Mi =1 , a, b, λ, κ, ζ )
Revised: end function function U PPER C ONFIDENCE B OUND ( ρ, φ, ˜ N, { f i } Mi =1 , { (cid:15) i } Mi =1 , a, b, λ, κ, ζ )

Original: Sample X 1 , . . .
Revised: Sample X 1 , . . .

Original: , X ˜ N ∼ ρ and compute Z i ← f ∗ λ (cid:0) κ − φ (cid:0) X
Revised: , X ˜ N ∼ ρ and compute Z i ← f ∗ λ (cid:0) κ − φ (cid:0) X

Original: i (cid:1)(cid:1) for i = 1 , . . . , ˜ N .
Revised: i (cid:1)(cid:1) for i = 1 , . . . , ˜ N .

Original: Set ¯ Z ← (cid:80) ˜
Revised: Set ¯ Z ← (cid:80) ˜

Original: Ni =1 Z i N .
Revised: Ni =1 Z i N .

Original: Set ¯ σ ← (cid:80) ˜
Revised: Set ¯ σ ← (cid:80) ˜

Original: Ni =

Revised: Ni =

Original: ( Z i − ¯ Z ) 2 N .
Revised: ( Z i − ¯ Z ) 2 N .

Original: Set R = max x ∈ ( a,b ) f ∗ λ ( κ − x ) − min x ∈ ( a,b ) f ∗ λ ( κ − x ) .
Revised: Set R = max x ∈ ( a,b ) f ∗ λ ( κ − x ) − min x ∈ ( a,b ) f ∗ λ ( κ − x ) .

Original: Set
Revised: Set

Original: E ub ← ¯ Z + (cid:115) 2¯ σ 2 log(3 /ζ )
Revised: E ub ← ¯ Z + (cid:115) 2¯ σ 2 log(3 /ζ )

Original: ˜ N + 3 R log(3 /ζ )
Revised: ˜ N + 3 R log(3 /ζ )

Original: ˜ N .
Revised: ˜ N .

Original: return E ub .
Revised: return E ub .

Original: end function
Revised: end function

Original: A. (cid:96) 0 SMOOTHING MEASURE
Revised: A (cid:96) 0 SMOOTHING MEASURE

Original: We can also handle discrete perturbations in our framework.
Revised: We can also handle discrete perturbations in our framework.

Original: A natural case to consider is (cid:96)perturbations.
Revised: A natural case to consider is (cid:96)perturbations.

Original: In this case, we assume that X
Revised: In this case, we assume that X

Original: =
Revised: =

Original: A d where
Revised: A d where

Original: A = { 1 , . . .
Revised: A = { 1 , . . .

Original: , K } is a discrete set.
Revised: , K } is a discrete set.

Original: Then, we can choose
Revised: Then, we can choose

Original: [Equation]
Revised: [Equation]

Original: where p + q = 1 , p ≥ q ≥ 0 , and p denotes the probability that the measure retains the value of x and qK − 1 denotes a uniform probability of switching it to a different value.
Revised: where p + q = 1 , p ≥ q ≥ 0 , and p denotes the probability that the measure retains the value of x and qK − 1 denotes a uniform probability of switching it to a different value.

Original: In this case, it can be shown that for every α > 0 that
Revised: In this case, it can be shown that for every α > 0 that

Original: [Equation]
Revised: [Equation]

Original:   so that we can derive a certiﬁcate with respect to (cid:96) 

Revised:   so that we can derive a certiﬁcate with respect to (cid:96)

Original: perturbations using any set of R´enyi divergences (or combinations of theses)
Revised: perturbations using any set of R´enyi divergences (or combinations of theses)

Original: [Equation]
Revised: [Equation]

Original: This can be extended to structured discrete perturbations by introducing coupling terms between the perturbations:
Revised: This can be extended to structured discrete perturbations by introducing coupling terms between the perturbations:

Original: [Equation]
Revised: [Equation]

Original: This would correlate perturbations between adjacent features (which for example may be useful to model correlated perturbations for time series data).
Revised: This would correlate perturbations between adjacent features (which for example may be useful to model correlated perturbations for time series data).

Original: Since this can be viewed as a Markov Chain,
Revised: Since this can be viewed as a Markov Chain,

Original: R ´ enyi divergences between µ ( x ) , µ ( x (cid:48) ) are still easy to compute.
Revised: R ´ enyi divergences between µ ( x ) , µ ( x (cid:48) ) are still easy to compute.

Original: A. C OMPARISON WITH L ECUYER ET AL . (2019) ON (cid:96) 1 PERTURBATIONS
Revised: A. C OMPARISON WITH L ECUYER ET AL . (2019) ON (cid:96) 1 PERTURBATIONS

Original: Here we compare our certiﬁcates to Lecuyer et al.
Revised: Here we compare our certiﬁcates to Lecuyer et al.

Original: on the MNIST, CIFAR-10 and ImageNet datasets.
Revised: on the MNIST, CIFAR-10 and ImageNet datasets.

Original: The smoothing distribution is as described in A.8; a zero mean Laplacian distribution with smoothing value deﬁned by the scale of the distribution.
Revised: The smoothing distribution is as described in A.8; a zero mean Laplacian distribution with smoothing value deﬁned by the scale of the distribution.

Original: We ﬁrst describe the hyperparameters used in training and certiﬁcation for each of the datasets.
Revised: We ﬁrst describe the hyperparameters used in training and certiﬁcation for each of the datasets.

Original: For all datasets, images were normalized into a [0,1] range.
Revised: For all datasets, images were normalized into a [0,1] range.

Original: MNIST hyperparameters: We trained a standard three layer CNN ReLU classiﬁer for 50, steps with a batch size of 128 and a learning rate of 0.001.
Revised: MNIST hyperparameters: We trained a standard three layer CNN ReLU classiﬁer for 50, steps with a batch size of 128 and a learning rate of 0.001.

Original: The smoothing value during training was set to 1.0.
Revised: The smoothing value during training was set to 1.0.

Original: For certiﬁcation we use N = 1 K, ˜ N = 10 M, ζ = . 99 , and sweep over a range of smoothing values between 0.5 and 1.5 and report the best certiﬁcate found.
Revised: For certiﬁcation we use N = 1 K, ˜ N = 10 M, ζ = . 99 , and sweep over a range of smoothing values between 0.5 and 1.5 and report the best certiﬁcate found.

Original: Certiﬁed accuracy is reported on 1,000 MNIST test set images.
Revised: Certiﬁed accuracy is reported on 1,000 MNIST test set images.

Original: CIFAR-10 hyperparameters: We trained a Wide ResNet classiﬁer for 50,000 training steps with a batch size of 32 and a learning rate of 0.001.
Revised: CIFAR-10 hyperparameters: We trained a Wide ResNet classiﬁer for 50,000 training steps with a batch size of 32 and a learning rate of 0.001.

Original: The smoothing value during training was set to 0.2.
Revised: The smoothing value during training was set to 0.2.

Original: For certiﬁcation we use N = 1 K, ˜ N = 1 M, ζ = . 99 , and sweep over a range of smoothing values between 0.1 and 0.5 and report the best certiﬁcate found.
Revised: For certiﬁcation we use N = 1 K, ˜ N = 1 M, ζ = . 99 , and sweep over a range of smoothing values between 0.1 and 0.5 and report the best certiﬁcate found.

Original: Certiﬁed accuracy is reported on 1,
Revised: Certiﬁed accuracy is reported on 1,

Original: CIFAR-10 test set images.
Revised: CIFAR-10 test set images.

Original: ImageNet hyperparameters: We trained a ResNet-152 classiﬁer for 1 million training steps with a batch size of 16 and an initial learning rate of 0.1 that is decayed by a factor of ten every 25, steps.
Revised: ImageNet hyperparameters: We trained a ResNet-152 classiﬁer for 1 million training steps with a batch size of 16 and an initial learning rate of 0.1 that is decayed by a factor of ten every 25, steps.

Original: The smoothing value during training was set to 0.1.
Revised: The smoothing value during training was set to 0.1.

Original: For certiﬁcation we use N = 1 K, ˜ N = 100 K, ζ = . 99 , and sweep over a range of smoothing values between 0.05 and 
Revised: For certiﬁcation we use N = 1 K, ˜ N = 100 K, ζ = . 99 , and sweep over a range of smoothing values between 0.05 and 0.

Original: Certiﬁcate
Revised: Certiﬁcate

Original: [Table]
Revised: [Table]

Original: (cid:15) =(cid:15) =(cid:15) =(cid:15) =(cid:15) =(cid:15) =(cid:15) =
Revised: (cid:15) =(cid:15) =(cid:15) =(cid:15) =(cid:15) =(cid:15) =(cid:15) =

Original: Lecuyer et al.
Revised: Lecuyer et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: C e r t i f i e d
Revised: C e r t i f i e d

Original: A cc u r a c y
Revised: A cc u r a c y

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original:  Results for MNIST can be seen in table 6, CIFAR-10 and ImageNet results are shown in ﬁgure 3.
Revised: [Figure] [Table] [Figure] Results for MNIST can be seen in Table 6, CIFAR-10 and ImageNet results are shown in Figure 3.

Original: We signiﬁcantly outperform Lecuyer et al.
Revised: We signiﬁcantly outperform Lecuyer et al.

Original: on all three datasets.
Revised: on all three datasets.