L EARNING PERTURBATION SETS FOR ROBUST MACHINE LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Although much progress has been made towards robust deep learning, a signiﬁcant gap in robustness remains between real-world perturbations and more narrowly deﬁned sets typically studied in adversarial defenses.
In this paper, we aim to bridge this gap by learning perturbation sets from data, in order to characterize realworld effects for robust training and evaluation.
Speciﬁcally, we use a conditional generator that deﬁnes the perturbation set over a constrained region of the latent space.
We formulate desirable properties that measure the quality of a learned perturbation set, and theoretically prove that a conditional variational autoencoder naturally satisﬁes these criteria.
Using this framework, our approach can generate a variety of perturbations at different complexities and scales, ranging from baseline spatial transformations, through common image corruptions, to lighting variations.
We measure the quality of our learned perturbation sets both quantitatively and qualitatively, ﬁnding that our models are capable of producing a diverse set of meaningful perturbations beyond the limited data seen during training.
Finally, we leverage our learned perturbation sets to train models which are empirically and certiﬁably robust to adversarial image corruptions and adversarial lighting variations, while improving generalization on non-adversarial data.
All code and conﬁguration ﬁles for reproducing the experiments as well as pretrained model weights can be found in the supplementary material.
I NTRODUCTION
Within the last decade, adversarial learning has become a core research area for studying robustness and machine learning.
Adversarial attacks have expanded well beyond the original setting of imperceptible noise to more general notions of robustness, and can broadly be described as capturing sets of perturbations that humans are naturally invariant to.
These invariants, such as facial recognition should be robust to adversarial glasses (Sharif et al., 2019) or trafﬁc sign classiﬁcation should be robust to adversarial grafﬁti (Eykholt et al., 2018), form the motivation behind many real world adversarial attacks.
However, human invariants can also include notions which are not inherently adversarial, for example image classiﬁers should be robust to common image corruptions (Hendrycks & Dietterich, 2019) as well as changes in weather patterns (Michaelis et al., 2019).
On the other hand, although there has been much success in defending against small adversarial perturbations, most successful and principled methods for learning robust models are limited to human invariants that can be characterized using mathematically deﬁned perturbation sets, for example perturbations bounded in (cid:96) p norm.
After all, established guidelines for evaluating adversarial robustness (Carlini et al., 2019) have emphasized the importance of the perturbation set (or the threat model) as a necessary component for performing proper, scientiﬁc evaluations of adversarial defense proposals.
However, this requirement makes it difﬁcult to learn models which are robust to human invariants beyond these mathematical sets, where real world attacks and general notions of robustness can often be virtually impossible to write down as a formal set of equations.
This incompatibility between existing methods for learning robust models and real-world, human invariants raises a fundamental question for the ﬁeld of robust machine learning:
How can we learn models that are robust to perturbations without a predeﬁned perturbation set?
In the absence of a mathematical deﬁnition, in this work we present a general framework for learning perturbation sets from perturbed data.
More concretely, given pairs of examples where one is a perturbed version of the other, we propose learning generative models that can “perturb” an example by varying a ﬁxed region of the underlying latent space.
The resulting perturbation sets are welldeﬁned and can naturally be used in robust training and evaluation tasks.
The approach is widely applicable to a range of robustness settings, as we make no assumptions on the type of perturbation being learned: the only requirement is to collect pairs of perturbed examples.
Given the susceptibility of deep learning to adversarial examples, such a perturbation set will undoubtedly come under intense scrutiny, especially if it is to be used as a threat model for adversarial attacks.
In this paper, we begin our theoretical contributions with a broad discussion of perturbation sets and formulate deterministic and probabilistic properties that a learned perturbation set should have in order to be a meaningful proxy for the true underlying perturbation set.
The necessary subset property ensures that the set captures real perturbations, properly motivating its usage as an adversarial threat model.
The sufﬁcient likelihood property ensures that real perturbations have high probability, which motivates sampling from a perturbation set as a form of data augmentation.
We then prove the main theoretical result, that a learned perturbation set deﬁned by the decoder and prior of a conditional variational autoencoder (CVAE)
(Sohn et al., 2015) implies both of these properties, providing a theoretically grounded framework for learning perturbation sets.
The resulting CVAE perturbation sets are well motivated, can leverage standard architectures, and are computationally efﬁcient with little tuning required.
We highlight the versatility of our approach using CVAEs with an array of experiments, where we vary the complexity and scale of the datasets, perturbations, and downstream tasks.
We ﬁrst demonstrate how the approach can learn basic (cid:96) ∞ and rotation-translation-skew (RTS) perturbations (Jaderberg et al., 2015) in the MNIST setting.
Since these sets can be mathematically deﬁned, our goal is simply to measure exactly how well the learned perturbation set captures the target perturbation set on baseline tasks where the ground truth is known.
We next look at a more difﬁcult setting which can not be mathematically deﬁned, and learn a perturbation set for common image corruptions on
CIFAR10 (Hendrycks & Dietterich, 2019).
The resulting perturbation set can interpolate between common corruptions, produce diverse samples, and be used in adversarial training and randomized smoothing frameworks.
The adversarially trained models have improved generalization performance to both in- and out-of-distribution corruptions and better robustness to adversarial corruptions.
In our ﬁnal setting, we learn a perturbation set that captures real-world variations in lighting using a multi-illumination dataset of scenes captured “in the wild” (Murmann et al., 2019).
The perturbation set generates meaningful lighting samples and interpolations while generalizing to unseen scenes, and can be used to learn image segmentation models that are empirically and certiﬁably robust to lighting changes.
All code and conﬁguration ﬁles for reproducing the experiments as well as pretrained model weights for both the learned perturbation sets as well as the downstream robust classiﬁers are in the supplementary material.
B ACKGROUND AND RELATED WORK
Perturbation sets for adversarial threat models Adversarial examples were initially deﬁned as imperceptible examples with small (cid:96) 1 , (cid:96) 2 and (cid:96) ∞ norm (Biggio et al., 2013; Szegedy et al., 2013; Goodfellow et al., 2014), forming the earliest known, well-deﬁned perturbation sets that were eventually generalized to the union of multiple (cid:96) p perturbations (Tram`er & Boneh, 2019;
Maini et al., 2019; Croce & Hein, 2019; Stutz et al., 2019).
Alternative perturbation sets to the (cid:96) p setting that remain well-deﬁned incorporate more structure and semantic meaning, such as rotations and translations (Engstrom et al., 2017), Wasserstein balls (Wong et al., 2019), functional perturbations (Laidlaw & Feizi, 2019), distributional shifts (Sinha et al., 2017; Sagawa et al., 2019), word embeddings (Miyato et al., 2016), and word substitutions (Alzantot et al., 2018;
Jia et al., 2019).
Other work has studied perturbation sets that are not necessarily mathematically formulated but welldeﬁned from a human perspective such as spatial transformations (Xiao et al., 2018b).
Real-world adversarial attacks tend to try to remain either inconspicuous to the viewer or meddle with features that humans would naturally ignore, such as textures on 3D printed objects (Athalye et al., 2017), grafﬁti on trafﬁc signs (Eykholt et al., 2018), shapes of objects to avoid LiDAR detection (Cao et al., 2019), irrelevant background noise for audio (Li et al., 2019a), or barely noticeable ﬁlms on cameras
(Li et al., 2019b).
Although not necessarily adversarial, Hendrycks & Dietterich (2019) propose the set of common image corruptions as a measure of robustness to informal shifts in distribution.
Generative modeling and adversarial robustness Relevant to our work is that which combines aspects of generative modeling with adversarial examples.
While our work aims to learn real-world perturbation sets from data, most work in this space differs in that they either aim to generate synthetic adversarial (cid:96) p perturbations (Xiao et al., 2018a), run user studies to deﬁne the perturbation set (Sharif et al., 2019), or simply do not restrict the adversary at all (Song et al., 2018; Bhattad et al., 2020).
Gowal et al.
(2019) trained a StyleGAN to disentangle real-world perturbations when no perturbation information is known in advance.
However the resulting perturbation set relies on a stochastic approximation, and it is not immediately obvious what this set will ultimately capture.
Most similar is the concurrent work of Robey et al.
(2020), which uses a GAN architecture from image-to-image translation to model simple perturbations between datasets.
In contrast to both of these works, our setting requires the collection of paired data to directly learn how to perturb from perturbed pairs without needing to disentangle any features or translate datasets, allowing us to learn more targeted and complex perturbation sets.
Furthermore, we formulate desirable properties of perturbation sets for downstream robustness tasks, and formally prove that a conditional variational autoencoder approach satisﬁes these properties.
This results in a principled framework for learning perturbation sets that is quite distinct from these GAN-based approaches in both setting and motivation.
Adversarial defenses and data augmentation Successful approaches for learning adversarially robust networks include methods which are both empirically robust via adversarial training (Goodfellow et al., 2014; Kurakin et al., 2016; Madry et al., 2017) and also certiﬁably robust via provable bounds (Wong & Kolter, 2017;
Wong et al., 2018; Raghunathan et al., 2018; Gowal et al., 2018;
Zhang et al., 2019) and randomized smoothing (Cohen et al., 2019; Yang et al., 2020).
Critically, these defenses require mathematically-deﬁned perturbation sets, which has limited these approaches from learning robustness to more general, real-world perturbations.
We directly build upon these approaches by learning perturbation sets that can be naturally and directly incorporated into robust training, greatly expanding the scope of adversarial defenses to new contexts.
Our work also relates to using non-adversarial perturbations via data augmentation to reduce generalization error (Zhang et al., 2017; DeVries & Taylor, 2017; Cubuk et al., 2019), which can occasionally also improve robustness to unrelated image corruptions (Geirhos et al., 2018; Hendrycks et al., 2019;
Rusak et al., 2020).
Our work differs in that rather than aggregating or proposing generic data augmentations, our perturbation sets can provide data augmentation that is targeted for a particular robustness setting.
P ERTURBATION SETS LEARNED FROM DATA
For an example x ∈ R m , a perturbation set S ( x ) ⊆
R m is deﬁned informally as the set of examples which are considered to be equivalent to x , and hence can be viewed as “perturbations” of x .
This set is often used when ﬁnding an adversarial example, which is typically cast as an optimization problem to maximize the loss of a model over the perturbation set in order to break the model.
For example, for a classiﬁer h , loss function (cid:96) , and label y , an adversarial attack tries to solve the following:
[Equation]
[Equation] A common choice for S ( x ) is an (cid:96) p ball around the unperturbed example, deﬁned as S ( x ) = { x + δ : (cid:107) δ (cid:107) p ≤ (cid:15) } for some norm p and radius (cid:15) .
This type of perturbation captures unstructured random noise, and is typically taken with respect to (cid:96) p norms for p ∈ { 0 , 1 , 2 , ∞} , though more general distance metrics can also be used.
Although deﬁning the perturbation set is critical for developing adversarial defenses, in some scenarios, the true perturbation set may be difﬁcult to mathematically describe.
In these settings, it may still be possible to collect observations of (non-adversarial) perturbations, e.g. pairs of examples ( x, ˜ x ) where ˜ x is the perturbed data .
In other words, ˜ x is a perturbed version of x , from which we can learn an approximation of the true perturbation set.
While there are numerous possible approaches one can take to learn S ( x ) from examples ( x, ˜ x ), in this work we take a generative modeling perspective, where examples are perturbed via an underlying latent space.
Speciﬁcally, let g : R k × R m →
R m be a generator that takes a k -dimensional latent vector and an input, and outputs a perturbed version of the input.
Then, we can deﬁne a learned perturbation set as follows:
[Equation]
In other words, we have taken a well-deﬁned norm-bounded ball in the latent space and mapped it to a set of perturbations with a generator g , which perturbs x into ˜ x via a latent code z .
Alternatively, we can deﬁne a perturbation set from a probabilistic modeling perspective, and use a distribution over the latent space to parameterize a distribution over examples.
Then, S ( x ) is now a random variable deﬁned by a probability distribution p (cid:15) ( z ) over the latent space as follows: [Equation] θ
[Equation]
g ( z [Equation] x )
[Equation]
where p (cid:15) has support { z : (cid:107) z (cid:107) ≤ (cid:15) } and p θ is a distribution parameterized by θ =
g ( z, x ) .
3. G ENERAL MEASURES OF QUALITY FOR PERTURBATION SETS
A perturbation set deﬁned by a generative model that is learned from data lacks the mathematical rigor of previous sets, so care must be taken to properly evaluate how well the model captures real perturbations.
In this section we formally deﬁne two properties relating a perturbation set to data, which capture natural qualities of a perturbation set that are useful for adversarial robustness and data augmentation.
We note that all quantities discussed in this paper can be calculated on both the training and testing sets, which allow us to concretely measure how well the perturbation set generalizes to unseen datapoints.
For this section, let d : R m × R m →
R be an distance metric (e.g. mean squared error) and let x,
˜ x ∈ R m be a perturbed pair, where ˜ x is a perturbed version of x .
To be a reasonable threat model for adversarial examples, one desirable expectation is that a perturbation set should at least contain close approximations of the perturbed data.
In other words, the set of perturbed data should be (approximately) a necessary subset of the perturbation set.
This notion of containment can be described more formally as follows:
Deﬁnition 1.
A perturbation set S ( x ) satisﬁes the necessary subset property at approximation error at most δ for a perturbed pair ( x, ˜ x ) if there exists an x (cid:48) ∈ S ( x ) such that d ( x (cid:48) , ˜ x ) ≤ δ .
For a perturbation set deﬁned by the generative model from Equation (2), this amounts to ﬁnding a latent vector z which best approximates the perturbed example ˜ x by solving the following problem:
[Equation]
This approximation error can be upper bounded with point estimates or can be solved more accurately with projected gradient descent.
Note that mathematically-deﬁned perturbation sets such as (cid:96) p balls around clean datapoints contain all possible observations and naturally have zero approximation error.
Our second desirable property is speciﬁc to the probabilistic view from Equation (3), where we would expect perturbed data to have a high probability of occurring under a probabilistic perturbation set.
In other words, a perturbation set should assign sufﬁcient likelihood to perturbed data, described more formally in the following deﬁnition:
Deﬁnition 2.
A probabilistic perturbation set S ( x ) satisﬁes the sufﬁcient likelihood property at likelihood at least δ for a perturbed pair ( x, ˜ x ) if E p (cid:15) ( z ) [ p θ (˜ x )] ≥ δ
where θ = g ( z, x ) .
A model that assigns high likelihood to perturbed observations is likely to generate meaningful samples, which can then be used as a form of data augmentation in settings that care more about average-case over worst-case robustness.
To measure this property, the likelihood can be approximated with a standard Monte Carlo estimate by sampling from the prior p (cid:15) .
V ARIATIONAL AUTOENCODERS FOR LEARNING PERTURBATIONS SETS
In this section we will focus on one possible approach using conditional variational autoencoders (CVAEs) to learn the perturbation set (Sohn et al., 2015).
We shift notation here to be consistent with the CVAE literature and consider a standard CVAE trained to generate x ∈ R m from a latent space z ∈ R k conditioned on some auxiliary variable y , which is traditionally taken to be a label.
In our setting, the auxiliary variable y is instead another datapoint such that x is a perturbed version of y , but the theory we present is agnostic to the choice in auxiliary variable.
Let the posterior distribution q ( z | x, y ) , prior distribution p ( z | y ) , and likelihood function p ( x | z, y ) be the following multivariate normal distributions with diagonal variance:
[Equation]
where µ ( x, y ) , σ 2 ( x, y ) , µ ( y ) , σ 2 ( y ) , and g ( z, y ) are arbitrary functions representing the respective encoder, prior, and decoder networks.
CVAEs are trained by maximizing a likelihood lower bound
[Equation]
also known as the SGVB estimator, where KL ( ·(cid:107)· ) is the KL divergence.
The CVAE framework lends to a natural perturbation set by simply restricting the latent space to an (cid:96) 2 ball that is scaled and shifted by the prior network.
For convenience, we will deﬁne the perturbation set in the latent space before the reparameterization trick, so the latent perturbation set for all examples is a standard (cid:96) 2 ball { u : (cid:107) u (cid:107) 2 ≤ (cid:15) }
where z = u · σ ( y ) + µ ( y ) .
Similarly, a probabilistic perturbation set can be deﬁned by simply truncating the prior distribution at radius (cid:15) (also before the reparameterization trick).
4. T HEORETICAL MOTIVATION OF USING CVAE S TO LEARN PERTURBATION SETS
Our theoretical results prove that optimizing the CVAE objective naturally results in both the necessary subset and sufﬁcient likelihood properties outlined in Section 3.1, which motivates why the CVAE is a reasonable framework for learning perturbation sets.
Note that these results are not immediately obvious, since the likelihood of the CVAE objective is taken over the full posterior while the perturbation set is deﬁned over a constrained latent subspace determined by the prior.
The proofs rely heavily on the multivariate normal parameterizations, with requiring several supporting results which relate the posterior and prior distributions.
We give a concise, informal presentation of the main theoretical results in this section, deferring the full details, proofs, and supporting results to Appendix B. Our results are based on the minimal assumption that the CVAE objective has been trained to some threshold as described in Assumption 1.
Assumption 1.
The CVAE objective has been trained to some thresholds R, K i as follows
[Equation]
where each K i bounds the KL-divergence of the i th dimension.
Our ﬁrst theorem, Theorem 1, states that the approximation error of a perturbed example is bounded by the components of the CVAE objective.
The implication here is that with enough representational capacity to optimize the objective, one can satisfy the necessary subset property by training a CVAE, effectively capturing perturbed data at low approximation error in the resulting perturbation set.
Theorem 1.
Let r be the Mahalanobis distance which captures 1 − α of the probability mass for a k -dimensional standard multivariate normal for some 0 < α < 1 .
Then, there exists a z such that (cid:13)(cid:13)(cid:13)
z − µ ( y ) σ ( y ) (cid:13)(cid:13)(cid:13) 2 ≤ (cid:15) and (cid:107) g
( z, y ) − x (cid:107) 22 ≤ δ for
[Equation]
where B is a constant dependent on K i .
Moreover, as R → − 12 m log(2 π ) and K i → 0 (the theoretical limits of these bounds 1 ), then (cid:15) → r and δ → 0 .
Our second theorem, Theorem 2, states that the expected approximation error over the truncated prior can also be bounded by components of the CVAE objective.
Since the generator g parameterizes a multivariate normal with identity covariance, an upper bound on the expected reconstruction error implies a lower bound on the likelihood.
This implies that one can also satisfy the sufﬁcient likelihood property by training a CVAE, effectively learning a probabilistic perturbation set that assigns high likelihood to perturbed data. [Table]
Theorem 2.
Let r be the Mahalanobis distance which captures 1 − α of the probability mass for a k -dimensional standard multivariate normal for some 0 < α < 1 .
Then, the truncated expected approximation error can be bounded with
[Equation]
where p r ( u ) is a multivariate normal that has been truncated to radius r and H is a constant that depends exponentially on K i and r .
The main takeaway from these two theorems is that optimizing the CVAE objective naturally results in a learned perturbation set which satisﬁes the necessary subset and sufﬁcient likelihood properties.
The learned perturbation set is consequently useful for adversarial robustness since the necessary subset property implies that the perturbation set does not “miss” perturbed data.
It is also useful for data augmentation since the sufﬁcient likelihood property ensures that perturbed data occurs with high probability.
We leave further discussion of these two theorems to Appendix B.3.
E XPERIMENTS
Finally, we present a variety of experiments to showcase the generality and effectiveness of our perturbation sets learned with a CVAE.
Our experiments for each dataset can be broken into two distinct types: the generative modeling problem of learning and evaluating a perturbation set, and the robust optimization problem of learning an adversarially robust classiﬁer to this perturbation.
We note that our approach is broadly applicable, has no speciﬁc requirements for the encoder, decoder, and prior networks, and avoids the unstable training dynamics found in GANs. Furthermore, we do not have the blurriness typically associated with VAEs since we are modeling perturbations and not the underlying image.
All code, conﬁguration ﬁles, and pretrained model weights for reproducing our experiments are included in the supplementary material and will be released with the paper.
In all settings, we ﬁrst train perturbation sets and evaluate them with a number of metrics averaged over the test set.
We present a condensed version of these results in Table 1, which establishes a quantitative baseline for learning real-world perturbation sets in three benchmark settings that future work can improve upon.
Speciﬁcally, the approximation error measures the necessary subset property, the expected approximation error measures the sufﬁcient likelihood property, and the reconstruction error and KL divergence are standard CVAE metrics.
The full evaluation is described in Appendix C, and a complete tabulation of our results on evaluating perturbation sets can be found in Tables 4, 7, and 10 in the appendix for each setting.
We then leverage our learned perturbation sets into new downstream robustness tasks by simply using standard, well-vetted techniques for (cid:96) 2 robustness directly on the latent space, namely adversarial training with an (cid:96) 2 PGD adversary (Madry et al., 2017), a certiﬁed defense with (cid:96) 2 randomized smoothing (Cohen et al., 2019), as well as an additional data-augmentation baseline via sampling from the CVAE truncated prior.
Pseudo-code for these approaches can be found in Appendix D.
We defer the experiments on MNIST to Appendix E, and spend the remainder of this section highlighting the main empirical results for the CIFAR10 and Multi-Illumination settings.
Additional details and supplementary experiments can be found in Appendix F for CIFAR10 common corruptions,
[Table]
5. CIFAR10 COMMON CORRUPTIONS
In this section
[Table]
we ﬁrst learn a perturbation set which captures common image corruptions for
CIFAR10 (Hendrycks & Dietterich, 2019).
2 We focus on the highest severity level of the blur, weather [Table] and digital categories, resulting in 12 different corruptions which capture more “natural” corruptions that are unlike random (cid:96) p noise (a complete description of the setting is in Appendix F).
We ﬁnd that the resulting perturbation set accurately captures common corruptions with a mean approximation error of 0.005 as seen in Table 1. A more in-depth quantitative evaluation as well as architecture and training details are in Appendix F.1.
We qualitatively evaluate our perturbation set in Figure 1, which depicts interpolations, random samples, and adversarial examples from the perturbation set.
For additional analysis of the perturbation set, we refer the reader to Appendix F.2 for a study on using different pairing strategies during training and Appendix F.3 which ﬁnds semantic latent structure and visualizes additional examples.
Robustness to corruptions We next employ the perturbation set in adversarial training and randomized smoothing to learn models which are robust against worst-case CIFAR10 common corruptions.
We report results at three radius thresholds { 2 . 7 , 3 . 9 , 10 . 2 } which correspond to the 25 th, 50 th, and 75 th percentiles of latent encodings as described in Appendix F.3.
We compare to two data augmentation baselines of training on perturbed data or samples drawn from the learned perturbation set, and also evaluate performance on three extra out-of-distribution corruptions (one for each weather, blur, and digital category denoted OOD) that are not present during training.
We highlight some empirical results in Table 2, where we ﬁrst ﬁnd that training with the CVAE perturbation set can improve generalization.
Speciﬁcally, using the CVAE perturbation set during training achieves 3 − 5% improved accuracy over training directly on the common corruptions (data augmentation) across all non-adversarial metrics.
These gains motivate learning perturbation sets beyond the setting of worst-case robustness as a way to improve standard generalization.
Additionally, [Table] the CVAE perturbation set improves worst-case performance, with the adversarially trained model being the most robust at 66% robust accuracy for (cid:15) = 3 .
9 whereas pure data augmentation only achieves 17 .
8% robust accuracy.
Finally, we include a comparison to models trained with standard training, AugMix data augmentation, (cid:96) ∞ adversarial training [Table]
and (cid:96) 2 adversarial training, none which can perform as well as our CVAE approach.
We note that this is not too surprising, since these approaches have different goals and data assumptions.
Nonetheless, we include these results for the curious reader, with additional details and discussion in Appendix F.5.
For certiﬁably robust models with randomized smoothing, we defer the results and discussion to Appendix F.
[Table]
5. M ULTI - ILLUMINATION
Our last set of experiments looks at learning a perturbation set of multiple lighting conditions using the Multi-Illumination (MI) dataset (Murmann et al., 2019).
These consist of a thousand scenes captured in the wild under 25 different lighting variations, and our goal is to learn a perturbation set which captures real-world lighting conditions.
Since the process of learning lighting variations is largely similar to the image corruptions setting, we defer most of the discussion on learning and evaluating the CVAE-based lighting perturbation set to Appendix G.1.
Note that our perturbation set accurately captures real-world changes in lighting with a low approximation error of 0.006 as seen in
Table 1. We qualitatively evaluate our perturbation set by depicting adversarial examples in Figure 2, with more visualizations (including samples and interpolations) in Appendix G.2.
Robustness to lighting perturbations We devote the remainder of this section to studying the task of generating material segmentation maps which are robust to lighting perturbations, using our CVAE perturbation set.
We highlight that adversarial training improves robustness to worst-case lighting perturbations over directly training on the perturbed examples, increasing robust accuracy from 27 .
1% to 35 .
4% at the maximum radius (cid:15) = 17 .
Additional results on certiﬁably robust models with randomized smoothing can be found in Appendix G.3.
C ONCLUSION
In this paper, we presented a general framework for learning perturbation sets from data when the perturbation cannot be mathematically-deﬁned.
We outlined deterministic and probabilistic properties that measure how well a perturbation set ﬁts perturbed data, and formally proved that a perturbation set based upon the CVAE framework satisﬁes these properties.
This work establishes a principled baseline for learning perturbation sets with quantitative metrics which future work can potentially improve upon, e.g. by using a different generative modeling frameworks.
The resulting perturbation sets open up new downstream robustness tasks such as adversarial and certiﬁable robustness to common image corruptions and lighting perturbations, while also potentially improving non-adversarial robust performance to natural perturbations.
Our work opens a pathway for practitioners to learn machine learning models that are robust to targeted, real-world perturbations that can be collected as data.
A A PPENDIX B T HEORETICAL RESULTS
In this section, we present the theoretical results in their full detail and exposition.
Both of the main theorems presented in this work require a number of preceding results in order to formally link the prior and the posterior distribution based on their KL divergence.
We will present and prove these supporting results before proving each main theorem.
B. P ROOF OF T HEOREM
Theorem 1 connects the CVAE objective to the necessary subset property.
In order to prove this, we ﬁrst prove three supporting lemmas.
Lemma 1 states that if the expected value of a function over a normal distribution is low, then for any ﬁxed radius there must exist a point within the radius with proportionally low function value.
This leverages the fact that the majority of the probability mass is concentrated around the mean and can be characterized by the Mahalanobis distance.
Lemma 1.
Let f ( u ) :
R k
→ R + be a non-negative integrable function, and let N k (0 , I ) be a k -dimensional standard multivariate normal random variable with zero mean and identity covariance.
Suppose E u ∼N (0 ,I )
[ f ( u )] ≤ δ for some δ > 0 .
Then, for any pair ( r, α ) where r is the Mahalanobis distance which captures 1 − α of the probability mass of N k (0 , I ) , there exists a u such that (cid:107) u (cid:107) 2 ≤ r and f ( u ) ≤ δ 1 − α .
Proof.
We will prove this by contradiction.
Assume for sake of contradiction that for all u such that (cid:107) u (cid:107) 2 ≤ r , we have f ( u ) > δ 1 − α .
We divide the expectation into two integrals over the inside and outside of the Mahalanobis ball:
[Equation]
Using the assumption on the ﬁrst integral and non-negativity of f in the second integrand, we can conclude
[Equation]
where the equality holds by deﬁnition of the Mahalanobis distance, which contradicts the initial assumption that E u ∼N (0 ,I )
[ f ( u )] ≤ δ .
Thus, we have proven by contradiction that there exists a u such that (cid:107) u (cid:107) 2 ≤ r and f ( u ) ≤ δ 1 − α .
Our second lemma, Lemma 2, is an important result that comes from the algebraic form of the KL divergence.
It is needed to connect the bound on the KL divergence to the actual variances of the prior and the posterior distribution, and uses the LambertW function to do so.
Lemma 2.
Let W k be the LambertW function with branch k .
Let x > 0 , and suppose x − log x ≤ y .
Then, x ∈ [ a, b ] where a = − W 0 ( − e − y ) and b = − W − 1 ( − e − y ) } .
Additionally, these bounds coincide at x = 1 when y = 1 .
Proof.
The LambertW function W k ( y ) is deﬁned as the inverse function of y =
xe x , where the path to multiple solutions is determined by the branch k .
We can then write the inverse of y = x − log x as one of the following two solutions:
[Equation]
Since x − log x is convex with minimum at x = 1 , and since the two solutions surround x = 1 , the set of points which satisfy x − log x ≤ y is precisely the interval [ − W 0 ( − e − y ) , − W − 1 ( − e − y )] .
Evaluating the bound at y = 1 completes the proof.
Lemma 3 is the last lemma needed to prove Theorem 1, which explicitly bounds terms involving the mean and variance of the prior and posterior distributions by their KL distance, leveraging Lemmato bound the ratio of the variances.
The two quantities bounded in this lemma will be used in the main theorem to bound the distance of a point with low reconstruction error from the prior distribution.
Lemma 3.
Suppose the
KL distance between two normals is bounded, so
KL
[Equation]
N
[Equation]
µ 1 , σ 21 )
[Equation]
µ 2 , σ 22 )) ≤ K for some constant K .
Then,
[Equation]
and also σ 21 σ 22 ∈ [ a, b ] where
[Equation]
Proof.
By deﬁnition of KL divergence, we have
[Equation]
Since x − log x ≥ 1 for x ≥ 0 , we apply this for x = σ 21 σ 22 to prove the ﬁrst bound on the squared distance as
[Equation]
Next, since ( µ 1 − µ 2 ) 2 1 σ 22 ≥ 0 , we can bound the remainder as the following
[Equation]
and using Lemma 2, we can bound σ 21 σ 22 ∈ [ a, b ] where
[Equation]
With these three results, we can now prove the ﬁrst main theorem, which we is presented below in its complete form, allowing us to formally tie the CVAE objective to the existence of a nearby point with low reconstruction error.
Theorem 1.
Consider the likelihood lower bound for x ∈ R n from the conditional VAE in kdimensional latent space conditioned on some other input y .
Let the posterior, prior, and decoder distributions be standard multivariate normals with diagonal covariance as follows
[Equation]
g ( z, y ) , I ) resulting in the following likelihood lower bound:
[Equation]
Suppose we have trained the lower bound to some thresholds R, K
i
[Equation]
where K i bounds the KL-divergence of the i th dimension.
Let r be the Mahalanobis distance which captures 1 − α of the probability mass for a k -dimensional standard multivariate normal for some 0 < α < 1 .
Then, there exists a z such that (cid:107) z − µ ( y ) σ ( y ) (cid:107) 2 ≤ (cid:15) and (cid:107) g
( z, y ) − x (cid:107) 22 ≤ δ for
[Equation]
where B is a constant dependent on K i .
Moreover, as R → − 12 m log(2 π ) and K i → 0 (the theoretical limits of these bounds, e.g. by training), then (cid:15) → r and δ → 0 .
Proof.
The high level strategy for this proof will consist of two main steps.
First, we will show that there exists a point near the encoding distribution which has low reconstruction error, where we leverage the Mahalanobis distance to capture nearby points with Lemma 1.
Then, we apply the triangle inequality to bound its distance under the prior encoding.
Finally, we will use Lemma 3 to bound remaining quantities relating the distances between the prior and encoding distributions to complete the proof.
Let z ( u ) =
u · σ ( x, y )+ µ ( x, y ) be the parameterization trick of the encoding distribution.
Rearranging the log probability in the assumption and using the reparameterization trick, we get
[Equation]
Applying Lemma 1, there must exist a u such that (cid:107) u (cid:107) ≤ r
and (cid:107) x − g
( z ( u ) , y ) (cid:107)
≤ δ , and so z ( u ) satisﬁes the reconstruction criteria for δ .
We will now show that z ( u ) fulﬁlls the remaining criteria for (cid:15) : calculating its (cid:96) 2 norm under the prior distribution and applying the triangle inequality, we get
[Equation]
We can use Lemma 3 on the KL assumption to bound the following quantities
[Equation]
) where [ a i , b i ] are as deﬁned from Lemma 3.
Let B = max i √ b
i , so σ i ( x,y ) σ i ( y ) ≤ B for all i .
Plugging this in along with the previous bounds we get the following bound on the norm of z ( u ) before the prior reparameterization:
[Equation]
Thus, the norm (before the prior reparameterization) and reconstruction error of z ( u ) can be bounded by (cid:15) and δ .
To conclude the proof, we note that from Lemma 2, K i → 0 for all i implies B → 1 , and so (cid:15)
→ r .
Similarly by inspection, R → − 12 m log(2 π ) implies that δ → 0 , which concludes the proof.
B. P ROOF OF T HEOREM
In this section, we move on to prove the second main result of this paper, Theorem 2, which connects the CVAE objective to the sufﬁcient likelihood property.
The proof for this theorem is not immediate, because since the generator is an arbitrary function, two normal distributions which have a difference in means have an exponentially growing ratio in their tail distributions, and so truncating the normal distributions is crucial.
This truncation is leveraged in Lemma 4, which bounds the ratio of two normal distributions constrained within an (cid:96) 2 ball, and is what allows us to connect the expectation over the prior with the expectation over the posterior.
Lemma 4.
Let p ∼ N (0 , 1) and q ∼ N ( − µ/σ 2 , 1 /σ 2 ) .
Then,
[Equation]
Furthermore if µ 2 ≤ K and σ ∈ [ a, b ] , then h ( r, µ, σ ) ≤ be max( C 1 ,C 2 ) where
[Equation]
Proof.
The proof here is almost purely algebraic in nature.
By deﬁnition of q and p , we have
[Equation] [Equation]
We will focus on bounding the exponent, (1 − σ 2 )
z 2 − 2 σµz − µ 2 .
We can bound this by considering two cases.
First, suppose that 1 − σ 2 < 0 and so the exponent is a concave quadratic.
Then, the maximum value of the quadratic is at its root:
[Equation]
Further assume that this z is within the interval of the indicator function,
[Equation]
Then, plugging in the maximum value into the quadratic results in the following bound f 1 for this case:
[Equation]
Consider the other case, so either 1 − σ 2 ≥ 0 , or the optimal value for z ∗ in the previous case when 1 − σ 2 < 0 is not within the interval | σz + µ | ≤ r .
Then, the maximum value of this quadratic must occur at | σz + µ | = r , so z = ± r − µ σ .
Plugging this into the quadratic for positive r , this results in
[Equation] [Equation]
and for negative r , this is
[Equation]
and so we can bound this case with the following function f 2 :
[Equation] [Equation]
Plugging in the maximum over both cases forms our ﬁnal bound on the ratio of distributions.
[Equation]
To ﬁnish the proof, assume we have the corresponding bounds on µ and σ .
Then, the ﬁrst case can be bounded with C 1 deﬁned as
[Equation]
The second case can be bounded with C 2 deﬁned as
[Equation]
And thus we can bound h ( r, µ, σ ) as
[Equation]
Lemma 4 can be directly applied to the setting of Theorem 2 for the case of a non-conditional VAE with a standard normal distribution for the prior.
However, since we are using conditional VAEs instead, the prior distribution has its own mean and variance and so the posterior distribution needs to be unparameterized from the posterior and reparameterized with the prior.
Corollary 1 establishes this formally, extending Lemma 4 to the conditional setting.
Corollary 1.
Let p ∼ N (0 , 1) and q ∼ N (cid:16) µ 2 − µ 1 σ 1 , σ 22 σ 2 1 (cid:17) .
Then,
[Equation]
where µ = 1 σ 2 ( µ 1 − µ 2 ) and σ = σ 1 σ 2 .
Furthermore, if KL ( N ( µ 1 , σ 21 ) (cid:107)N ( µ 2 , σ 22 )) ≤ K for some constant K , then h ( r, µ, σ ) ≤ H where H is a constant which depends on K .
Proof.
First, note that Lemma 3 implies that µ 2 ≤ K and σ ∈ [ a, b ] .
Observe that q ∼ N (cid:0) − µσ , 1 σ 2 (cid:1) and σ 1 z + µ 1 − µ 2 σ= σz + µ , and so the proof reduces to an application of Lemma 4 to this particular µ and σ .
These results allow us to bound the ratio of the prior and posterior distributions within a ﬁxed radius, which will allow us to bound the expectation over the prior with the expectation over the posterior.
We can now ﬁnally prove Theorem 2, presented in its complete form below.
Theorem 2.
Consider the likelihood lower bound for x ∈ R n from the conditional VAE in kdimensional latent space conditioned on some other input y .
Let the posterior, prior, and decoder distributions be standard multivariate normals with diagonal covariance as follows
[Equation]
resulting in the following likelihood lower bound:
[Equation]
Suppose we have trained the lower bound to some thresholds R, K
i
[Equation]
where K i bounds the KL-divergence of the i th dimension.
Let r be the Mahalanobis distance which captures 1 − α of the probability mass for a k -dimensional standard multivariate normal for some 0 < α < 1 .
Then, the truncated expected reconstruction error can be bounded with
[Equation]
where p r ( u ) is a multivariate normal that has been truncated to radius r and H is a constant that depends exponentially on K i .
Proof.
The overall strategy for this proof will be as follows.
First, we will rewrite the expectation over a truncated normal into an expectation over a standard normal, using an indicator function to control the radius.
Second, we will do an “unparameterization” of the standard normal to match the parameterized objective in the assumption.
Finally, we will bound the ratio of the unparameterized density over the normal prior, which allows us to bound the expectation over the prior with the assumption.
This last step to bound the ratio of densities is made possible by the truncation, which would otherwise grow exponentially with the tails of the distribution.
For notational simplicity, let f ( · ) = (cid:107) g
( · , y ) − x (cid:107) 22 .
The quantity we wish to bound can be rewritten using the Mahalanobis distance as
[Equation] [Equation]
where we used the fact that p r ( u ) =
[Equation]
≤ r ) p ( u ) , which simply rewrites the density of a truncated normal using a scaled standard normal density and an indicator function.
We can do a parameterization trick z
= u · σ ( y ) + µ ( y ) to rewrite this as
[Equation]
followed by a reverse parameterization trick with v =
z − µ ( x,y ) σ ( x,y ) to get the following equivalent expression
[Equation]
where p v ∼ N (cid:16) µ ( x ) − µ ( x,y ) σ ( x,y ) , σ 2 ( x ) σ 2 ( x,y ) (cid:17) .
For convenience, we can let
[Equation] [Equation]
which can be interpreted as a truncated version of p v , and so the expectation can be represented more succinctly as
[Equation]
We will now bound ˆ p v ( v ) with the standard normal distribution p ( v ) so that we can apply our assumption.
Because f is non-negative, the (cid:96) 2 constraint in ˆ p v ( v ) can be relaxed to an element-wise (cid:96) ∞ constraint to get
[Equation]
where we also used the fact that p is a diagonal normal, so (cid:81)
ki =1 p ( v i ) =
p ( v ) .
Each term in this product can be bounded by Lemma 4 to get
[Equation]
33) where H i is as deﬁned in Corollary 1 for each i using the corresponding K i , and we let H =
(cid:81) ki
=1 H i .
Thus we can now bound the truncated expected value by plugging in our bound for ˆ p v ( v ) into Equation (31) to get
[Equation]
Using our lower bound on the expected log likelihood from the assumption, we can bound the remaining integral with
[Equation]
and so combining this into our previous bound from Equation (34) results in the ﬁnal bound
[Equation]
B. D ISCUSSION OF THEORETICAL RESULTS
We ﬁrst note that the bound on the expected reconstruction error in Theorem 2 is larger than the corresponding bound from Theorem 1 by an exponential factor.
This gap is to some extent desired, since Theorem 1 characterizes the existence of a highly accurate approximation whereas Theorem 2 characterizes the average case, and so if this gap were too small, the average case of allowable perturbations would be constrained in reconstruction error, which is not necessarily desired in all settings.
This relates to the overapproximation error, or the maximum reconstruction error within the perturbation set.
Initially one may think that having low overapproximation error to be a desirable property of perturbation sets, in order to constrain the perturbation set from deviating by “too much.”
However, perturbation sets such as the rotation-translation-skew perturbations for MNIST will always have high overapproximation error, and so reducing this is not necessary desired.
Furthermore, it not theoretically guaranteed for the CVAE to minimize the overapproximation error without further assumptions.
This is because it is possible for a function to be arbitrarily small in expectation but be arbitrarily high at some nearby point, so optimizing the CVAE objective does not imply low overapproximation error.
A simple concrete example demonstrating this is the following piecewise linear function
[Equation]
for a > 0 as seen in Figure 3.
Note that max | z |≤ (cid:15) δ a ( z ) =
a which is unbounded as a → ∞ for all (cid:15) ≥ 0 , while at the same time E N (0 , 1) [ δ a ( z )] ≤ 1 a → 0 .
Thus, explicitly minimizing overapproximation error is not necessarily desired nor is it guaranteed by the CVAE.
Nonetheless, we report it anyways to give the reader a sense of how much the CVAE can deviate within the learned perturbation set.
C E VALUATING A PERTURBATION SET
In this section we describe in more detail the process of selecting a radius for a learned perturbation set and formally describe all the details regarding the evaluation metrics for a perturbation set learned with a CVAE.
As a reminder, let µ ( x, ˜ x ) , σ ( x, ˜ x ) be the posterior encoding, µ ( x ) , σ ( x ) be the prior encoding, and g ( x, z ) be the decoder for the CVAE.
We will use u to represent the latent space before the parameterization trick.
Then the perturbation set deﬁned by our generative process is
[Equation]
and the probabilistic perturbation set is deﬁned by the truncated normal distribution N (cid:15) before the parameterization trick as follows,
[Equation]
Selecting a radius To select a radius, we take the following conservative estimate calculated on a held-out validation set, which computes the smallest (cid:96) 2 ball under the prior which contains all the mean encodings under the posterior:
[Equation]
The beneﬁts of such a selection is that by taking the maximum, we are selecting a learned perturbation set that includes every point with approximation error as low as the posterior encoder.
To some extent, this will also capture additional types of perturbations beyond the perturbed data, which can be both beneﬁcial and unwanted depending on what is captured.
However, in the context of adversarial attacks, using a perturbation set which is too large is generally more desirable than one which is too small, in order to not underspecify the threat model.
Evaluation metrics We present each evaluation metric in detail for a single perturbed pair ( x, ˜ x ) .
These evaluation metrics can then be averaged over the test dataset to produce the evaluations presented in this paper.
Encoder approximation error (Enc.
AE)
We can get a fast upper bound of approximation
error by taking the posterior mean, unparameterizing it with respect to the prior, and projecting it to the (cid:15) ball as follows:
Enc.
[Equation] [Equation]
[Equation]
) (cid:107)where z = u · σ ( x ) + µ ( x ) , u = Proj (cid:15) (cid:18) µ ( x, ˜ x ) − µ ( x ) σ ( x ) (cid:19) 2.
PGD approximation error (PGD AE)
We can reﬁne the upper bound by solving the following problem with projected gradient descent:
[Equation]
where z = u · σ ( x ) + µ ( x )
In practice, we implement this by warm starting the procedure with the solution from the encoder approximation error, and run 50 iterations of PGD at step size (cid:15)/ 20 .
Expected approximation error (EAE)
We can compute this by drawing N samples u i ∼
N (cid:15) (0 , I, (cid:15) ) and calculating the following Monte Carlo estimate:
[Equation]
where z i =
u i · σ ( x ) + µ ( x )
In practice, we ﬁnd that N = 5 is sufﬁcient for reporting means over the dataset with near-zero standard deviation.
Over approximation error (OAE)
We can compute this by performing a typical (cid:96) 2 PGD adversarial attack:
[Equation]
where z = u · σ ( x ) + µ ( x )
In practice, we implement this by doing a random initialization and run 50 iterations of PGD at step
size (cid:15)/ 20 . 5.
Reconstruction error (Recon. err)
This is the typical reconstruction error of a variational autoencoder, which is a Monte Carlo estimate over the full posterior with one sample:
[Equation] [Equation]
where z = u · σ ( x, ˜ x ) + µ ( x, ˜ x ) u ∼ N (0 , I )
Note that we report the average over all pixels to be consistent with the other metrics in this paper, however it is typical to implement this during training as a sum of squared error instead of a mean. 6.
KL divergence (KL)
This is the standard KL divergence between the posterior and the prior distributions
[Equation]
D A DVERSARIAL TRAINING , RANDOMIZED SMOOTHING , AND DATA AUGMENTATION WITH LEARNED PERTURBATION SETS
In this section, we describe how various standard, successful techniques in robust training can applied to learned perturbation sets.
We note that each method is virtually unchanged, with the only difference being the application of the method to the latent space of the generator instead of directly in the input space.
Algorithm
Given a dataset D , perform an epoch of adversarial training with a learned perturbation set given by a generator g and a radius (cid:15) with step size γ , using a PGD adversary with T steps and step size α for ( x, y ) ∈
D do δ := 0 //
Initialize perturbation for t ∈ T do g :=
∇ δ (cid:96) ( h ( g ( z + δ, x )) , y ) //
Calculate gradient δ :=
δ + α · g/ (cid:107) g
(cid:107)
Take a gradient step if (cid:107) δ (cid:107) 2 > (cid:15) then δ := (cid:15) · δ/ (cid:107) δ (cid:107) 2 // Project onto (cid:15) ball θ := θ − γ ∇ θ (cid:96) ( h ( g ( z + δ, x )) , y ) //
Optimize model weights
D. A DVERSARIAL TRAINING
Adversarial training, or training on adversarial examples generated by an adversary, is a leading empirical defenses for learning models which are robust to adversarial attacks.
The adversarial attack is typically generated with a PGD adversary and is used in the context of (cid:96) p adversarial examples.
We follow closely the adversarial training formulation from (Madry et al., 2017).
In order to do adversarial training on learned perturbation sets, it sufﬁces to simply run the PGD adversarial attack in the latent space of the generator, as shown in Algorithm 1.
Algorithm
Given a datapoint x , pseudocode for certiﬁcation and prediction for a classiﬁer which has been smoothed over a learned perturbation set given by a generator g and a radius (cid:15) using a noise level σ with probability at least 1 − α .
function S AMPLE U NDER N OISE ( h, g, x, n, σ ) c
i := 0 for i ∈
[ k ] // Initialize counts for t ∈ [ n ] do //
Draw random samples from the generator and count classes for prediction z := N (0 , σ 2 ) j := arg max i h ( g ( z, x ))
c j :=
c j +return c function P REDICT
( h, g, x, n, σ, α ) c :=
S
AMPLE U NDER N OISE ( h, g, x, n, σ ) i, j := top two indices in c if B INOM PV ALUE ( c i , c
i + c j , 0 . 5 ) ≤ α then return prediction i elsereturn ABSTAIN function C ERTIFY ( h, g, x, n 0 , n, σ, α ) c 0 :
=
S
AMPLE U NDER N OISE ( h, g, x, n 0 , σ ) k :=
arg max i ( c 0 )
i c := S AMPLE
U NDER N OISE ( h, g, x, n, σ ) p a :=
L OWER C ONFIDENCE B OUND ( c (cid:48) k , n, 1 − α ) if p a > 0 .
5 then return prediction k with radius σ Ψ − 1 ( p a ) elsereturn ABSTAIN
D. R ANDOMIZED SMOOTHING
Randomized smoothing is a method for learning models robust to adversarial examples which come with certiﬁcates that can prove (with high probability) the non-existance of adversarial examples.
We follow closely the randomized smoothing procedure from (Cohen et al., 2019), which trains and certiﬁes a network by augmented the inputs with large Gaussian noise.
In order to do randomized smoothing on the learned perturbation sets, it sufﬁces to simply train, predict, and certify with augmented noise in the latent space of the generator, as shown in Algorithm 2 for prediction and certiﬁcation.
The algorithms are almost identical to that propose by Cohen et al.
(2019), with the exception that the noise is passed to the generator before going through the classiﬁer.
Note that L OWER C ONFIDENCE B OUND ( k, n, 1 − α ) returns a one-sided 1 − α conﬁdence interval for the Binomial parameter p given a sample k ∼ B INOMIAL ( n, p ) , and B INOM PV ALUE ( a, a + b, p ) returns the p value of a two-sided hypothesis test that a ∼ B INOMIAL ( a + b, p ) .
E MNIST
In this section, we discuss some benchmark tasks on MNIST, where the perturbation set is mathematically-deﬁned. Since an exact perturbation set will always be better than a learned approximation, the goal here is simply to explore the data and architectural requirements for learning accurate perturbations sets with CVAEs, and establish a baseline where the ground truth is known. We consider learning the classic (cid:96) ∞ perturbation and rotation-translation-skew (RTS) perturbations (Jaderberg et al., 2015). We use the standard MNIST dataset consisting of 60,000 training examples with 1,000 examples randomly set aside for validation purposes, and 10,000 examples in the test set.
These experiments were run on a single GeForce RTX
Ti graphics card, with the longest perturbation set takinghour to train.
E. E XPERIMENTAL DETAILS
(cid:96) ∞ details For the (cid:96) ∞ setting, perturbations with radius (cid:15) = 0 .
Our fully connected network encoders have one hidden layer with 784 hidden units, then two linear modules with 784 outputs each for computing the mean and log variance of the latent space distributions.
The generator network has the same structure, one hidden layer with 784 hidden units.
The posterior and generator networks simply concatenate their inputs into a single vector.
The encoders for the convolutional network for the (cid:96) ∞ setting uses two hidden convolutional layers with 32 and 64 channels, each followed by ReLU and max pooling.
To generate the mean and log variance, there are two ﬁnal convolution with 128 channels.
The posterior network concatenates its inputs via the channel dimension.
The generator has two analogous hidden layers using convolutions with 64 and 32 channels followed by ReLU and nearest neighbor upsampling.
One ﬁnal convolution reduces the number of channels to 1, and the conditioned input x is downsampled with 1 ×convolutions and concatenated with the feature map before each convolution.
All convolutions other than the downsampling convolutions use a 3 × 3 kernel with one padding.
Both networks are trained for 20 epochs, with step size following a piece-wise linear schedule of [0 , 0 . 001 , 0 . 0005 , 0 . 0001] over epochs [0 , 10 , 15 , 20] with the Adam optimizer using batch size 128.
The KL divergence is weighted by β which follows a piece-wise linear schedule from [0 , 0 . 001 , 0 . 01] over epochs [0 , 5 , 20] .
RTS details For the RTS setting, we follow the formulation studied by Jaderberg et al.
(2015), which consists of a random rotation between [ − 45 , 45] degrees, random scaling factor between [0 . 7 , 1 . 3] , and random placement in a 42 × 42 image.
The architecture uses a typical pre-activation convolution block, which consists of 2 convolutions with reﬂection padding which are pre-activated with batchnorm and ReLU.
The encoders uses a single block followed by two linear modules tounits to generate the mean and log variance, where the posterior network concatenates its inputs via the channel dimension.
The generator has a linear layer to map the latent vector back to a 42 ×feature map, followed by ﬁve parallel spatial transformers which use the latent vector to transform the conditioned input.
Multiple spatial transformers here are used simply because training just one can be inconsistent.
The outputs of the spatial transformers are concatenated via the channel dimension, followed by a convolutional block and one ﬁnal convolution to reduce it back to one channel.
[Table]
All networks are trained for 100 epochs with cyclic learning rate peaking at 0 .
on the 40 th epoch with the Adam optimizer using batch size 128.
The KL divergence is weighted by β which follows a piece-wise linear schedule from [0 , 0 . 01 , 1 , 1] over epochs [0 , 10 , 50 , 100] .
E. S AMPLES AND VISUALIZATIONS
Figure 4 plots samples from the convolutional model producing (cid:96) ∞ perturbations and the RTS model trained with only one perturbed datapoint per example.
We note that the ﬁnal β value for the (cid:96) ∞ perturbation was tuned to produce reasonable samples in order to balance the KL divergence and reconstruction error, likely due to the inherent difﬁculty of this setting.
E. E VALUATING THE PERTURBATION SET
Table 4 contains the full tabulated results evaluating the MNIST perturbation sets.
The KL divergence for the (cid:96) ∞ perturbation sets are much higher due to the KL divergence being weighted by 0 .
01 in order to produce reasonable samples.
We note that the convolutional network and fully connected network evaluate to about the same, except that the convolutional network has a much lower overapproximation error and is likely restraining the size of the learned perturbation set.
The RTS-1 results demonstrate how ﬁxing the number of perturbations seen during training to one per datapoint is still enough to learn a perturbation set with as much approximation error as a perturbation set with an inﬁnite number of samples, denoted RTS.
Increasing the number of perturbations to ﬁve per datapoint allows the remaining metrics to match the RTS model, and so this suggests that not many samples are needed to learn a perturbation set in this setting. [Table] [Table]
F CIFAR10 COMMON CORRUPTIONS
In this section, we describe the CIFAR10 common corruptions setting and experiments in greater detail.
The dataset comes with 15 common corruptions covering noise, blurs, weather, and digital corruptions.
We omit the three noise corruptions due to similarity to unstructured (cid:96) p noise, leaving us with the following 12 corruptions for each example:
Blurs: defocus blur, glass blur, motion blur, zoom blur 2.
Weather: snow, frost, fog 3.
Digital: brightness, contrast, elastic, pixelate, jpeg
The dataset comes with different corruption levels, so we focus on the highest severity, corruption level ﬁve, so the total training set has 600,000 perturbations (12 for each of 50,000 examples) and the test set has 120,000 perturbations.
The dataset also comes with several additional corruptions meant to be used as a validation set, however for our purposes will serve as a way to measure performance on out-of-distribution corruptions.
These are Gaussian blur, spatter, and saturate which correspond to the blur, weather, and digital categories respectively.
We generate a validation set from the training set by randomly setting aside 1 / 50 of the CIFAR10 training set and all of their corresponding corrupted variants.
These experiments were run on a single GeForce RTX
Ti graphics card, takinghours to train the CVAE and 28 hours to run adversarial training.
F.
P ERTURBATION MODEL ARCHITECTURE AND TRAINING SPECIFICS
We use standard preactivation residual blocks (He et al., 2016), with a convolutional bottleneck which reduces the number of channels rather than downsampling the feature space.
This aids in learning to produce CIFAR10 common corruptions since the corruptions are reﬂective of local rather than global changes, and so there is not so much beneﬁt from compressing the perturbation information into a smaller feature map.
Speciﬁcally, our residual blocks uses convolutions that go from 64 → 16 →channels denoted as Residual (64) .
Then, our encoder and prior networks are as shown in Table 5,
where k = 0 for the prior network and k = 3 for the encoder network, and the decoder network is as shown in Table 6.
Stabilizing the exponential function Note that the CVAE encoders output the log variance of the prior and posterior distributions, which need to be exponentiated in order to calculate the KL distance.
This runs the risk of of numerical overﬂow and exploding gradients, which can suddenly cause normal training to fail.
To stabilize the training procedure, it is sufﬁcient to use a scaled Tanh activation function before predicting the log variance, which is a Tanh activation which has been scaled to output a log variance betwen [ln(10 − 3 ) , ln(10)] .
This has the effect of preventing the variance from being outside the range of [10 − 3 , 10] , and stops overﬂow from happening.
In practice, the prior and posterior converge to a variance within this range, and so this doesn’t seem to adversely effect the resulting representative power of the CVAE while stabilizing the exponential calculation.
Training is done for 1000 epochs using a cyclic learning rate (Smith, 2017), peaking at 0 .
001 on the 400th epoch using the Adam optimizer (Kingma & Ba, 2014) with momentum 0 .
9 and batch size [Table] [Table]
The hyperparameter β is also scheduled to increase linearly from 0 to 10 − 2 over the ﬁrstepochs.
We use standard CIFAR10 data augmentation with random cropping and ﬂipping.
F. P AIRING STRATEGIES
Here
[Table]
we quantitatively evaluate the effect of the different pairing strategies for the CIFAR10 common corruptions setting when training a perturbation set with a CVAE in Table 7.
We ﬁrst note that the approach using random pairs of both original and perturbed data has a larger radius than the others, likely since the generator needs to produce not just the perturbed data but also the original as well.
We next see that all metrics across the board are substantially lower for the approach which centers the prior by always conditioning on the unperturbed example, and so if such an unperturbed example is known, we recommend conditioning on said example to learn a better perturbation set.
F. P ROPERTIES OF THE CVAE LATENT SPACE
We next study closely the properties of the underlying latent space which has been trained to generate
CIFAR10 common corruptions.
We summarize a number of statistics in Table 8, namely the mean, standard deviation, and some percentile thresholds where we can clearly see that the latent vectors for the CVAE centered around the original are much smaller in (cid:96) 2 norm and thus learns the most compact perturbation set.
We remind the reader that maximum (cid:96) 2 norm reported here is calculated on the test set and is not necessarily the same as the radius reported in Table 7 since the radius is selected based on a validation set.
Distribution of (cid:96) 2 distances in latent space We plot the distribution of the (cid:96) 2 norm of the latent space encodings in Figure 5, and further ﬁnd the centered CVAE has semantic latent structure.
Speciﬁcally, corruptions can be ordered by their (cid:96) 2 norm in the latent space where each corruption inhabits a particular range of distance, so reducing the radius directly constrains the type of corruptions in the perturbation set.
For example, restricting the perturbation set to radius (cid:15) = 5 corresponds to leaving out frost fog, snow, elastic, and glass blur corruptions, while containing all defocus blur, jpeg compression, and motion blur corruptions.
A larger version of Figure 5a with a complete legend listing all the corruptions is in Figure 6.
Additional samples and interpolations from the CVAE To further illustrate what corruptions are represented in the CVAE latent space, we plot additional samples and interpolations from the
CVAE.
In Figure 7 we draw four random samples for eight different examples, which show a range of corruptions and provide qualitative evidence that the perturbation set generates reasonable samples.
In Figure 8 we see a number of examples being interpolated between weather, blur, and
[Figure]
[Figure] [Figure]
[Figure]
[Figure]
defocus_blur jpeg_compression motion_blur brightness pixelate contrast zoom_blur frost fog snow elastic_transform glass_blur digital corruptions, which also demonstrate how the perturbation set not only includes the corrupted datapoints, but also variations and interpolations in between corruptions.
F. A DVERSARIAL TRAINING
For learning a robust CIFAR10 classiﬁer, we use a standard wide residual network (Zagoruyko &
Komodakis, 2016) with depth 28 and width factor 10.
All models are trained with the Adam optimizer with momentum 0.9 for 100 epochs with batch size 128 and cyclic learning rate schedule which peaks at 0.2 at epoch 40.
This cyclic learning rate was tuned to optimize the validation performance for the data augmentation baseline, and kept ﬁxed as-is for all other approaches.
We use validation-based early stopping for all approaches to combat robust overﬁtting (Rice et al., 2020).
However, we ﬁnd that in this setting and also likely due to the cyclic learning rate, we do not observe much overﬁtting.
Consequently, the ﬁnal model at the end of training ends up being selected for all methods regardless.
Adversarial training is done with a PGD adversary at the full radius of (cid:15)
=
28 with step size (cid:15)/ 5 = 5 .and 7 iterations.
At evaluation time, for a given radius (cid:15) we use a PGD adversary with 50 steps of size (cid:15)/ 20 .
Additional examples of adversarial attacks under this adversary are in Figure 9, which still appear to be reasonable corruptions despite being adversarial.
F. C OMPARISONS TO OTHER BASELINES
One may be curious as to how these results compare to other methods for learning models which are robust to different threat models.
In this section, we elaborate more on the comparison to other baselines.
We preface this discussion with the following disclaimers:
It not necessarily expected for robustness to one threat model to generalize to another threat model.
It is expected that training against a given threat model gives the best results to that threat model, in comparison to training against a different threat model.
These alternative baselines have different data assumptions (no access to corrupted data) or solve for a different type of robustness than what is being considered in this paper ( (cid:96) p robustness)
Consequently, the fact that these baselines perform universally worse than the CVAE approaches is unsurprising and conﬁrms our expectations.
Nonetheless, we provide this comparison to satiate the readers curiosity.
As alternative baselines, we compared to standard training, methods in general data augmentation (AugMix), (cid:96) 2 adversarial robustness, and (cid:96) ∞ adversarial robustness for CIFAR10.
For (cid:96) 2 and [Table] ∞ adversarially robust models, we download pretrained state-of-the-art models ranked by the
AutoAttack 3 leaderboard.
For AugMix, there is no released CIFAR10 model, so we retrained an
AugMix model using the ofﬁcial AugMix implementation 4 using the parameters recommended by the authors.
We note that the AugMix results for CIFAR10 in this paper are worse than what is reported by Hendrycks et al.
(2019), which is a known issue in the repository that others have encountered (see https://github.com/google-research/augmix/issues/15 ).
F. R ANDOMIZED SMOOTHING
Randomized smoothing is done in its most basic form, with normal Gaussian data augmentation at the speciﬁed noise level σ .
We predict labels with n 0 = 100 samples and do certiﬁcation with n = 10 , 000 samples at conﬁdence α = 0 .
001 , so each example has a 0 .
1% chance of being incorrectly certiﬁed.
Unlike in the typical setting, we have a known maximum radius for the perturbation set.
This allows us to backwards engineer a noise level to have a speciﬁc maximum certiﬁed radius as follows: let (cid:96) be the maximum possible lower bound obtained from a conﬁdence interval at level 1 − α with n classiﬁcations that are in unanimous agreement.
Then, the maximum certiﬁable radius is r = σ Φ − 1 ( (cid:96) ) where Φ − 1 is the inverse cumulative distribution function of a standard normal distribution.
By setting r = (cid:15) , we can solve for the desired noise level σ = r
Φ − 1 ( (cid:96) ) to naturally select a noise level with maximum certiﬁed radius (cid:15) .
We do this for the (cid:15) ∈ { 2 . 7 , 3 . 9 , 10 . 2 } which are the 25 th, 50 th, and 75 th percentiles of the perturbation set, and get σ ∈ { 0 . 84 , 1 . 22 , 3 . 19 } respectively.
The results of randomized smoothing at these thresholds are presented in Table 9.
Note that by construction, each noise level cannot certify a radius larger than what is calculated for.
We ﬁnd that clean accuracy isn’t signiﬁcantly harmed when smoothing at the lower noise levels, and actually improves perturbed accuracy over the adversarial training approach from Table 2 but with lower out-of-distribution accuracy than adversarial training.
For example, smoothing with a noise level of σ = 0 . 84 can achieve 92.5% perturbed accuracy, which is 2% higher than the best empirical approach from Table 2. Furthermore, all noise levels are able to achieve non-trivial degrees of
¾
=0 :
¾ =1 :¾ =3 : certiﬁed robustness, with the highest noise level being the most robust but also taking a hit to standard test set accuracy metrics. Most notably, due to the latent structure present in the perturbation set as discussed in Appendix F.3, even the ability to certify small radii translates to meaningful provable guarantees against certain types of corruptions, like defocus blur and jpeg compression which are largely captured at radius (cid:15) = 3 .
We note that although the largest noise level can theoretically reach the 75th percentile of the perturbation set at radius (cid:15)
= 10 .
2 , none of the examples can be certiﬁed to that extent.
This is possibly a limitation of the certiﬁcation procedure, since although a noise level of σ = 3 .
19 can theoretically certify a radius of up to (cid:15)
= 10 . 2 , the (cid:96) 2 norm of the random samples needed to perform randomized smoothing is approximately 23 , which is well beyond the 75th percentile and towards the boundary of the perturbation set.
The complete robustness curves for all three noise levels are plotted in Figure 10.
G M ULTI - ILLUMINATION
In this ﬁnal section we present the multi-illumination experiments in greater detail with an expanded discussion.
We use the test set provided by Murmann et al.
(2019) which consists of 30 held out scenes and hold out 25 additional “drylab” scenes for validation.
Unlike in the CIFAR10 common corruptions setting, there is no such thing as an “unperturbed” example in this dataset
so we train on random pairs selected from the 25 different lighting variations for each scene.
These experiments were run on a single Quadro RTX 8000 graphics card, taking 16 hours to train the CVAE andhours to run adversarial training.
G. A RCHITECTURE AND TRAINING SPECIFICS
We convert a generic UNet architecture (Ronneberger et al., 2015) to use as a CVAE, by inserting the variational latent space in between the skip connections of similar resolutions.
At a high level, the encoder and prior networks will be based on the downsampling half of a UNet architecture, the conditional generator will be based on the full UNet architecture, and the components will be linked via the latent space of the VAE.
Speciﬁcally, our networks have [64 , 128 , 256 , 512 , 512] channels when downsampling, and twice the number of channels when upsampling due to the concatenation from the skip connection.
Each skip connection passes through a 1 × 1 convolution that reduces the number of channels to 16, and an adaptive average pooling layer that reduces the feature maps to height and width [(127 , 187) , (62 , 93) , (31 , 46) , (15 , 23) , (7 , 11)] respectively.
The adaptive average pooling layer has the effect of a null operator for the MIP5 resolution, but allows higher resolutions [Table] [Table] to use the same architecture.
These ﬁxed-size feature maps are then passed through a fully connected layer to output a mean and log variance for the latent space with dimension [128 , 64 , 32 , 16 , 16] for each respective feature map.
The concatenation of all the latent vectors from each skip connection forms the full, 256 dimensional latent space vector for the CVAE.
Similar to the CIFAR10 setting, we use a scaled Tanh activation to stabilize the log variance calculation
[Table]
The generator of the CVAE UNet is implemented as a typical UNet that takes as input the conditioned example, where intermediate feature maps are concatenated with extra feature maps from the latent space of the CVAE.
Speciﬁcally, each latent space sub-vector is mapped with a fully connected layer back to a feature map with size [(127 , 187) , (62 , 93) , (31 , 46) , (15 , 23) , (7 , 11)] with one channel.
It is then interpolated to the actual feature map size of the UNet (which is a no-op for the MIP resolution) and concatenated to the standard UNet feature map before upsampling.
We train the model for 1000 epochs with batch size 64, using the Adam optimizer with 0 .
9 momentum, weight decay 5 · 10 − 7 , and a cyclic learning rate from [0 . 0001 , 0 .
001 , 0] over [0 , 400 , 1000] epochs.
Same as in the CIFAR10 setting, the learning rate and weight decay were chosen to optimize the validation performance of the baseline data augmentation approach and kept ﬁxed as-is for all other methods.
We weight the KL divergence with a β hyperparameter which is scheduled from [0 , 1 , 1] over [0 , 400 , 1000] , using random ﬂip and crop with padding of 10 for data augmentation.
We then ﬁne tune the model for higher resolutions: for MIP4 we ﬁne tune for 100 epochs with the same learning rate schedule scaled down by a factor of 4, and for MIP3 we ﬁne tune for 25 epochs with the learning rate schedule scaled down by 40.
In order to keep the samples looking reasonable, for
MIP3 we also increase the β weight on the KL divergence to 10 .
Random cropping augmentation is proportionately increased with the size of the image, using padding of 20 and 40 for MIP4 and MIP respectively.
Learning and evaluating an illumination perturbation set at multiple scales The results of this ﬁne tuning procedure to learn a perturbation at higher resolutions are summarized in Tables 10 and 11.
As expected, the quality metrics of the perturbation set get slightly worse at higher resolutions which is counteracted to some degree by the increase in weighting for the KL divergence at MIP3.
Despite using the same architecture size, the perturbation set is able to reasonably scale to images withtimes more pixels and generate reasonable samples while keeping relatively similar quality metrics.
However, to keep computation requirements at a reasonable threshold, we focus our experiments at the MIP5 resolution, which is sufﬁcient for our robustness tasks.
G. A DDITIONAL SAMPLES AND INTERPOLATIONS FROM THE CVAE
We present additional samples and interpolations of lighting changes learned by the CVAE perturbation set.
Figure 11 shows interpolations between three randomly chosen lighting perturbations for four different scenes, while Figure 12 shows 24 additional random samples from the perturbation set, showing a variety of lighting conditions.
These demonstrate qualitatively that the perturbation set contains a reasonable set of lighting changes.
G. A DVERSARIAL TRAINING AND RANDOMIZED SMOOTHING
In this ﬁnal task, we leverage our learned perturbation set to learn a model which is robust to lighting perturbations.
Speciﬁcally the multi-illumination dataset comes with annotated material maps that label each pixel with the type of material (e.g. fabric, plastic, and paper), so a natural task to perform in this setting is to learn material segmentation maps.
We report robustness results at (cid:15) ∈ { 7 . 35 , 8 . 81 , 17 } which correspond to the the 50th, 75th, and 100th percentiles.
We select a ﬁxed lighting angle which appears to be neutral and train a material segmentation model to generate a baseline, which achieves 37 . 2% accuracy on the test set but only 14.9% robust accuracy under the full perturbation model.
We evaluate several methods for improving robustness of the segmentation maps to lighting perturbations, namely data augmentation with the perturbed data, data augmentation with the CVAE, and adversarial training with the CVAE.
The results are tabulated in Table 3.
The CVAE data augmentation approach is not as effective in this setting at improving robust accuracy, as the pure data augmentation approach does reasonably well.
However, the adversarial training approach unsurprisingly has the most robust accuracy, maintaining 35 .
4% robust accuracy under the full perturbation set at (cid:15) = and outperforming the data augmentation approaches.
We plot adversarial perturbations at (cid:15) = 7 .and the resulting changed segmentation maps in Figure 13 for a model trained with pure data augmentation and in Figure 14 for a model trained to be adversarially robust, the latter of which has, on average, less pixels that are effected by an adversarial lighting perturbation.
Adversarial examples at
Wood
Plastic - opaque
Fabric/cloth
Painted
Metal
Plastic - clear
Other
Tile
Plastic - opaque
Painted
Fabric/cloth
Wood
Plastic - clear
Other
Metal
Glass the full radius of (cid:15) = 17 are shown in Figure 15, where we see that the perturbation set is beginning to cast dark shadows over regions of the image to force the model to fail.
Finally, we train a certiﬁably robust material segmentation model using the perturbation set.
We train using a noise level of σ = 6 . 90 which can certify a radius of at most (cid:15) = 17 , or the limit of the perturbation set.
The resulting robustness curve is plotted in Figure 16.
The model achieves 30 .
7% perturbed accuracy and is able to get 12 .
4% certiﬁed accuracy at the 50 th percentile of radius (cid:15) = 7 . 35 .
The key takeaway is that we can now certify real-world perturbations to some degree, in this case certifying robustness to 50% of lighting perturbations with non-trivial guarantees.
Plastic - opaque
Fabric/cloth
Painted
Wood
Plastic - clear
Other
Metal
Glass
[-Data augmentation 45 .38 .
-]
[Figure] [Figure]