Certiﬁably Robust Variational Autoencoders
Abstract
We introduce an approach for training Variational Autoencoders (VAEs) that are certiﬁably robust to adversarial attack.
Speciﬁcally, we ﬁrst derive actionable bounds on the minimal size of an input perturbation required to change a VAE’s reconstruction by more than an allowed amount, with these bounds depending on certain key parameters such as the Lipschitz constants of the encoder and decoder.
We then show how these parameters can be controlled, thereby providing a mechanism to ensure a priori that a VAE will attain a desired level of robustness.
Moreover, we extend this to a complete practical approach for training such
VAEs to ensure our criteria are met.
Critically, our method allows one to specify a desired level of robustness upfront and then train a VAE that is guaranteed to achieve this robustness.
We further demonstrate that these Lipschitz–constrained
VAEs are more robust to attack than standard VAEs in practice.
Introduction
Variational autoencoders (VAEs) are a powerful method for learning deep generative models [1, 2], ﬁnding application in areas such as image and language generation [3, 4] as well as representation learning [5].
Yet like other deep learning methods [6], VAEs are susceptible to adversarial attacks, whereby small perturbations of an input can induce meaningful, unwanted changes in output.
For example, VAEs can be induced to reconstruct images similar to an adversary’s target through only moderate perturbation of the input image [7, 8, 9].
This is undesirable for two main reasons.
First, VAEs have been used to improve the robustness of classiﬁers [10, 11] and the encodings of VAEs are also commonly used in downstream tasks [12, 13].
Second, the susceptibility of VAEs to distortion from input perturbations challenges an original ambition for VAEs: that they should capture “semantically meaningful [...] factors of variation in data” [14].
If this ambition is to be fulﬁlled, VAEs should be more robust to spurious inputs, and so the robustness of VAEs is intrinsically desirable.
While previous work has already sought to obtain more robust VAEs empirically [15, 16, 17], this work lacks formal guarantees.
This is a meaningful worry because in other model classes, robustiﬁcation techniques showing promise empirically but lacking guarantees have later been circumvented by more sophisticated attacks [18, 19].
It stands to reason that existing techniques for robustifying
VAEs might be similarly ineffectual.
Further, though previous theoretical work [20] can ascertain robustness post-training , it cannot enforce and control robustness a priori , before training.
Our work looks to alleviate these issues by providing VAEs whose robustness levels can be controlled and certiﬁed by design.
To this end, we show how certiﬁably robust VAEs can be learned by enforcing Lipschitz continuity in the encoder and decoder, which explicitly upper-bounds changes in their outputs with respect to changes in input; we call the resulting models Lipschitz-VAEs .
[Figure]
Unlike those of the standard VAE, the Lipschitz-VAE’s reconstructions are robust to the attack.
In
Appendix E we supplement these results with latent space attacks (see Eq. 3).
We derive two different bounds on the robustness of these models, each covering a slightly different setting.
First, we derive a per-datapoint lower bound that guarantees a certain probability of reconstructions of distorted inputs being close to the reconstructions of undistorted inputs.
More precisely, this per-datapoint lower bound is on the probability that the (cid:96) 2 distance between an attacked
Lipschitz-VAE’s reconstruction and its original reconstruction is less than some value r .
This probability is with reference to the stochasticity of sampling in a VAE’s latent space.
Using this bound we can then obtain a margin that holds for all inputs.
This second, global bound means that we can guarantee, for any input, that perturbations within the margin induce reconstructions that fall within an r –sized ball of the original reconstruction with at least some speciﬁed probability.
The latter margin is the ﬁrst of its kind for VAEs: a margin that does not depend on the value of the input data and can have its value speciﬁed a priori from setting a small number of network hyperparameters.
It thus enables VAEs with a chosen level of robustness.
In summary, our contributions are to develop a novel approach to inducing robustness in VAEs through Lipschitz continuity constraints, and to show theoretically that VAEs with such constraints are endowed with certiﬁable robustness properties, such that we can choose a desired level of robustness upfront and then ensure it is achieved.
We experimentally validate our approach (see Figure 1), and in doing so realize the ﬁrst VAEs that are certiﬁably robust.
Background 2.1 VAEs
Assume we have a collection of observations D
= { x 1 , . . .
, x n } with x ∈ X , which is generated according to an unknown process involving latent variables z ∈ Z .
We want to learn a latent variable model with joint density p θ ( x , z ) =
p θ ( x | z ) p ( z ) , parameterized by θ , that captures this process.
Learning θ by maximum likelihood is often intractable and variational inference addresses this intractability by introducing inference model q φ ( z | x ) [14], parameterized by φ , which yields a tractable lower bound on the marginal likelihood,
[Equation]
Here, KL ( ·||· ) denotes the Kullback-Leibler divergence, while θ and φ represent the parameters of deep neural networks — the decoder and encoder network respectively — which can be optimized using unbiased gradient estimates obtained through Monte Carlo samples from q φ ( z | x ) .
Given a VAE, we will refer to sampling z i ∼ q φ ( z | x i ) on input x i as the encoding process , and — following convention — to g θ
( z i ) as a reconstruction of x i , where g θ ( · ) denotes the deterministic component of the decoder [21].
2.2 Adversarial Attacks on VAEs
In adversarial attacks on machine learning models, an adversary tries to alter the behavior of a model.
Although much work has focused on classiﬁers, adversarial attacks have also been proposed for VAEs, whereby the model is “fooled” into reconstructing an unintended output.
More formally, given original input x o and the adversary’s target output x t , the attacker seeks a perturbation δ ∈ X such that the VAE’s reconstruction of the perturbed input ( x o + δ ) is similar to x t .
The best performing attack on VAEs in the current literature is a latent space attack [7, 8, 9], where an adversary perturbs input x o to have a posterior q φ similar to that of the target x t , optimizing arg min
[Equation]
In Eq.
(2), the ﬁrst term encourages similarity between the two posterior distributions, the second term favors smaller perturbations such that the original input x o is altered less, and λ is a hyperparameter controlling this trade-off.
In our work we strictly constrain this norm by some constant c ∈ R + to ensure more consistent comparisons:
[Equation]
We also use another type of attack, the maximum damage attack [20], which for z δ ∼ q φ ( z | x o + δ ) , z ¬ δ ∼ q φ ( z | x o ) , and some constant c ∈ R + optimizes
[Equation]
2.3 Deﬁning Robustness in VAEs
VAE reconstructions are typically continuous–valued, and a VAE’s encoder, q φ ( z | x ) , is usually chosen to be a continuous distribution.
Any change to a VAE’s input will thus almost surely result in a change in its reconstructions, since changes to the input will translate to changes in q φ ( z |· ) , and in turn, almost surely to changes in the reconstruction g θ ( z ) [20].
This observation rules out established robustness criteria that specify robustness using margins around inputs within which model outputs are constant [22, 23].
To further complicate matters,
VAEs are probabilistic: a VAE’s outputs will vary even under the same input.
To account for these considerations, we employ the robustness criterion of [20]:[-
-]
Deﬁnition 2.1.
( r -robustness) For r ∈ R + , a model f operating on a point x and outputting a continuous random variable is r -robust to a perturbation δ if and only if
[Equation]
The notion of r -robustness states that a model is robust if, more likely than not, changes in the model’s outputs induced by an input perturbation δ fall within a hypersphere of radius r about the model’s outputs on the unperturbed input.
The smaller the value of r for which r -robustness holds, the stricter the notion of robustness which is implied.
We will refer to P [ || f
( x + δ )
− f ( x ) || 2 ≤ r ] as the r -robustness probability .
Note that while we pick a threshold of 0 .
5 for notational simplicity, r -robustness admits any threshold in [0 , 1) , and indeed can be made arbitrarily strong to suit the level of robustness required.
The deﬁnition of r -robustness naturally leads to the notion of an r -robustness margin [20]:
Deﬁnition 2.2.
( r -robustness margin) For r ∈ R + , a model f has r -robustness margin R r ( x ) about input x if || δ
|| 2 <
R r ( x ) = ⇒ P
[ || f ( x + δ ) − f ( x ) || 2 ≤ r ] > 0 .
An r -robustness margin upper-bounds the norm that an input perturbation can have while rrobustness is preserved.
If a model has r -robustness margin R r ( x ) for input x , we can guarantee that the model will not be undermined by any perturbation of x with norm less than R r
( x ) [20].
2.4 Lipschitz Continuity
Deﬁnition 2.3.
(Lipschitz continuity)
A function f : R n → R m is Lipschitz continuous if for all x 1 , x 2 ∈ R n , || f ( x 1 ) − f ( x 2 ) || 2 ≤ M ||
x
1 − x 2 || 2 for constant M ∈ R + .
The least M for which this holds is called the Lipschitz constant of f .
If a function f is Lipschitz continuous with Lipschitz constant M , we say that f is M -Lipschitz.
Certiﬁably Robust VAEs 3.1 Lipschitz-VAEs
We now introduce our approach for achieving a VAE whose robustness levels can be controlled and certiﬁed.
We do so by targeting the “smoothness” of a VAE’s encoder and decoder network, requiring these to be Lipschitz continuous, since a VAE’s vulnerability to input perturbation is thought to inversely correlate with the smoothness of its encoder and decoder.
By choosing and maintaining
Lipschitz continuity with a known, set Lipschitz constant, we will be able to obtain a chosen degree of robustness a priori .
3.2 Bounding the r -Robustness Probability
We ﬁrst construct an approach for guaranteeing that a VAE’s reconstructions will change only to a particular degree under distortions.
We achieve this by specifying our VAEs such that their rrobustness probability is bounded from below.
Our bounds depend on the Lipschitz constants of the constituent networks, the magnitude of the distortion and the encoder’s standard deviation.
In the standard setting this yields an input-dependent characterization of the behavior of the VAE, while taking the encoder standard deviation to be a hyperparameter yields global, input-agnostic bounds.
This means that for a given input perturbation norm we can guarantee similarity up to a threshold with a particular probability.
Our bounds provide the ﬁrst global guarantees about the robustness behavior of a VAE.
We use the (cid:96) 2 distance as our notion of similarity as it corresponds to the log probability of a
Gaussian — a frequently-used likelihood function for VAEs with continuous data — and has also been the basis for previous theoretical work on VAE robustness [20].
The following result shows that, under the common choice of a diagonal-covariance multivariate
Gaussian encoder, a lower bound on the r -robustness probability can be provided for LipschitzVAEs.
We use the parameterization q φ ( z | x ) = N (cid:16) z ; µ φ ( x ) , diag (cid:16) σ 2 φ ( x ) (cid:17)(cid:17) , where µ φ : X →
R d z is the encoder mean and σ φ : X → R d z ≥ 0 is the encoder standard deviation .
Theorem 1 (Probability Bound) .
Assume q φ ( z | x ) is as above and that the deterministic component of the Lipschitz-VAE decoder g θ ( · ) is a -Lipschitz, the encoder mean µ φ ( · ) is b -Lipschitz, and the encoder standard deviation σ φ ( · ) is c -Lipschitz.
Finally, let z δ ∼ q φ ( z | x + δ ) and z ¬ δ ∼ q φ ( z | x ) .
Then for any r ∈ R + , any x ∈ X , and any input perturbation δ ∈ X ,
P [Equation] { p 1 ( x
[Equation]
, p 2 ( x ) } , where p 1 ( x ):=min
[Equation]
and p
[Equation]
x
[Equation]
:= [Equation]
[Equation]
.w. for u ( x ) :
= ( ra − b ||
δ || 2 )
( c || δ ||
{++
+}
||
σ φ ( x ) || 2 ) 2 and constant C ( d z ) :
= 1 √ π exp (cid:8)
[Equation]
d z [Equation]
Proof.
See Appendix.
(cid:4)
Theorem 1 tells us that a Lipschitz-VAE’s r -robustness probability can be bounded in terms of: r ; the Lipschitz constants of the encoder and decoder; the norm of the encoder standard deviation; the dimension of the latent space; and the norm of the input perturbation.
The term involving the norm of the input perturbation is most important, as it allows us to link the magnitude of input perturbations to the probabilities of distortions in reconstructions.
The proof leverages the Lipschitz continuity of the decoder network to relate the distances between reconstructed points in X to the corresponding distances between their latents in Z .
The
Lipschitz continuity of the encoder then allows the distribution of distances between samples in latent space — from perturbed and unperturbed posteriors q φ ( z | x + δ ) and q φ ( z | x ) respectively — to be characterized in terms of distances between inputs.
We note that the distribution of (cid:96) 2 distances between these samples is a generalized χ 2 distribution, which has no closed-form
CDF [24].
The proof therefore employs two tail bounds, Markov’s Inequality and a tail bound for standard χ 2 distributions, which varyingly dominate each other in tightness for different ||
δ
|| 2 (see
Figure 2) and respectively yield p 1 ( x ) and p 2 ( x ) .
[Figure]
3.3 Bounding the r -Robustness Margin
While Theorem 1 allows for the r -robustness probability of a Lipschitz-VAE to be lower-bounded for a given input and input perturbation, ideally we would like to guarantee a VAE’s robustness at a given input to all input perturbations up to some magnitude.
The following result provides exactly such a guarantee for Lipschitz-VAEs, in terms of a lower bound on the r -robustness margin.
Lemma 1.1 (Margin Bound) .
Given the assumptions of Theorem 1 and a Lipschitz-VAE satisfying these assumptions, the r -robustness margin of this VAE on input x ,
[Equation]
where
[Equation]
and m 2 ( x ) :
= sup {|| δ || 2 : p 2 ( δ , x ) ≤ 0 . 5 } , where p 2 ( δ , x ) is as deﬁned in Theorem 1 but we make explicit the dependence on δ .
Proof.
See Appendix.
[Equation]
Lemma 1.1 shows that we can lower-bound the radius R r about x within which no input perturbation can undermine r -robustness.
In particular, when at least one of m 1 ( x ) and m 2 ( x ) is positive, robustness can be certiﬁed.
The proof exploits the relationship established in Theorem 1 between the r -robustness probability and the magnitude of input perturbations, ﬁnding the largest input perturbation norm such that our lower bound on the r -robustness probability still exceeds 0 .
3.4 A Global r -Robustness Margin
A global margin can now be obtained via Theorem 1.
We wish to bound R ( x ) from below for all x ∈ X .
The only input dependence is via σ φ ( x ) , which can be lifted, however, by setting σ φ ( x ) = σ ∈ R d z ≥ 0 , a chosen hyperparameter.
This can be done either during training — since VAEs can be trained with a ﬁxed encoder standard deviation without serious degradation in performance [25] — or after, since all that matters to the bound is the value of σ at test time 2 .
Theorem 2 (Global Margin Bound) .
Given the assumptions of Theorem 1 and a Lipschitz-VAE satisfying these assumptions, but with σ φ ( x ) = σ ∈ R d z , the global r -robustness margin of this
VAE for all inputs is
[Equation]
where
[Equation]
[Equation]
b for (cid:16) 4 || σ || 22 − 0 .
5 (cid:0) ra (cid:1) 2 (cid:17) < 0 ; and m 2 := sup {|| δ
|| 2 : p 2 ( δ ) ≤ 0 . 5 } , where p 2 is as deﬁned in
Theorem 1, but with u :=
( ra − b ||
δ || 2 )
2 4 || σ ||.
Proof.
See Appendix.
[Equation]
This result provides guarantees solely in terms of parameters we can choose ahead of training, namely the Lipschitz constants of the networks and σ , the ﬁxed value of the encoder standard deviation.
This importantly distinguishes ours from previous work, which has only provided robustness bounds based on intractable model characteristics that must be empirically estimated after training [20].
Implementing Lipschitz-VAEs
In the last section we introduced guarantees on robustness given the Lipschitz constants of the VAE’s networks.
We now consider how to train a VAE in a manner that ensures these guarantees are met.
Letting F be the set of functions that can be learned by an unrestricted neural network, and L M ⊂
F be the (further restricted) subset of M -Lipschitz continuous functions associated with the sets of neural network parameters L θM , L φM , our constraint can be thought of simply as replacing the standard VAE objective in Eq.
(1) with the modiﬁed objective
[Equation]
Referring to VAEs trained this way as Lipschitz-VAEs , the question becomes how to enforce this objective.
Using [26], we focus on fully-connected networks, although similar ideas extend to convolutional architectures [27].
First, note that if layer l has Lipschitz constant M l , then the Lipschitz constant of the entire network is M = (cid:81) Ll =1 M l [6].
For an L -layer fully-connected neural network to be M -Lipschitz, it thus sufﬁces to ensure that each layer l has Lipschitz constant M 1 L .
If we choose the network non-linearity ϕ l ( · ) to be 1 -Lipschitz, and ensure that linear transformation W l is also 1 -Lipschitz, then Lipschitz constant M 1 L in layer l follows from scaling the outputs of each layer’s linear transformation by M 1 L .
Building on this, our approach to controlling the Lipschitz continuity of VAE encoders and decoders can be seen in Algorithm 1.
The key components are Bj ¨ orck Orthonormalization, which ensures each
{+Algorithm 
+}
The forward pass in a Lipschitz-VAE’s encoder or decoder network.
Bj¨orckOrthonormalizefor k = 1 , . .
K do
[Equation]
where Q ( k ) = I − (cid:16) W ( k ) l (cid:17) (cid:124) W ( k ) l , and K and p are hyperparameters.
Input: Data point x
Result:
Network output h L
Requires: Lipschitz constant M
Forward pass h 0 ← x for l = 1 , . . .
, L do
W l ← Bj¨orckOrthonormalize( W l ) pre-activation ← M 1 L W l h l −h l ← GroupSort( pre-activation )
[Figure]
layer’s linear transformation is 1 -Lipschitz (see Appendix B for details), and the norm-preserving
GroupSort non-linearity from [26], which is 1 -Lipschitz.
This function groups the entries of matrixvector product W l h l − 1 in each layer l into some number of groups, and then sorts the entries of each group by ascending order.
It can be shown that when each group has size two, for any scalar y
[Equation]
Experiments
Our aim now is to establish that our theoretical results allow us to certify and guarantee the robustness of VAEs in practice.
Additionally, we would like to verify that Lipschitz continuity constraints can endow VAEs with greater robustness to adversarial input perturbations than standard VAEs.
We pick a latent space with dimension d z = 10 (unless otherwise stated) and use the same architecture across experiments: encoder mean µ φ ( · ) , encoder standard deviation σ φ ( · ) and deterministic component of the decoder g θ ( · ) are all three-layer fully-connected networks with hidden dimensions 512 (for more details, see Appendix F).
We now validate that our bounds in Lemma 1.1 and Theorem 2 allow us to provide absolute robustness guarantees.
In particular, for a given Lipschitz-VAE, we compute max { m 1 ( x ) , m 2 ( x ) } and max { m 1 , m 2 } for Lemma 1.1 and Theorem 2 respectively on a randomly-selected sample from MNIST and Fashion-MNIST (see Figure 4).
This experiment highlights two notable aspects.
First, it empirically validates our bounds, since in all instances the estimated r -robustness margins (see the following section) are larger than the corresponding bounds on these margins provided by Lemma 1.1 and Theorem 2.
Second, we see that the bounds on the r -robustness margin are strictly positive, providing a priori guarantees of robustness when choosing a ﬁxed encoder standard deviation and encoder and decoder network Lipschitz constants as in the setting of Theorem 2.
Our results demonstrate the existence of Lipschitz-VAEs for which meaningful robustness can be certiﬁed, a priori.
We empirically assess the r -robustness margins of
Lipschitz-VAEs using the approach of [20], leveraging maximum damage attacks as in Algorithm 2 (see Appendix).
In particular, assuming no defects in the optimization of Eq.
(4) and access to inﬁnite samples from the encoder q φ ( z |· ) , if for a given c a maximum damage attack cannot identify a δ ∗ such that P [ || g θ ( z δ ∗ ) − g θ ( z ¬ δ ∗ )
|| 2 ≤ r ] ≤ 0 .
5 , then we can rest assured that the r -robustness margin of a VAE on input x is at least c – that is, R r ( x ) ≥ c .
If one VAE’s estimated r -robustness margins are consistently larger than another’s, this strongly suggests that the former is more robust.
In Figure 5 (left), we estimate the r -robustness margins of several Lipschitz- and standard-VAEs on a randomly-selected collection of images from the MNIST test set.
On the same inputs and for all Lipschitz constants considered, Lipschitz-VAEs exhibit larger estimated r -robustness margins on average than a standard VAE.
[Figure]
The above result also allows us to verify an implication of our theory, namely that a LipschitzVAE’s r -robustness margins should broadly be larger the smaller its Lipschitz constants are.
Our empirical ﬁndings exactly corroborate this, since the average r -robustness margins we estimate in
Lipschitz-VAEs monotonically increase as we decrease in their Lipschitz constants.
Hence, Figure 5 (left) demonstrates that we can manipulate the robustness levels of Lipschitz-VAEs through judicious choices of their Lipschitz constants, fulﬁlling our objective to develop a VAE whose robustness levels could be controlled a priori .
Note that in these experiments (with a unit
Gaussian prior), we found the useful range of Lipschitz constants for all networks to be between around 5 and 10 .
Less than this the reconstructive performance of the Lipschitz-VAE is excessively impacted, while greater than this the Lipschitz-VAE behaves comparably to a standard VAE in terms of robustness.
We previously evaluated the robustness of LipschitzVAEs.
We have not yet empirically explored, however, whether Lipschitz-VAEs are otherwise different from standard VAEs as models, for example in the functions they learn.
As shown in Figure 6, the aggregate posteriors learned by Lipschitz- and standard VAEs differ in their scale.
The aggregate posterior of a standard VAE is tightly clustered about the prior p ( z ) =
N ( z ; 0 , I ) , but that of the Lipschitz-VAE disperses mass more widely over the latent space.
Though this could be an issue when generating samples from the prior, as the prior and aggregate posterior have little overlap, the remedy to this issue is very simple.
We ﬁnd that upweighting the
KL term by a hyperparameter β , as in a β -VAE [5], mitigates this scaling of the latent space (see
Figure 6c).
For the details of this exploration, see Appendix C.
Choosing Lipschitz Constants
Previously, we saw that the r -robustness margins of a LipschitzVAE could be manipulated through its Lipschitz constants, with smaller Lipschitz constants consistently affording greater robustness.
In practice, however, robustness might only be one consideration, alongside reconstruction performance, in choosing between VAEs.
To explore these considerations, we plot reconstruction performance against estimated robustness in
Figure 5 (right), measuring reconstruction performance as the mean log likelihood achieved, and estimating robustness in terms of R r ( x ) .
Recalling that larger log likelihoods imply better reconstructions, we see that reconstruction performance is negatively correlated with estimated robustness, with behavior on each of these dimensions determined by the Lipschitz constants.
We note that our bounds are relatively loose (see Figures 2 and 4).
This is consistent with applications of Lipschitz continuity constraints in other settings [22]: while such conditions enable certiﬁable bounds, resulting bounds are also inherently loose because Lipschitz continuity is “stricter” than conditions of “normal” continuity (which may allow for tighter but nonenforceable and/or non-certiﬁable bounds [20]).
Nevertheless, our approach is useful in scenarios where robustness must be absolutely guaranteed, even if at times that guarantee is weaker than the practical behavior.
Related Work Certiﬁable Robustiﬁcation
Prior work on robustifying models to adversarial attacks can be delineated into techniques which empirically provide robustness to known types of adversarial attack, and certiﬁable techniques providing provable robustness under certain assumptions.
It has been argued that certiﬁable techniques should be favored [22], since empirical ﬁndings of robustness are predicated on a choice of attack and thus cannot indicate effectiveness against other known or as yet unknown attacks.
Indeed, we previously noted instances where empirical techniques seemed to induce robustness but were subsequently undone by later-developed attacks [18, 19].
Given their advantages, certiﬁable robustiﬁcation techniques have already been targeted in classiﬁers, where approaches employing Lipschitz continuity are particularly illustrative.
In particular [28, 29, 26, 30] use Lipschitz continuity to provide certiﬁed robustness margins for classiﬁers.
We note, however, that in this setting one does not need to handle the probabilistic aspects and continuous changes that one ﬁnds in VAEs.
In the VAE context, [15] argues that the susceptibility of a VAE to adversarial perturbations depends on two factors: how much the encoder q φ ( z | x ) can be changed through small changes in input x , and how much reconstruction g θ ( z ) can be changed through small changes in the latent variable z .
Relating these factors to the “smoothness” of the encoder and decoder, [15] targets greater smoothness by controlling the noisiness of the VAE encoding process, so that “nearby” inputs correspond to “nearby” latent variables and changes in q φ ( z |· ) induced by an input perturbation have little effect on reconstruction g θ ( z ) .
Similarly, [17] holds that adversarial examples are possible in VAEs due to non-smoothness in the encoding-decoding process, relating this to dissimilarity between a VAE’s reconstructions of its reconstructions.
Lastly, r -robustness is proposed in [20], which obtains an approximate bound on the r -robustness margin of VAEs that allows their robustness to be assessed.
That work assumes however that input perturbations only affect the encoder’s mean, not its standard deviation.
These works only allow for the assessment of the robustness of already trained VAEs.
Unlike our methods, they do not directly enforce guaranteed robustness.
Conclusion
We have introduced an approach to training VAEs that allows their robustness to adversarial attacks to be guaranteed a priori .
Speciﬁcally, we derived provable bounds on the degree of robustness of a VAE under input perturbation, with these bounds depending on parameters such as the Lipschitz constants of its encoder and decoder networks.
We then showed how these parameters can be controlled, enabling our bounds to be invoked in practice and thereby presenting an actionable way of ensuring the robustness of a VAE ahead of training.
A Proofs
Theorem 1 (Probability Bound) .
Assume q φ ( z | x ) = N (cid:16) z ; µ φ ( x ) , diag (cid:16) σ 2 φ ( x ) (cid:17)(cid:17) and that the deterministic component of the Lipschitz-VAE decoder g θ ( · ) is a -Lipschitz, the encoder mean µ φ ( · ) is b -Lipschitz, and the encoder standard deviation σ φ ( · ) is c -Lipschitz.
Finally, let z δ ∼ q φ ( z | x + δ ) and z ¬ δ ∼ q φ ( z | x ) .
Then for any r ∈ R + , any x ∈ X , and any input perturbation δ ∈ X ,
P [ || g θ ( z δ ) − g θ ( z ¬ δ ) ||
2 ≤ r ] ≥ 1 − min { p 1 ( x ) , p 2 ( x ) } , where p 1 ( x ):= min
[Equation]
[Equation]
and p
[Equation]
x ) :=
[Equation]
o.w. for u ( x ) :=
( ra − b ||
δ || 2 )
( c ||
δ
||
+
||
σ φ ( x ) || 2 ) 2 and constant C ( d z ) :
= 1 √ π exp (cid:8) 1 2 ( d z − ( d z − 1) log d z ) (cid:9) .
Proof.
Since g θ ( · ) is a -Lipschitz,
[Equation]
for all z 1 , z 2 ∈ Z .
Now assume z 1 ∼ q φ ( z | x 1 ) and z 2 ∼ q φ ( z | x 2 ) for some x 1 , x 2 ∈ X , such that g θ ( z 1 ) and g θ ( z 2 ) are random variables.
Eq.
(5) then implies
[Equation]
which in turn implies
[Equation]
Letting x 1 = x
+ δ and x 2 = x such that z 1 = z δ and z 2 =
z ¬ δ , q φ ( z | x ) =
N (cid:16) z ; µ [Equation] [Equation] means
[Equation]
and
[Equation]
Further, since samples from q φ ( z |· ) are drawn independently in every VAE forward pass, we also know z δ and z ¬ δ are independent, and thus, because the difference of independent multivariate
Gaussian random variables is multivariate Gaussian,
[Equation]
Returning to (6), since || z δ − z ¬ δ || 2 is a continuous random variable, we can write
[Equation]
The proof now diverges, yielding p 1 ( x ) and p 2 ( x ) respectively.
Obtaining p 1 ( x ) : Recall Z = R d z , apply the deﬁnition of the (cid:96) 2 norm [Equation] and invoke Markov’s
Inequality to obtain
[Equation]
Now note that
[Equation]
so that by the linearity of expectations,
[Equation]
Because z δ − z ¬ δ is diagonal-covariance multivariate Gaussian, the ( z δ − z ¬ δ )
j are jointly independent for all j = 1 , . . .
, d z , and so we recognize that
[Equation]
has a non-central χ 2 distribution with one degree of freedom and non-centrality parameter
[Equation]
Since for a non-central χ 2 random variable Y with n degrees of freedom and non-centrality parameter (cid:15) [31], E [ Y ] = n + (cid:15) , we have E
[Equation]
, and so plugging into (9),
E
[Equation]
Using d z
[Equation]
j = [Equation] x
[Equation]
||(the deﬁnition of the (cid:96) 2 norm), and || µ φ ( x + δ ) − µ φ ( x ) || 2 ≤ b ||
δ || 2 ,
(since µ φ ( · ) is b -Lipschitz), we obtain d
[Equation]
Similarly, using
[Equation]
(where the above inequality follows from σ φ : X → R d z ≥ 0 , and the last equality follows from the deﬁnition of the (cid:96) 2 norm), and
[Equation]
(where the ﬁrst inequality follows by the triangle inequality, and the second follows from the assumption that σ φ ( · ) is c -Lipschitz), we ﬁnd d z (cid:88)
[Equation]
Hence, returning to (8), we see
[Equation]
r
[Equation]
, such that
[Equation]
Noting that the right-most term is non-negative, and wanting to have a well-deﬁned probability, we take p 1 ( x
[Equation]
:=min
[Equation]
such that
[Equation]
Obtaining p 2 ( x ) : Return to Eq.
By the triangle inequality,
[Equation]
and hence
[Equation]
Then, again recalling Z = R d z ,
[Equation]
[Equation]
where the ﬁrst equality uses the deﬁnition of the (cid:96) 2 norm [Equation] and the above inequality between probabilities uses the inequality from (16) [Equation]
Now [Equation] since
[Equation]
it follows that
[Equation]
In particular [Equation] note that since z δ − z ¬ δ is diagonal-covariance multivariate Gaussian, the
[Equation]
are jointly independent for all j = 1 , . . .
, d z .
Hence, because the sum of squares of d z independent standard Gaussian random variables has a standard χ 2 distribution with d z degrees of freedom,
[Equation]
Letting
[Equation] (cid:0) ra − || µ φ ( x + δ ) − µ φ ( x ) || 2 (cid:1)
( c ||
δ ||
+ 2 || σ φ ( x ) || 2 )and u ( x ) :
= (cid:0) ra
− b
||
δ || 2 (cid:1)
( c ||
δ ||
+ 2 || σ φ ( x ) || 2 ) 2 , we have u (cid:48)
( x ) ≥ u ( x ) by the assumption that µ φ ( · ) is b -Lipschitz, since || µ φ ( x + δ ) − µ φ ( x ) || 2 ≤ b ||
δ || 2 , and therefore (cid:16) r a − || µ φ ( x + δ ) − µ φ ( x ) || 2 (cid:17) ≥ (cid:16) r a − b || δ || 2 (cid:17)
(note also that ( c || δ ||
+ 2 || σ φ ( x ) || 2 ) 2 ≥ 0 ).
Then, using (21) with the requirement that
[Equation]
to ensure the inequality in (20) is meaningful,
[Equation]
The tail bound for standard χ 2 random variables in (3.1) from [32] (which requires u ( x ) > d z −and d z ≥ 2 ) then yields
[Equation]
for constant C ( d z ) := 1 √ π exp (cid:8) 12 ( d z − ( d z − 1) log d z ) (cid:9) .
Since the expression on the right-hand side is non-negative under the above conditions, we deﬁne p
[Equation]
[Equation]
to ensure a well-deﬁned probability.
Then, by the inequalities starting from (17),
[Equation]
Obtaining the ﬁnal bound:
Choosing the least of p 1 ( x ) and p 2 ( x ) to obtain the tighter upper bound on P (cid:2)
||
z δ − z
¬
δ
|| 2 ≥ ra (cid:3)
, we can plug in to (7), which gives
[Equation]
Lemma 1.1 (Margin Bound) .
Given the assumptions of Theorem 1 and a Lipschitz-VAE satisfying these assumptions, the r -robustness margin of this VAE on input x ,
[Equation]
where
[Equation]
and m 2 ( x ) :
= sup {|| δ || 2 : p 2 ( δ , x ) ≤ 0 . 5 } , where p 2 ( δ , x ) is as deﬁned in Theorem 1 but we make explicit the dependence on δ .
Proof.
By Theorem 1, for any input perturbation δ ∈ X and any input x ∈ X ,
[Equation]
Hence, for our Lipschitz VAE to be r -robust to perturbation δ on input x , by Deﬁnition 2.1 it sufﬁces that [Equation] min { p [Equation]
p
[Equation]
x [Equation] } > 0 . 5 . Recalling Deﬁnition 2.2, since for a model f R r ( x ) is deﬁned by
[Equation]
] > [Equation] [Equation]
for our Lipschitz-VAE R r ( x ) is at least the maximum perturbation norm such that
[Equation]
or equivalently,
[Equation]
(where we make explicit the dependence on δ ).
Denoting m 1 ( x ) := sup {||
δ || 2 : p 1 ( δ , x ) ≤ 0 .
5 } and rearranging, m 1 ( x ) becomes sup
[Equation]
Excluding the degenerate case of c = 0 , that is assuming c > 0 , this is attained at the maximum root of the quadratic equation
[Equation]
provided a root exists, and so by the quadratic formula,
[Equation]
The second case does not admit a closed-form solution, so we will simply write [Equation]
| [Equation] p [Equation]
Choosing the maximum of m 1 ( x ) and m 2 ( x ) then yields
[Equation]
Theorem 2 (Global Margin Bound) .
Given the assumptions of Theorem 1 and a Lipschitz-VAE satisfying these assumptions, but with σ φ ( x ) = σ ∈ R d z , the global r -robustness margin of this
VAE for all inputs is
[Equation]
where
[Equation]
b for (cid:16) 4 || σ || 22 − 0 .
5 (cid:0) ra (cid:1) 2 (cid:17) < 0 ; and m 2 := sup {|| δ
|| 2 : p 2 ( δ ) ≤ 0 . 5 } , where p 2 is as deﬁned in
Theorem 1, but with u :=
( ra − b ||
δ || 2 )
2 4 || σ ||.
Proof.
Given a ﬁxed encoder standard deviation, that is substituting σ φ ( x ) = σ ∈ R d z , we ﬁrst have to derive a lower bound on the r -robustness probability to then bound the r -robustness margin globally.
We do this using the machinery of Theorem 1, which — lifting the now-redundant requirement that the encoder standard deviation be c -Lipschitz — can be invoked without loss of generality.
In the case of p 1 (recall the two bounds in the proof of Theorem 1), plugging in σ yields [Equation] [Equation] =
[Equation]
for p 1 := a [Equation] [Equation]
|
[Equation]
22 [-+
-]+
[Equation]
(where the penultimate step follows by (10) and (16)).
In the case of p 2 , we can directly substitute, obtaining
[Equation]
for
[Equation]
and u := ( ra −
[Equation] | [Equation]
| [Equation]
Theorem 2 then follows by identical reasoning to Lemma 1.1.
[Equation]
B Implementing Lipschitz-VAEs
Previously, we assumed that Lipschitz continuity could be imposed in a VAE’s encoder and decoder.
In practice, ensuring the Lipschitz continuity of a deep learning architecture is non-trivial.
Using [26] as a guide, this section outlines how to provably control the Lipschitz constants of an encoder and decoder network.
We deﬁne a fully-connected network with L layers as the composition of linear transformations W l and element-wise activation functions ϕ l ( · ) for l = 1 , . . .
, L , where the output of the l -th layer h l := ϕ l ( W l h l − 1 ) .
We let network input x =: h 0 and network output y :=
h L .
B.1 Ensuring Lipschitz Continuity with Constant
We would like to ensure a fully-connected network is M -Lipschitz for arbitrary Lipschitz constant
M .
It has been shown that a natural way to achieve this is by ﬁrst requiring Lipschitz continuity with constant 1 [26].
As 1 -Lipschitz functions are closed under composition, if we can ensure that for every layer l ,
W l and ϕ l ( · ) are 1 -Lipschitz, then the entire network will be 1 -Lipschitz.
Most commonly-used activation functions, such as the ReLU and Sigmoid, are already 1 -Lipschitz [33, 34], and hence we need only ensure that W l is also 1 -Lipschitz.
This can be done by requiring W l to be orthonormal, since W l being 1 -Lipschitz is equivalent to the condition
[Equation]
24) where || W l || 2 equals the largest singular value of W l .
The singular values of an orthonormal matrix all equal 1 , and so the orthonormality of W l implies (24) is satisﬁed.
In practice, W l can be made orthonormal through an iterative algorithm called Bj¨orck Orthonormalization , which on input a matrix A ﬁnds the “nearest” orthonormal matrix to A [26].
Bj¨orck
Orthonormalization is differentiable and so allows the encoder and decoder networks of a LipschitzVAE to be trained using gradient-based methods, just like a standard VAE.
B.2 Ensuring Lipschitz Continuity with Arbitrary Constants
Now that we can train a 1 -Lipschitz network, we would like to generalize this method to arbitrary
Lipschitz constant M .
To do so, note that if layer l has Lipschitz constant M l , then the Lipschitz constant of the entire network is M = (cid:81) Ll =1 M l [6].
Hence, for our L -layer fully-connected neural network to be M -Lipschitz, it sufﬁces to ensure that each layer l has Lipschitz constant M 1 L .
This is actually simple to achieve, because if we continue to assume ϕ l ( · ) is 1 -Lipschitz, Lipschitz constant M 1 L in layer l follows from scaling the outputs of each layer’s linear transformation by M 1 L .
B.3 Selecting Activation Functions
While the above approach is sufﬁcient to train networks with arbitrary Lipschitz constants, a result from [26] shows it is not sufﬁcient to ensure the resulting networks are also expressive in the space of Lipschitz continuous functions.
Informally, the result states that the expressivity of a Lipschitzconstrained network is limited when its activation functions are not gradient norm-preserving [26] .
Since activation functions such as the ReLU and the Sigmoid do not preserve the gradient norm, the expressivity of Lipschitz-constrained networks that use such activations will be further limited.
To address this, [26] introduces a gradient norm-preserving activation function called GroupSort , which in each layer l groups the entries of matrix-vector product W l h l − 1 into some number of groups, and then sorts the entries of each group by ascending order.
It can be shown that when each group has size two,
[Equation]
for any scalar y [26].
Unless we need to restrict a network’s outputs to a speciﬁc range, we employ the GroupSort activation in our implementation of Lipschitz-VAEs.
C Investigated Learned Latent Spaces
While we are primarily interested in the robustness of Lipschitz-VAEs, we may also wish to understand whether Lipschitz-VAEs differ from standard VAEs as models, for example in the functions they learn.
To build our understanding in this regard, we study the latent spaces learned by Lipschitz-VAEs, training standard and Lipschitz-VAEs with latent space dimension d z = 2 and visualizing their learned encoders q φ ( z | x ) .
As shown in Figure 6, the encoders learned by Lipschitz and standard
VAEs differ in their scale.
Whereas the encoder of a standard VAE remains tightly clustered about the prior p ( z ) =
N ( z ; 0 , I ) , the encoders of the Lipschitz-VAEs disperse mass widely in latent space.
This apparent rescaling of the latent space in Lipschitz-VAEs has two important consequences, the ﬁrst of which is that the prior and encoder have little overlap.
This is signiﬁcant because it is common to generate data points with a trained VAE by drawing samples from the prior and passing these to the decoder.
In a rescaled latent space where the prior and encoder have little overlap, many samples from the prior will be “out-of-distribution” inputs to the decoder.
The second consequence of the latent space being rescaled is that there risks being less overlap between q φ ( z |· ) for any two inputs.
In the limit, the latent space then devolves into a look-up table [35], which is undesirable because the meaning of interpolated points in latent space — that is, points between areas of high density in terms of q φ ( z |· ) — is lost.
We speculate that the rescaling of latent spaces in Lipschitz-VAEs can be explained by the relative importance of the likelihood and KL terms, log p θ ( x | z ) and KL ( q φ ( z | x ) || p
( z )) respectively, in the
VAE objective in (1).
By Deﬁnition 2.3, a Lipschitz continuous function is one whose rate of change is constrained, so in some sense such a function is “simpler” than others not satisfying the property.
It seems plausible then that — to achieve good input reconstructions while using simpler functions than a standard VAE — a Lipschitz-VAE might rescale the latent space to be able to adequately differentiate between latent samples corresponding to different inputs.
This might happen even at the expense of the encoder being distant from the prior, causing KL ( q φ ( z | x ) || p ( z )) to grow, since the likelihood term typically dominates the KL term and so gains in the likelihood term from rescaling the latent space might outweigh the resulting penalty from the KL term.
We test this hypothesis by training Lipschitz-VAEs with the KL term upweighted by hyperparameter β , as in a β -VAE [5] (we term Lipschitz-VAEs trained with this modiﬁed objective Lipschitz βVAEs ).
As can be seen in Figure B.7, and as predicted by our hypothesis, we ﬁnd that by increasing the weight assigned to the KL term — that is, using β > 1 — the scaling of the latent space is mitigated.
In sum, the experiments in this section reveal that Lipschitz-VAEs learn qualitatively different encoders from standard VAEs, exhibiting rescaling behavior that we link both to the challenge of performing reconstructions using Lipschitz continuous functions and the characteristics of the VAE objective.
Our experiments also outline how possible adverse effects of Lipschitz continuity con-
[Figure]
straints on data generation and latent space interpretability might be addressed through a small modiﬁcation of the VAE objective.
D Estimating R r ( x )
Algorithm 2 [20]’s algorithm to estimate r -robustness margin R r ( x ) . Starting with estimate max R and decrementing by step size α at each iteration (until reaching 0 ), the algorithm performs T maximum damage attacks with input perturbations constrained to the current estimate for the rrobustness margin. The ﬁrst time r -robustness is satisﬁed under all T attacks, the algorithm returns the current estimate as the estimated r -robustness margin ˆ R r ( x ) .
Inputs : x , r , starting estimate max R , step size α , number of samples S , number of random restarts
T
Output: Estimated r -robustness margin
ˆ R r ( x )
Estimation routine ˆ R r ( x ) ← max R while ˆ R r ( x ) > 0 do probabilities ← [] for t = 1 , . .
, T do // Performs a maximum damage attack according to the objective in (4) δ t ← MaxDamageAttack with the constraint ||
δ || 2 ≤ ˆ R r
x ) , randomly initialized distances ← [] for s = 1 , . .
. , S do z δ t ∼ q φ ( z | x + δ t )
z
¬ δ t ∼ q φ ( z | x ) distances.append( || g θ
( z δ t ) − g θ ( z ¬ δ t ) || 2 ) //
Estimates the r -robustness probability probability ← length(distances[distances ≤ r ]) S probabilities.append(probability) //
Checks that the estimated probabilities are greater than 0.5, across random restarts if length(probabilities[probabilities > 0 . 5 ])
=
T then return ˆ R r ( x ) ˆ R r ( x ) ← ˆ R r ( x ) − α // Indicates when no positive r -robustness margin is found
[Figure]
F Network Architectures
To properly handle reconstructions on [0 , 1] -valued data, we let the likelihood in the VAE objective be Continuous Bernoulli [36].
In the Lipschitz-VAEs we train, all activation functions bar the ﬁnal-layer activations are chosen to be the GroupSort activation (recall Section B.3), while in the standard VAEs we train, these are chosen to be the ReLU.
In both types of VAE, the ﬁnal-layer activation in the encoder standard deviation σ φ ( · ) uses a Sigmoid to ensure positivity, while the ﬁnal-layer activation in the deterministic component of the decoder uses a Sigmoid to restrict reconstructions for binary data.
The ﬁnal layer of the encoder mean takes no activation function.
All models were trained on a 13-inch Macbook Pro from 2017 with 8GB of RAM and 2 CPUs.