A U NIFIED C ONTRASTIVE E NERGY - BASED M ODEL FOR U NDERSTANDING THE G ENERATIVE A BILITY OF A DVERSARIAL T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial Training (AT) is known as an effective approach to enhance the robustness of deep neural networks.
Recently researchers notice that robust models with AT have good generative ability and can synthesize realistic images while the reason behind it is yet under-explored.
In this paper
, we demystify this phenomenon by developing a uniﬁed probabilistic framework called Contrastive
Energy based Models
CEM).
On the one hand
, we provide the ﬁrst probabilistic characterization of AT through a uniﬁed understanding of robustness and generative ability
On the other hand
, our uniﬁed framework can be extended to the unsupervised scenario which interprets unsupervised contrastive learning as an important sampling of CEM
Based on these
, we propose a principled method to develop adversarial learning and sampling methods.
Experiments show that the sampling methods derived from our framework improve the sample quality in both supervised and unsupervised learning.
Notably our unsupervised adversarial sampling method achieves an Inception score of
61 on CIFAR which is superior to previous energy-based models and comparable to state of
-the- art generative models.
I NTRODUCTION
Adversarial Training (AT) is one of the most effective approaches developed so far to improve the robustness of deep neural networks (DNNs) (Madry et al., 2018).
AT solves a minimax optimization problem, with the inner maximization generating adversarial examples by maximizing the classiﬁcation loss, and the outer minimization ﬁnding model parameters by minimizing the loss on adversarial examples generated from the inner maximization.
Recently, researchers have noticed that such robust classiﬁers obtained by AT are able to extract features that are perceptually aligned with humans (Engstrom et al., 2019).
Furthermore, they are able to synthesize realistic images on par with stateof-the-art generative models (Santurkar et al., 2019).
Nevertheless, it is still a mystery why AT is able to learn more semantically meaningful features and turn classiﬁers into generators.
Besides, AT needs the labeled data { ( x i , y i ) } for training while canonical deep generative models do not, e.g. ,
VAE (Kingma & Welling, 2014) and GAN (Goodfellow et al., 2015) only require { x i } .
Thus, it is worth exploring if it is possible to train a robust model without labeled data.
Several recent works (Jiang et al., 2020; Kim et al., 2020; Ho & Vasconcelos, 2020) have proposed unsupervised AT by adversarially attacking the InfoNCE loss (Oord et al., 2018)
(a widely used objective in unsupervised contrastive learning), which indeed improves the robustness of contrastive encoders.
However, a depth investigation and understanding for unsupervised AT is still missing.
To address the above issues, in this work, we propose a uniﬁed probabilistic framework, Contrastive
Energy-based Models (CEM), that provides a principled understanding on the robustness and the generative ability of different training paradigms.
Speciﬁcally, we make the following contributions:
• Demystifying adversarial training and sampling.
We ﬁrstly propose a probabilistic interpretation for AT, that is, it is inherently a (biased) maximum likelihood training of the corresponding energy-based model, which explains the generative ability of robust models learned by AT.
Inspired by this, we propose some novel sampling algorithms with better sample quality than previous methods.
• A uniﬁed probabilistic framework.
Based on the understanding above, we propose Contrastive Energy-based Model (CEM) that incorporates both supervised and unsupervised learning paradigms.
Our CEM provides a uniﬁed probabilistic understanding of previous standard and adversarial training methods in both supervised and unsupervised learning.
• Principled unsupervised adversarial training and sampling.
Speciﬁcally, under our proposed CEM framework, we establish the equivalence between the importance sampling of CEM and the InfoNCE loss of contrastive learning, which enables us to design principled adversarial sampling for unsupervised learning.
Notably,
we show
that the sampling methods derived from our framework achieve state-of-the-art sample quality (9.61 Inception score) with unsupervised robust models, which is comparable to both the supervised counterparts and other state-of-the-art generative models.
R ELATED W ORK
Robust generative models.
Researchers recently notice that features extracted by robust classiﬁers are perceptually aligned with humans, while standard classiﬁers are not (Engstrom et al., 2019; Kaur et al., 2019).
Furthermore, Santurkar et al.
(2019) show that we can also generate images of high quality with robust classiﬁers by iteratively updating from a randomly sampled noise.
They show the sample quality of robust classiﬁers is comparable to state-of-the-art generative models like BigGAN (Brock et al., 2018).
Contrastive learning.
Oord et al.
(2018) ﬁrstly propose unsupervised contrastive learning by maximizing a tractable lower bound on mutual information (MI), i.e., the negative InfoNCE loss.
However, later works ﬁnd that the lower bounds degrade a lot with a large MI, and the success of these methods cannot be attributed to the properties of MI alone (Poole et al., 2019; Tschannen et al., 2020).
Our work provides an alternative understanding of unsupervised contrastive learning as importance sampling of an energy-based model, which also enables us to characterize the limitations of existing methods from a new perspective.
In fact, contrastive learning can also be seen as a general learning framework beyond the unsupervised scenarios.
For example, SupContrast (Khosla et al., 2020) extends contrastive learning to supervised scenarios.
Our work further bridges supervised, unsupervised and adversarial contrastive learning with a uniﬁed probabilistic framework.
CEM: A U NIFIED P ROBABILISTIC F RAMEWORK
Inspired by previous work that bridges discriminative models with energy-based models (Grathwohl et al., 2019), in this work, we propose a uniﬁed framework, called Contrastive Energy-based Model (CEM), that incorporates both supervised and unsupervised scenarios.
Our proposed CEM is a special kind of Energy-based Models (EBMs) that models the joint distribution p θ ( u , v ) over two variables ( u , v ) with a similarity function f θ ( u , v ) deﬁned in a contrastive form,
[Equation]
where Z ( θ )
= (cid:82) exp
( f θ ( u , v ))
d u d v is the corresponding partition function.
In other words, in
CEM, a pair of samples ( u , v ) has higher probability if they are more alike.
In particular, it can be instantiated into the two following variants under different learning scenarios.
Parametric CEM.
In the supervised scenario, we specify the Parametric CEM (P-CEM) that models the joint distribution p θ ( x , y ) of data x and label y in the following form,
[Equation]
where g θ : R n → R m denotes the encoder, g ( x ) ∈ R m is the representation of x , and w k ∈ R m refers to the parametric cluster center of the k -th class.
Denote the linear classiﬁcation weight as
W = [ w 1 , · · · , w K ] and the logit vector as h ( x ) =
g ( x ) (cid:62) W , we can see the equivalence between
P-CEM and JEM (Grathwohl et al., 2019) as
[Equation]
Non-Parametric CEM.
In the unsupervised scenario, we do not have access to labels, thus we instead model the joint distribution between data x and representation z as
[Equation]
and the corresponding likelihood gradient is
[Equation]
In contrastive to P-CEM that incorporates parametric cluster centers, the joint distribution of NPCEM (Non-Parametric CEM) is directly deﬁned based on the similarity between instances.
We deﬁne the joint data distribution p d ( x , z ) =
p d ( x ) p d ( z | x ) through re-parameterization,
[Equation]
(6) where u.a.r.
denotes sampling uniformly at random and T refers to the user-deﬁne pretext, e.g., a set of data augmentation operators T = { t : R n → R n } 1 .
S UPERVISED S CENARIO : R EDISCOVERING A DVERSARIAL T RAINING AS M AXIMUM L IKELIHOOD T RAINING
In this section, we investigate why robust models have a good generative ability.
The objective of
AT is to solve the following minimax optimization problem:
[Equation]
The inner maximization problem is to ﬁnd an adversarial example ˆ x within the (cid:96) p -norm ε -ball around the natural example x that maximizes the CE loss.
While the outer minimization problem is to ﬁnd model parameters that minimize the loss on the adversarial examples ˆ x .
4. M AXIMIZATION P ROCESS
For the inner maximization problem, Projected Gradient Descent (PGD) (Madry et al., 2018) is the commonly used method, which generates the adversarial example ˆ x by maximizing the CE loss(i.e.
, minimizing the log conditional probability) starting from ˆ x 0 = x :
[Equation]
while the Langevin dynamics for sampling P-CEM starts from random noise ˆ x 0 = δ and updates with
[Equation]
Eqns.
8 and 9 both have a positive logsumexp gradient (the second term) to push up the marginal probability p θ (ˆ x ) .
As for the third term, PGD starts from a data point ( x , y ) such that it requires the repulsive gradient to be away from the original data point and do the exploration in a local region.
Langevin dynamics instead starts from a random noise and an additive noise ε is injected for exploration.
Comparing PGD and Langevin.
Following the above analysis, the maximization process in AT can be seen as a model p θ (ˆ x ) .
Compared to Langevin dynamics, PGD imposes speciﬁc inductive bias for sampling.
With the additional repulsive gradient and ε -ball constraint, it explicitly encourages the samples to be misclassiﬁed around the original data points.
In practice, adversarial training with such adversarial examples is generally more stable than training JEM with Langevin samples, which indicates that
PGD attack is a competitive alternative for the negative sampling method for JEM training.
4. M INIMIZATION P ROCESS
To begin with, the gradient of the joint log likelihood for P-CEM can be written as follows:
[Equation]
where ( x , y ) ∼ p
d ( x , y ) denotes the positive data pair, and (ˆ x , ˆ y ) ∼ p θ (ˆ x , ˆ y ) denotes the negative sample pair.
As discussed above, the adversarial examples ˆ x generated by the maximization process can be regarded as negative samples, and ˆ y ∼ p θ (ˆ y | ˆ x ) denotes the predicted label of ˆ x .
To see how the maximum likelihood training of P-CEM is related to the minimization process of AT, we add an interpolated adversarial pair (ˆ x , y ) into Eq.
and decompose it as the consistency gradient and the contrastive gradient:
[Equation]
Next, we show that the two parts correspond to two effective mechanisms developed in the adversarial training literature.
AT loss.
As the two sample pairs in the contrastive gradient share the same input ˆ x , we can see that the contrastive gradient can be written equivalently as
[Equation]
which is exactly the negative gradient of the robust CE loss (AT loss) in Eq.
7, in other words, gradient ascent with the contrastive gradient is equivalent to gradient descent w.r.t.
the AT loss.
Regularization.
As for the consistency gradient, original AT (Madry et al., 2018) simply ignores it.
Its variant TRADES (Zhang et al., 2019) instead proposes the KL regularization KL( p ( ·| ˆ x ) (cid:107) p ( ·| x )) that regularizes the consistency of the predicted probabilities on all classes, whose optimum implies that p ( ·| ˆ x )
= p ( ·| x ) →
f θ ( x , y )
=
f θ ( ˆ x , y ) .
Comparing AT and JEM training paradigms.
The above analysis indicates that the minimization objective of AT is closely related to the maximum likelihood training of JEM (Grathwohl et al., 2019).
Compared to JEM that decomposes the joint likelihood into an unconditional model p θ ( x ) and a discriminative model p θ ( y | x ) , the decomposition of AT in Eq.
10 instead stabilizes training by introducing an intermediate adversarial pair (ˆ x , y ) that bridges the positive pair ( x , y ) and the negative pair (ˆ x , ˆ y ) .
Besides, it can inject the adversarial robustness bias by regularizing the consistency gradient.
Together with our analysis on the maximization process, we show that AT is a competitive alternative for training JEM (a generative model) with more stable training behaviors.
That explains why robust models with AT are also generative.
4. P ROPOSED S UPERVISED A DVERSARIAL S AMPLING A LGORITHMS
Our interpretation of AT also inspires us to design principled sampling algorithms from robust classiﬁers.
Targeted Attack (TA).
Previously, to draw samples from a robust classiﬁer, Santurkar et al.
(2019) utilize a targeted attack that optimizes the input initialized by a random noise ˆ x 0 towards a speciﬁed class
ˆ y ∼ p
d (ˆ y ) :
[Equation]
Compared to PGD attack in Eq. 8, while pushing ˆ x towards ˆ y , TA has a negative logsumexp gradient that decreases the marginal probability p θ (ˆ x ) .
This could explain why TA is less powerful for adversarial attack and is rarely used for adversarial training.
Conditional Sampling (CS).
To overcome the drawback of targeted attack, a natural idea would be dropping the negative logsumexp gradient.
In fact, we can show that this is equivalent to sampling from the conditional distribution:
[Equation]
and its Langevin dynamics takes the form:
[Equation]
Samples drawn this way essentially follow an approximated model distribution, p θ ( ˆ x , ˆ y )
≈ p d (ˆ y )
p
θ (ˆ x | ˆ y ) .
Thus, CS can be seen as a debiased targeted attack algorithm.
Reinforced Conditional
Sampling (RCS).
Inspired by the above analysis, we can design a biased sampling method that deliberately injects a positive logsumexp gradient:
[Equation]
With our designed bias, RCS will sample towards the target class ˆ y by maximizing p θ (ˆ x | ˆ y )
(with the f θ (ˆ x n , ˆ y ) term), and at the same time improve the marginal probability p θ (ˆ x ) (with the logsumexp term).
As shown in our experiment, RCS indeed obtains improved sample quality.
4. D ISCUSSION ON S TANDARD T RAINING
In the above discussion, we have explained why adversarial training is generative from the perspective of CEM.
In fact, it can also help characterize why classiﬁers with Standard Training (ST) are not generative (i.e., poor sample quality).
A key insight is that if we replace the model distribution p θ (ˆ x ) with the data distribution p d ( x ) in Eq. 10, we have
[Equation]
which is the negative gradient of the CE loss in Eq.
Thus, ST is equivalent to training CEM by simply replacing model-based negative samples ˆ x ∼ p θ ( x ) with data samples x ∼ p
d ( x ) .
This approximation makes ST computationally efﬁcient with good accuracy on natural data, but signiﬁcantly limits its robustness on adversarial examples (as model-based negative samples).
Similarly, because ST ignores exploring negative samples while training, standard classiﬁers also fail to generate realistic samples.
P RINCIPLED E XTENSION OF A DVERSARIAL T RAINING TO U NSUPERVISED S CENARIO
In this section, we show that with our uniﬁed framework, we can naturally extend the interpretation developed for supervised adversarial training to the unsupervised scenario.
5. U NDERSTANDING U NSUPERVISED S TANDARD T RAINING
Recently, the following InfoNCE loss is widely adopted for unsupervised contrastive learning of data representations (Oord et al., 2018;
Chen et al., 2020; He et al., 2020),
[Equation] [Equation]
where f θ ( x , ˆ x ) =
g θ ( x ) (cid:62) g θ (ˆ x ) calculates the similarity between the representations of the two data samples, x , ¯ x are generated by two random augmentations (drawn from T ) of the same data example, and
ˆ X = { ˆ x j } Kj =1 consists of K independently drawn negative samples.
In practice, one of the K negative samples is chosen to be the positive sample ¯ x .
Therefore, InfoNCE can be seen as an instance-wise K -class cross entropy loss for non-parametric classiﬁcation.
Perhaps surprisingly, we show that the InfoNCE loss is equivalent to the importance sampling estimate of our NP-CEM (Eq. 4) by approximating the negative samples from p θ ( x ) with data samples from p d ( x ) , as what we have done in standard supervised training (Section 4.4):
[Equation]
which is exactly the negative gradient of the InfoNCE loss.
In the above analysis, for an empirical estimate, we draw N positive pairs ( x i , z i ) ∼ p
d ( x , z ) , and for each anchor x i , we further draw K negative samples { z − ik } from p d (ˆ z ) through reparameterization (Eq. 6).
The negative phase of NP-CEM is supposed to sample from p
θ (ˆ x | ˆ z ) , where samples semantically close to the anchor sample ˆ x , a.k.a.
hard negative samples, should have high probabilities.
However,
InfoNCE adopts a non-informative uniform proposal p d (ˆ z ) for importance sampling, which is very sample inefﬁcient because most samples are useless (Kalantidis et al., 2020).
This observation motivates us to design more efﬁcient sampling scheme for contrastive learning by mining hard negatives.
For example, Robinson et al.
(2021) directly replace the plain proposal with ˜ p
θ (ˆ x | ˆ z ) =
exp( βf θ (ˆ x , ˆ z ))
Z β ( θ ) while keeping the reweighing term.
From the perspective of NP-CEM, it will introduce bias into the importance sampling, which should be treated carefully.
In all, NP-CEM provides a principled framework to characterize and develop contrastive learning algorithms.
5. P ROPOSED U NSUPERVISED A DVERSARIAL T RAINING
AT is initially designed for supervised learning, where adversarial examples can be clearly deﬁned by misclassiﬁcation.
However, it remain unclear what is the right way to do Unsupervised Adversarial Training (UAT) without access to any labels.
Previous works (Jiang et al., 2020; Ho & Vasconcelos, 2020; Kim et al., 2020) have carried out UAT with the adversarial InfoNCE loss, which works well but lacks theoretical justiﬁcation.
Our uniﬁed CEM framework offers a principled way to generalize adversarial training from supervised to unsupervised scenarios.
Maximization Process.
Sampling from p θ ( x ) can be more difﬁcult than that in supervised scenarios because it does not admit a closed form for variable z .
Thus, we perform Langevin dynamics with
K negative samples { z − k } ,
[Equation] [Equation]
While the PGD attack of the InfoNCE loss (Eq. 29),
[Equation]
resembles the Langevin dynamics as they both share the positive logsumexp gradient that pushes up p θ (ˆ x ) , and differs by a repulse negative gradient − f θ (ˆ x , z ) away from the anchor z , which is a direct analogy of the PGD attack in supervised learning (Section 4.1).
Therefore, we believe that the
PGD attack of InfoNCE is a proper way to craft adversarial examples by sampling from p θ ( x ) .
Minimization Process.
Following the same routine in Section 4.2, with the adversarial example ˆ x ∼ p θ (ˆ x ) , we can insert an interpolated adversarial pair (ˆ x , z ) and decompose the gradient of
NP-CEM into the consistency gradient and the contrastive gradient,
[Equation]
In this way, we can directly develop the unsupervised analogy of AT loss and regularization (Sec. 4.2).
Following Eq.
28, it is easy to see that the contrastive gradient is equivalent to the gradient of the Adversarial InfoNCE loss utilized in previous work (Jiang et al., 2020; Ho & Vasconcelos, 2020;
Kim et al., 2020) with adversarial example ˆ x , E
[Equation]
where { ˆ z − ik } denotes the adversarial negative samples from p
θ (ˆ z ) .
5. P ROPOSED U NSUPERVISED A DVERSARIAL S AMPLING
In supervised learning, a natural method to draw a sample ˆ x from a robust classiﬁer is to maximize its conditional probability w.r.t.
a given class ˆ y ∼ p
d (ˆ y ) , i.e., max ˆ x p (ˆ y | ˆ x ) , by targeted attack (Santurkar et al., 2019).
However, in the unsupervised scenarios, we do not have access to labels, and this approach is not applicable anymore.
PGD.
A direct analogy would be using the PGD attack (Eq. 20) for sampling.
However, different from the training process, at the sampling stage, we do not have access to real data points as our anchor points for adversarial attack.
Therefore, PGD attack may not be effective at sampling.
Langevin.
Instead, we believe that Langevin dynamics (Eq. 19) is more suitable for sampling from unsupervised robust models without data and labels.
In particular, we initialize a batch of samples ˆ
X =
{ ˆ x i } independently, and update each sample
ˆ x i by maximizing its marginal probability p θ (ˆ x i ) following Eq.
19, where we take the rest of samples ˆ X\{ ˆ x i } as its negative samples.
This joint evolution will gradually reﬁne the quality of all samples in the batch.
E XPERIMENTS
In this section, we evaluate the adversarial sampling methods derived from our CEM framework, showing that they can bring signiﬁcant improvement to the sample quality of previous work.
Besides, in Appendix A, we also conduct a range of experiments on adversarial robustness to verify our probabilistic understandings of AT.
We show adversarial training objectives derived from our
CEM can indeed signiﬁcantly improve the performance of AT in both supervised and unsupervised scenarios, which helps justify our interpretations and our framework.
6. E XPERIMENTAL S ETUP
Models.
For supervised robust models, we adopt the same pretrained ResNet50 checkpoint on
CIFAR-10 as Santurkar et al.
(2019) 3 for a fair comparison.
The model is adversarially trained
[Table]
with (cid:96) 2 -norm PGD attack with random start, maximal perturbation norm 0 .
5 , step size 0 .
1 andsteps.
As for the unsupervised case, we are the ﬁrst to consider sampling from unsupervised robust models.
We train ResNet18 and ResNet
(He et al., 2016) encoders following the setup of an existing unsupervised adversarial training method ACL (Jiang et al., 2020).
The training attack is kept the same as that of the supervised case for a fair comparison.
More details are provided in
Appendix.
Sampling protocol.
In practice, our adversarial sampling methods take the following general form as a mixture of the PGD and Langevin dynamics,
[Equation]
where g k is the update gradient, ε k is the diffusion noise, Π S is the projector operator, and δ is the (conditional) initial seeds drawn from the multivariate normal distribution whose mean and covariance are calculated from the CIFAR-10 test set following Santurkar et al.
We evaluate the sample quality quantitatively with Inception Score (IS) (Salimans et al., 2016) and Fr´echet Inception
Distance (FID) (Heusel et al., 2017).
More details can be found in in Appendix D.1.
6. R ESULTS
Comparison with other generative models .
In Table 1, we compare the sample quality of adversarial sampling methods with different kinds of generative models.
We analyze the results in terms of the following aspects:
• Our adversarial sampling methods outperform many deep generative models like PixelCNN++, WGAN-GP and PresGAN, and obtain state-of-the-art Inception scores on par with StyleGAN2 (Karras et al., 2020).
• Comparing our AT-based methods with previous methods for training EBMs (Grathwohl et al., 2019; Gao et al., 2021), we see that it obtains state-of-the-art Inception scores among
the EBM-based methods.
Remarkably, our unsupervised CEM with ResNet18 obtains both better IS and FID scores than the original JEM, which adopts WideResNet-28- (Zagoruyko & Komodakis, 2016) with even more parameters.
• Compared with previous AT-based method (Santurkar et al., 2019), our CEM-based methods also improve IS by a large margin (even with the unsupervised ResNet18).
Remarkably, the supervised and unsupervised methods obtain similar sample quality, and the supervised methods are better (higher) at IS while the unsupervised methods are better (lower) at FID.
• We obtain similar Inception scores as state-of-the-art score-based models like NCSN++, while fail to match their FID scores.
Nevertheless, a signiﬁcant drawback of these methods is that they typically require more than 1,000 steps to draw a sample, while ours only require less than 50 steps.
[Table] [Table]
Comparison among adversarial sampling methods.
In Table 2, we further compare the sample quality of different adversarial sampling methods discussed in Sections 4.3 & 5.3.
For supervised models, we can see that indeed TA obtains the lowest IS, while CS can signiﬁcantly reﬁne the sample quality, and RCS can further improve the sample quality by the injected bias.
For unsupervised models, we can see that Langevin dynamics outperforms PGD consistently by a large margin.
In particular, conditional sampling initialized with class-wise noise can improve a little on the sample quality compared to unconditional sampling.
The average visual sample quality in Figure 1 is roughly consistent with the quantitative results.
The mismatch between IS and FID.
A notable issue of adversarial sampling methods is the mismatch between the IS and FID scores.
For example, in Table 1, DDPM and our unsupervised CEM (w/ ResNet50) have similar Inception scores, but the FID of DDPM (2.20) is signiﬁcantly smaller than ours (40.25), a phenomenon also widely observed in previous methods (Santurkar et al., 2019;
Grathwohl et al., 2019; Song & Ermon, 2019).
Through a visual inspection of the samples in Figure 1, we can see that the samples have realistic global structure, but as for the local structure, we can ﬁnd some common artifacts, which could be the reason why it has a relatively large distance (FID) to the real data.
C ONCLUSION
In this paper, we proposed a uniﬁed probabilistic framework, named Contrastive Energy-based
Model (CEM), which not only explains the generative ability of adversarial training, but also provides a uniﬁed perspective of adversarial training and sampling in both supervised and unsupervised paradigms.
Extensive experiments show that adversarial sampling methods derived from our framework indeed demonstrate better sample quality than state-of-the-art methods.
A E VALUATING A DVERSARIAL R OBUSTNESS
In Section 6, we have shown that the new adversarial sampling algorithms derived from our CEM framework indeed obtain improved sample quality, which helps justiﬁes our interpretation of AT from a generative aspect.
In this section, we further take a complementary way to verify our interpretation by studying its discriminative part.
In particular, we develop new variants of AT regularization and verify their effectiveness on improving the adversarial robustness of AT.
As CEM has both supervised and unsupervised variants, we develop AT variants for each scenario respectively while following the same spirit.
A. S UPERVISED A DVERSARIAL T RAINING A.1. P ROPOSED M ETHOD
In Section 4.2, we have mentioned that for the consistency gradient, original AT (Madry et al., 2018) simply ignores it.
Denoting x and ˆ x as the natural and adversarial inputs, the state-of-the-art AT variant, TRADES (Zhang et al., 2019), instead adopts the KL regularization KL( p ( ·| ˆ x ) (cid:107) p ( ·| x )) that explicitly regularizes the consistency of the predicted probabilities on all classes, whose optimum implies that p ( ·| ˆ x )
= p ( ·| x ) →
f θ ( x , y )
=
f θ (ˆ x , y ) .
Inspired by this discussion, alternatively, we can directly regularize the consistency gradient to zero.
We achieve this with the following Consistency Regularization (CR) with least square loss:
[Equation]
We note the our proposed AT+CR differs to ALP (Kannan et al., 2018) as we only minimizes the gap between the logits of the label class f θ ( x , y ) .
ALP was shown to be ineffective for improving
AT, while in the experiments below, we show that our AT+CR objective indeed achieves comparable (even slightly better) results as TRADES.
A.1. E MPIRICAL E VALUATION
Experimental setup.
Following the conventions, we compare different AT methods with Preactivated ResNet18 (He et al., 2016) and WideResNet34 (Zagoruyko & Komodakis, 2016) on
CIFAR-10.
The maximum perturbation is bounded by ε = 8 / 255 under (cid:96) ∞ norm.
The training attack is PGD 10 (Madry et al., 2018) with random start and step size ε/ 4 .
The test attack is PGDwith random start and step size ε/ 4 .
We evaluate both the ﬁnal epoch and the early stopped epoch with the best robust accuracy.
[Table]
Result analysis.
From Table
[Table]
we can see that the AT+CR objective derived from our framework indeed enjoy much better robustness than vanilla AT.
Compared to the state-of-the-art AT variant TRADES, we can see that AT+CR is comparable, and sometimes slightly better at robustness.
When the two have similar robustness (for WideResNet34)
[Table]
AT+CR obtains better natural accuracy than TRADES (86.6 v.s. 83.4).
These results empirically justify that our interpretation of AT and
TRADES from a probabilistic perspective.
A. U NSUPERVISED A DVERSARIAL T RAINING
Similarly, we can develop the same regularization technique for unsupervised adversarial training through a uniﬁed perspective of supervised and unsupervised AT offered by our CEM.
A.2. P ROPOSED M ETHOD
In Section 5.2, we have developed a principled unsupervised adversarial training routine by an analogy with the supervised AT (Section 4.2).
Besides, we can also consider an unsupervised version of the consistency regularization above, namely, the Unsupervised Consistency Regularization (UCR),
[Equation]
which encourages the consistency between the similarity between the natural pair ( x , z ) and the adversarial pair (ˆ x , z ) .
A.2. E MPIRICAL E VALUATION
Experimental setup.
Among the many variants of contrastive learning, we adopt SimCLR as our baseline method and take the recently proposed ACL (Jiang et al., 2020) as our Unsupervised Adversarial Training (UAT) following the same default setup as in Section D.1.
The training attack conﬁguration is kept the same as the supervised case, while after training, we freeze the encoder and ﬁne-tune a linear classiﬁcation layer (standard training) on top with labeled data for evaluating the learned features.
In particular, we evaluate the composed model on the test data with two different attack methods, FGSM (Goodfellow et al., 2015)
(one-step attack) and PGD 20 (multi-step attack), both with ε = 8 / 255 , and report their natural and adversarial accuracy, respectively.
: FGSM (Goodfellow et al., 2015) and PGD (Madry et al., 2018).
[Table]
Result analysis.
As shown in Table 4, features learned by UAT is indeed more robust than standard training, e.g., 0.8% to 21.4% under PGD attack.
Moreover, with our proposed UCR regularizer (Eq. 24), we not only effectively improve both natural accuracy (66.6% to 72.0%), and also improve adversarial robustness: 26.2% to 30.7% under FGSM attack and 21.4% to 24.6% under PGD attack.
This also helps justify the connection between contrastive learning and our CEM.
A. C ONCLUDING R EMARK
With the above experiments on adversarial robustness, we empirically verify that our framework could serve as a valid probabilistic understanding of adversarial training and can be used to develop new effective adversarial training objectives.
B O MITTED T ECHNICAL D ETAILS
For a concise presentation, we have omitted several technical details in the main text.
Here we present a complete description of the derivation process.
B. L OG L IKELIHOOD G RADIENT OF EBM
In Section 3, we have introduced Energy-based Models (EBM) and the gradient of their log likelihood.
We now show how it can be derived out.
For a EBM in the following form,
[Equation]
the gradient of the log likelihood can be derived as follows:
[Equation]
B. C ONNECTION BETWEEN S TANDARD T RAINING AND JEM
In Section 4.
[Equation]
we have claimed that if we replace the model distribution p θ (ˆ x ) with the data distribution p d ( x ) in Eqn.
[Equation]
the log likelihood gradient of JEM is equivalent to the negative gradient of the CE loss.
Here we give a detailed proof as follows:
[Equation]
(ˆ y | x ) ∇ θ f θ ( x , ˆ y ) (cid:3) =
E p d ( x ,y ) (cid:34) ∇ θ f θ ( x , y ) −
K (cid:88) k
= p θ ( k | x ) ∇ θ f θ ( x , k ) (cid:35) =
E p d ( x ,y ) (cid:34) ∇ θ f θ ( x , y ) −
K (cid:88) k
= exp( f θ ( x , k )) ∇ θ f θ ( x , k ) (cid:80) Kj
=
exp
( f θ ( x , j )) (cid:35)
=
E p d ( x ,y ) (cid:34) ∇ θ f θ ( x , y ) −
K (cid:88) k
= ∇ θ exp( f θ ( x , k ))
(cid:80) Kj =1 exp( f θ ( x , j )) (cid:35) =
E p d ( x ,y ) (cid:34) ∇ θ f θ ( x , y ) − ∇ θ (cid:80)
Kk =1 exp( f θ ( x , k ))
(cid:80)
Kj =1 exp( f θ ( x , j )) (cid:35) =
E p d ( x ,y ) (cid:34) ∇ θ f θ ( x , y ) − ∇ θ log
K (cid:88) k
= exp( f θ ( x , k ))
(cid:35) =
E p d ( x ,y ) ∇ θ log exp
( f θ ( x , y ))
(cid:80)
Kk =1 exp( f θ ( x , k ) = ∇ θ E p d ( x ,y ) log p θ ( y | x ) .
B. E QUIVALENCE BETWEEN AT L OSS AND C ONTRASTIVE G RADIENT IN S UPERVISED L EARNING
In Section 4.3, we have claimed that the contrastive gradient equals to the negative gradient of the robust CE loss (AT loss) following the same deduction in Eqn. 27,
[Equation]
which is exactly the negative gradient of the canonical AT loss (Madry et al., 2018).
B. E QUIVALENCE BETWEEN I NFO NCE L OSS AND N ON - PARAMETRIC CEM
In Section 6.
[Equation]
we have claimed that the the log likelihood gradient of NP-CEM equals to exactly the negative gradient of the InfoNCE loss when we approximate p θ ( x ) with p d ( x ) .
The derivation is presented as follows:
[Equation]
[Equation]
B. E QUIVALENCE BETWEEN A DVERSARIAL I NFO NCE AND NP-CEM
In Section 6.
[Equation]
we have developed the unsupervised analogy of AT loss and regularization.
In particular
[Equation]
we have claimed that contrastive gradient is equivalent to the gradient of the Adversarial
InfoNCE loss (i
[Equation]
e., the InfoNCE loss of the adversarial example ˆ x ) utilized in previous work (Jiang et al., 2020; Ho & Vasconcelos, 2020; Kim et al., 2020).
It can be derived following Eqn. 29:
[Equation]
C D ETAILS OF THE P ROPOSED A LGORITHMS
In this section, we provide a detailed description of several algorithms we propose.
C. S UPERVISED A DVERSARIAL S AMPLING A LGORITHMS
We describe the general procedure of supervised adversarial sampling in Algorithm 1.
Input: trained robust classiﬁer f θ ( x ) , test dataset D test ; step size α , (cid:96) 2 -norm maximal range β , noise scale η , number of sampling steps N , and speciﬁc update function g ( x ) ;
Output: drawn sample ˆ x N ;
Initialization.
Drawn seed ˆ x 1 =
δ as a random noise n ∼ N ( µ , Σ ) , where µ , Σ are estimated from D test ; for each step n ∈ [1 , N − 1] do
Proposal step
[Equation]
Projection step.
Project x (cid:48) to the feasible set by: if (cid:107) x (cid:48) − ˆ x n (cid:107) 2 ≤ β then
Let ˆ x n +1 = x (cid:48) ; else
Let ˆ x n +
=
ˆ x n + β x (cid:48) − ˆ x n (cid:107) x
(cid:48) − ˆ x n (cid:107) 2 ; end if
The four sampling algorithms discussed in our paper have the following update functions g ( x ) :
• Targeted Attack (TA) (Santurkar et al., 2019): given a drawn class
y ∼ p
d ( y ) (ﬁxed along the updating), the update rule of
TA follows
[Equation]
• Our Conditional Sampling (CS) : given a randomly drawn class
y ∼ p
d ( y ) (ﬁxed along the updating), the update rule of CS follows
[Equation]
• Our Reinforced Conditional Sampling (RCS) :
given a randomly drawn class
y ∼ p
d ( y ) (ﬁxed along the updating), the update rule of RCS follows
[Equation]
C. U NSUPERVISED A DVERSARIAL S AMPLING A LGORITHMS
We describe the general procedure of supervised adversarial sampling in Algorithm 2.
The two sampling algorithms discussed in our paper have the following update functions g ( x ) :
• PGD.
Given a batch of examples { ˆ x k } K k =1 , the PGD attack has the following update function g (ˆ x k ) for ˆ x k ,
[Equation]
where its positive pair ¯ x k is a randomly drawn sample.
Input: trained unsupervised robust model f θ ( x ) , test dataset D test ; step size α , (cid:96) 2 -norm maximal range β , noise scale η , number of steps N , batch size K , and speciﬁc update function g ( x ) ;
Output: drawn samples { ˆ x kN } Kk =1 ;
Initialization.
Drawn K seeds { ˆ x k 1 } Kk =1 , where ˆ x k 1 = δ k refers to a random noise δ k ∼ N ( µ , Σ ) , where µ , Σ are estimated from D test ; for each step n ∈ [1 , N − 1] do
Proposal step.
For each sample x kn at step n , we propose ( x k ) (cid:48)
=
ˆ x kn + α g
(ˆ x kn ) (cid:107) g (ˆ x kn ) (cid:107) 2 + η ε k , ε k ∼
N ( 0 , 1 );
Projection step.
For each sample x kn at step n , we project ( x k ) (cid:48) to the feasible set by: if (cid:107) ( x k ) (cid:48) − ˆ x kn (cid:107) 2 ≤ β then
Let ˆ x kn +
=
( x k ) (cid:48) ; else
Let ˆ x kn +
=
ˆ x kn + β ( x k ) (cid:48) − ˆ x kn (cid:107) ( x k ) (cid:48)
− ˆ x kn (cid:107) 2 ;
[Figure]
9 10 Maximal 2 norm
I n c ep t i on sc o r e
• Langevin.
Given a batch of examples { ˆ x k } Kk =1 , the Langevin dynamics has the following update function g (ˆ x k ) for ˆ x k ,
[Equation]
D M ORE D ETAILS AND R ESULTS ON A DVERSARIAL S AMPLING D. D ETAILED E XPERIMENTAL S ETUP
Supervised Adversarial Sampling.
For supervised robust models, we adopt the same pretrained
ResNet50 checkpoint on CIFAR-10 as Santurkar et al.
(2019) 4 for a fair comparison.
As described in Santurkar et al.
(2019), the ResNet-50 model is adversarially trained for 350 epochs with learning rate 0.01 and batch size 256.
The learning rate is dropped by 10 at epoch 150 and 200.
The training attack is (cid:96) 2 -norm PGD attack with random start, maximal perturbation norm 0.5, step size 0.1 andsteps.
Unsupervised Adversarial Sampling.
As far as we know, we are the ﬁrst to consider sampling from unsupervised robust models.
Therefore, to obtain an unsupervised robust model for sampling, we adopt ACL (Jiang et al., 2020) as the baseline method and follow their default hyper-parametersto train it.
The ofﬁcial implementation of ACL is built upon the SimCLR (Chen et al., 2020) framework for contrastive learning, while they choose a speciﬁc hyper-parameter conﬁguration for
[Figure]
I n c ep t i on sc o r e 0.
3.7 Noise ratio 1.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
I n c ep t i on sc o r e
14 Maximal 2 norm 6.
I n c ep t i on sc o r e
[Table]
adversarial training.
In particular, they choose 512 for batch size and train for 1000 epochs with
ResNet-18 backbone (He et al., 2016).
The base learning rate is 1.0, where they use a linear warm up strategy for the ﬁrst 10 epochs, and apply cosine annealing scheduler after that.
The training attack is kept the same as that of the supervised case for a fair comparison.
For ResNet-18, we adopt the ACL(A2A) setting, which adopts the normal ResNet with only one Batch Normalization module.
Instead, for ResNet-50, we notice that the ACL(A2S) setting yields slightly better results.
The ResNet variants in the ACL(A2S) setting contains two BN modules, where we assign natural and adversarial examples to different modules.
We refer more details to the original paper (Jiang et al., 2020).
Evaluation of sample quality.
Note that there are four hyper-parameters in our sampling protocol: step size α , (cid:96) 2 -ball size β , noise scale η , and iteration steps K , for which we list our choice in Table 5.
We evaluate sample quality quantitatively w.r.t.
the Inception Score (IS) (Salimans et al., 2016) and Fr´echet Inception Distance (FID) (Heusel et al., 2017) with 50,000 samples, where the standard variation of IS is around 0.1.
D. A DDITIONAL A NALYSIS
Besides the results presented in the main text, we also conduct more experiments to analyze the behavior of our proposed adversarial sampling algorithms, both quantitatively and qualitatively.
We conduct a detailed analysis of our proposed sampling algorithms and present the results of supervised adversarial sampling (with RCS) in Figure 2 and the results of unsupervised adversarial sampling (with Langevin) in Figure 3.
Note that in both cases, we adopt the ResNet-50 backbone and use the default hyper-parameters unless speciﬁed.
D.2. C HAIN L ENGTH
From the left panels of Figure 2 and Figure 3, we can see the two adversarial algorithms both have a sweet spot of sampling steps N (length of the sampling Markov chains) at around 30 to 40 steps, before and after which the results will be slightly worse.
D.2. N OISE RATIO
In the proper Langevin dynamics, the scale of the noise is determined by the step size, η = √ 2 α .
However, in practice, this would lead to a catastrophically degraded sample quality as the noise takes over the gradient information.
Therefore, following Song & Ermon (2019) and Grathwohl et al.
(2019), we also anneal the noise ratio η to a smaller value for better sample quality.
As shown in the middle panels of Figure 2 and Figure 3, the optimal noise ratio is around 0.01 for both cases.
D.2. M AXIMAL N ORM
An apparent difference of our adversarial sampling algorithms to the canonical Langevin dynamics is that ours have a projection step that limits the distance between the samples and the initial seeds.
In the right panels of Figure 2 and Figure 3, we show the impact of the scale of the (cid:96) 2 -norm ball for the sample quality.
We can see that generally speaking, as the ball grows larger, the samples get reﬁned.
In the supervised case, the sample quality gets slightly worse with a large norm, which does not happen in the unsupervised case.
D.2. S AMPLING T RAJECTORY
Aside from the qualitative inspection of the proposed sampling algorithms, we also demonstrate the sampling trajectory of our supervised (RCS) and unsupervised (Langevin) adversarial sampling methods in Figure 4 & 5.
We can see that the samples get gradually reﬁned in terms both low-level textures and high-level semantics.