I DENTITY -D ISENTANGLED A DVERSARIAL A UGMENTATION FOR S ELF - SUPERVISED L EARNING
A BSTRACT
Data augmentation is critical to contrastive self-supervised learning, whose goal is to distinguish a sample’s augmentations (positives) from other samples (negatives).
However, strong augmentations may change the sample-identity of the positives while weak augmentation produces easy positives/negatives leading to nearlyzero loss and ineffective learning.
In this paper we study a simple adversarial augmentation method that can modify training data to be hard positives/negatives without distorting the key information about their original identities.
In particular we decompose a sample x to be its variational auto-encoder (VAE reconstruction
G ( x ) plus the residual R ( x ) = x − G ( x ) , where R ( x ) retains most identitydistinctive information due to an information-theoretic interpretation of the VAE objective.
We then adversarially perturb G ( x ) in the VAE’s bottleneck space and adds it back to the original R ( x ) as an augmentation, which is therefore sufﬁciently challenging for contrastive learning and meanwhile preserves the sample identity intact.
We apply this “identity disentangled adversarial augmentation
IDAA
” to different self supervised learning methods
On multiple benchmark datasets
IDAA consistently improves both their efﬁciency and generalization performance
We further show that
IDAA learned on a dataset can be transferred to other datasets
I NTRODUCTION
Empowered by deep neural networks and the computational capability of recent hardware/infrastructure, machine learning (ML) has achieved great breakthroughs on some challenging problems when sufﬁcient labeled data is available.
However, deep learning is known to be data-hungry and annotating data in many domains, e.g., medical care or predictions of protein structures, are either difﬁcult or expensive.
To overcome this limitation, self-supervised learning (SSL) methods train a model on unlabeled data in a supervised learning manner using self-generated labels by manipulating the data, e.g., rotation degrees (Gidaris et al., 2018), Jigzaw puzzle solutions (Noroozi & Favaro, 2016), clustering (Caron et al., 2018), back-translation (Zhu et al., 2017), etc.
These methods recently start to perform on par with supervised learning and exhibit potential to even surpass it Chen et al.
; Chen & He (2021).
Moreover, their learned representations can be generally applied to different downstream tasks.
Many widely-used SSL methods are built upon sample-identity preservation tasks, e.g., contrastive learning (Oord et al., 2018; Tian et al., 2020a;
Chen et al., 2020) and consistency regularization (Chen & He, 2021;
Grill et al., 2020; Caron et al., 2020), which aim at learning representations that can preserve the identity of the original sample after applying data augmentations and thus distinguish the augmentations of different samples.
For example, contrastive learning targets on an representation space in which a sample (anchor) is closer to its own augmentations (positives) than other samples or their augmentations (negatives).
The effectiveness of contrastive learning therefore heavily depends on the quality of data augmentations.
Most SSL methods (Ye et al., 2019; Chen et al., 2020; He et al., 2020; Chen & He, 2021) utilize pre-deﬁned data augmentations to generate positives and negatives.
However, as shown in the example (green point) of Fig. 1, they are not adaptive to the data manifold in the embedding space during training and the generated augmentations can be too easy for the sample identiﬁcation task.
In practice, these SSL methods need tens to hundreds times of epochs required by supervised learning to reach comparable performance on the downstream classiﬁcation tasks (Chen et al., 2020;
He et al., 2020; Chen & He, 2021).
For the same reason, large batch-size is common and necessary for contrastive learning in order to involve sufﬁcient hard negatives.
Therefore, how to modify the positives/negatives for more informative contrastive loss is a critical yet open challenge towards more efﬁcient SSL.
Learnable and adaptive data augmentations have been explored for supervised
[Figure]
D i s t a n c e o f a n c h o r t o t h e n e
[Figure]
r e s t n e g a t i v
[Figure]
i s t
[Figure]
n c
[Figure]
o f a n c h o r t o t h e n e a r e s t n e g a t
i v e
[Figure]
IDAACLAE
[Figure]
c) learning (Cubuk et al., 2018; 2020; Liu et al., 2021) but have not been widely studied in SSL.
Probably the most related augmentation method for SSL is CLAE (Ho & Vasconcelos, 2020), which generates hard positives/negatives by adversarially attacking the input.
However, as shown in Fig. 1, adversarial augmentations may change the original sample identity and it is infeasible to tune the attack strength for every sample to preserve the identity.
Hence, another challenge for data augmentation in SSL is how to preserve sample identity.
Although hard positives/negatives might be achievable by stronger augmentations or adversarial augmentations listed above, they can also distort the true identities of the original samples as the red point in Fig.so the model may erroneously identify other different samples or their augmentations as the anchor sample.
Training with those identity-changed augmentations might lead to trivial solutions for SSL and poor representations.
To address the aforementioned two primary challenges of contrastive learning, we have to consider how they interfere with each other.
For better efﬁciency, the data augmentation needs to generate positives and negatives as challenging as possible for the model to distinguish the sample identity, but it should not remove or distort the minimum necessary information retaining the true identity.
Thereby, the sample identiﬁcation task is neither too trivial nor infeasible to learn.
In this paper, we study how to automatically generate data augmentations that fulﬁll the above conditions and improve both the efﬁciency and effectiveness of current self-supervised learning.
We relate the objective of variational antoencoder (VAE) with the sample identiﬁcation task in contrastive learning from an information-theoretical perspective, which inspires us to disentangle the identity-essential information of an input x as the residual of VAE reconstruction G ( x ) , i.e., R ( x ) = x − G ( x ) .
As illustrated in Fig.
2, in order to modify x to be more challenging in terms of sample identiﬁcation, we propose to apply adversarial perturbations to the bottleneck features of VAE, which maximize the contrastive loss in an (cid:15) -ball and results in a modiﬁed G (cid:48)
( x ) .
We then utilize x (cid:48)
=
G (cid:48)
( x ) + R ( x ) as a data augmentation of x so the identity information captured by R ( x ) remains intact in x (cid:48) .
Our method, called “identity-disentangled adversarial augmentation (IDAA)”, only needs a pretrained VAE model and does not require any labeled data.
In the experiments on multiple benchmarks, when applied to different SSL methods, this simple yet principal data augmentation approach consistently brings improvements on both the efﬁciency and the performance on downstream tasks.
In addition, we present a thorough ablation study to analyze the inﬂuence of hyperparameters (e.g., VAE hyperparameters, bottleneck dimensions, (cid:15) controlling the attack strength) and experimental settings (e.g., batch size, training epochs, model architecture) on the contrastive learning’s performance.
B ACKGROUND
Contrastive learning (CL) (Wu et al., 2018; Zhuang et al., 2019; Tian et al., 2020a; Ye et al., 2019;
Kalantidis et al., 2020; Hu et al., 2021; Robinson et al., 2020) aims at learning representations that can distinguish different samples and their augmentations.
Speciﬁcally, it formulates this sampleidentiﬁcation task as a classiﬁcation problem on each sample (anchor), where the positives are its own augmentations and the negatives are other samples and their augmentations.
A widely-used loss for contrastive learning is “InfoNCE” (Tian et al., 2020a) built upon the positives and negatives created by data augmentations.
For each mini-batch x = { x 1 , x 2 , · · · , x N } of size N , InfoNCE loss 𝒛 𝑮 𝒙 𝒙 − 𝑮 𝒙
Identity-relevant Identity-disentangled … 𝑬 ⋅ 𝑫 ⋅ … …
𝑮 ⋅ 𝒙
Original image
Adversarially attacked 𝑮(𝒙) Augmented data … … 𝑮′ 𝒙 𝒙 − 𝑮 𝒙 + 𝑮′(𝒙) + = -
Adversarial attack: 𝒛
[Figure]
is computed as:
[Equation]
where (cid:104)· , ·(cid:105) denotes inner product, A ( · ) and B ( · ) denote two different transformations for data augmentation, f ( · ) and h ( · ) are the embedding networks that can be either identical (Chen et al., 2020;
Ye et al., 2019) or different (Misra & Maaten, 2020;
He et al., 2020), and q NCE ( i | x
= x i ) is the softmax probability of identifying the augmentation B ( x i ) is transformed from the original sample x i .
The choices of data augmentation are critical to the performance and efﬁciency of CL: weak augmentations might already result in fully distinguishable representations over different samples and nearly zero CL loss, while strong augmentations may overly distort the identity of a sample and make CL too challenging, infeasible, or inefﬁcient.
In previous works such as SimCLR (Chen et al., 2020), different (compositions of) augmentations can result in large gaps on CL’s performance but ﬁnding the best one needs brute-force search of all possible transformations and their compositions, which is usually computationally forbidden in practice.
Another challenge in CL is to keep a large memory bank or mini-batch to cover sufﬁcient amount of “hard negatives” because a sample might be distant from most other samples (and their augmentations) and hard negatives close to it are sparse in the training set.
Wu et al.
(2018) trains a non-parametric classiﬁer to maximally scatter the representations of all samples over a unit sphere.
However, it needs to build a memory bank to store all these representations, which is infeasible for large-scale datasets.
He et al.
(2020) instead maintain a ﬁxed-sized dynamic dictionary based on a FIFO queue of representations of incoming mini-bathes during training.
It ﬁnds that a relatively large dictionary is critical to the success of CL.
The lack of hard negatives forms a bottleneck to the sample efﬁciency of CL.
Therefore, how to select or modify the data to increase the chance of including more hard negatives is a signiﬁcant open problem for CL.
Consistency regularization consistency regularization (CR) has been studied in several recent works (Chen & He, 2021;
Grill et al., 2020; Caron et al., 2020) for self-supervised or semi-supervised learning, which achieves comparable or even better performance than CL in certain scenarios.
Compared to CL, CR removes the comparison to negatives and only focus on maximizing the similarity between model outputs for two augmentations of the same sample, e.g.,
[Equation]
Similar to CL, CR also aims at preserving the sample identity on the learned representations and it heavily relies on the choice of data augmentations.
For example, FixMatch (Sohn et al., 2020) chooses to apply multiple weak augmentation for generating the pseudo- strong
I DENTITY -D ISENTANGLED A DVERSARIAL A UGMENTATION
In this section, we will ﬁrstly relate the sample-identiﬁcation task broadly used in designs of selfsupervised learning with the training objective of variational auto-encoder (VAE).
Speciﬁcally, we will show that VAE’s training tries to remove the sample-identity related information from its bottleneck features.
Hence, the residual of VAE reconstruction may retain the sample-identity information that we wish to preserve in the data augmentation for self-supervised learning.
We then propose a data generation and augmentation model based on VAE and analyze the lower bound for identity preservation in its augmented data.
In the end of this section, we apply adversarial attack methods to this data augmentation model, which modify samples to be hard positives and negatives for more efﬁcient contrastive learning without changing their sample-identities.
3. I DENTITY -D ISENTANGLEMENT IN C ONTRASTIVE L EARNING
The goal of sample-identiﬁcation can be formulated as maximizing q ( y = i | x
= x i ) , i.e., the likelihood estimation of identifying x i or its augmentations as sample-
i , where x and y are two random variables for a sample and an identity label, respectively.
The following proposition shows that q ( y = i | x
= x i ) provides a lower bound for the mutual information I ( x ; y ) between x and y .
Proposition 1.
(Sample-identiﬁcation likelihood as a lower bound of I ( x ; y ) ) .
If x is a random minibatch of size N and the sample-identiﬁcation likelihood estimation of x i on its correct identiﬁcation label y
= i to be q ( y = i | x
= x i ) , the mutual information I ( x ; y ) can be lower bounded by
I ( x ; y ) = log N + E x (cid:34)
N
N (cid:88) i
= log p ( y = i | x = x i ) (cid:35) ≥ log N + E x
[Equation]
where p ( · ) is the true probability and q ( · ) denotes an estimation of p ( · ) .
A detailed proof is given in the
Appendix.
In contrastive learning, p ( y = i | x = x i ) can be modeled and estimated by q NCE ( i | x = x i ) deﬁned in Eq.
(1) using neural network embedding f ( · ) and h ( · ) of data augmentations A ( · ) and
B ( · ) .
Hence, InfoNCE loss can provides a lower-bound of I ( x ; y ) , i.e.,
[Equation]
The above lower bound relates the mutual information I ( x ; y ) and contrastive learning: increasing the batch size N and/or minimizing the InfoNCE loss can improve the tightness of the lower bound and results in representations with better capability on sample identiﬁcation.
Due to the natural sparsity of hard positives and negatives in the original data x , data augmentations and perturbations of x are necessary to improve the sample efﬁciency when minimizing E x [ L NCE ( x )] .
However, without any constraints, strong data augmentations may remove the identity-related information of some data and change their identities, which results in q NCE ( i | x
= x i ) (cid:28) p ( y = i | x
= x i ) , a loose lower bound in Eq.
(4), and poor representations via contrastive learning.
3. I DENTITY -D ISENTANGLEMENT VIA V ARIATIONAL A UTO -E NCODER
Next, we will show that the training objective of variational anto-encoder (VAE) (Kingma & Welling, 2013) is also related to I ( x, y ) and sample identiﬁcation.
In particular, VAE aims at removing identity-speciﬁc information from the bottleneck features z , i.e., minimizing I ( z, y ) .
With an output
G ( x ) decoded from z , VAE naturally disentangles the most identity-relevant information from x .
If we can remain such information, i.e., x − G ( x ) , intact in data augmentations and only perturb the rest part G ( x ) , the above problem of contrastive learning can be resolved.
Lemma 1.
(VAE objective and I ( z ; y ) from Eq.
(29) in Appendix B of Alemi et al.
Assume that the bottleneck features of VAE are denoted by z , the encoder is E ( · ) and produces distribution p E ( z | x ) , the decoder is D ( · ) and produces distribution q D ( x | z ) , the prior for z is p ( z ) , and
the
KL-divergence regularization in the VAE objective L VAE has a weight β , we have:
[Equation]
In L VAE , β (Higgins et al., 2016) controls how close the distribution of bottleneck features p ( z | x = x i ) is to the prior p ( z ) , e.g., a standard Gaussian distribution independent of the sample identity
y
= i .
So β controls the strength of identity disentanglement on z .
Lemma 1 shows that VAE is trained to minimize an upper bound of − I ( z, x ) + βI ( z, y ) , i.e., preserving most information of input x in the bottleneck feature z but removing from z the critical information about sample identity y
= i .
Hence, the most identity-relevant information is disentangled from the VAE output
G ( x ) ∼ p D ( x | z )
= p ( x | D ( z )) and preserved in the residual R ( x ) (cid:44) x − G ( x ) .
3. I DENTITY -D ISENTANGLED D ATA G ENERATION AND A UGMENTATION
[Figure]
We can study a data generative model based on identitydisentanglement of VAE, which generates D ( z ) and R ( x ) from z and y respectively and combine them to generate x = G ( x ) +
R ( x ) , as shown in Fig.
The following lemma compares
I ( R ( x ); y ) with I ( x ; y ) and analyzes how much identity information can be preserved in R ( x ) .
Lemma 2.
(Identity-disentangled data generation) .
For a data generative model described above,
[Equation] [Equation]
The detailed proof of Lemma 2 is given in the appendix.
Theorem 1.
(Identity-disentangled data augmentation) .
If we use a VAE in the identity-disentangled data generative model for Lemma 2, and if we deﬁne an augmentation x (cid:48)
=
R ( x ) + G (cid:48) ( x ) with
G (cid:48)
( x ) ∼ q D ( x | z (cid:48) ) and z (cid:48) =
z
+ δ (a δ -perturbed z ), we can lower bound I ( x (cid:48) ; y ) as
[Equation]
Theorem 1 can be proven by combing Lemma 1 and Lemma 2, which is illustrated in the appendix.
Therefore, the augmentation x (cid:48) preserves most of the identity information in x if L VAE
+ I
( z ; x ) is small.
VAE training aims at minimizing L VAE so the ﬁrst term can be kept small.
There is a trade-off between the second term and identity preservation I ( x (cid:48) ; y ) since a sufﬁciently large I ( z ; x ) is necessary to produce augmentation x (cid:48) approximately drawn from the true data distribution.
A key observation of the above theorem is that we can perturb z to generate x (cid:48) without hurting the lower-bound of I ( x (cid:48) ; y ) .
This implies that we can adversarially perturb z to produce hard negatives and positives for more efﬁcient contrastive learning without heavily distorting the original identity information.
In contrast, most data augmentation techniques used in self-supervised learning have not taken this into account so they may change the sample-identity and result in poor representations.
3. I DENTITY -D ISENTANGLED A DVERSARIAL A UGMENTATION (IDAA)
As discussed in Sec. 2 and Sec. 3.1, keeping a sufﬁcient amount of hard positives and negatives in each batch x is critical to the effectiveness and sample efﬁciency of contrastive learning.
Although any perturbation on z can produce an identity-preserved data augmentation x (cid:48) according to Theorem 1, adversarially perturbing the VAE-bottleneck feature z of the original samples can produce hard positives/negatives with sample-identities preserved.
Meanwhile, we need to make sure that x (cid:48) is still close to the original data manifold, e.g., an natural image, so x (cid:48) needs to reside in the vicinity of x .
Fortunately, we can use off-the-shelf adversarial attack algorithms for this purpose.
The main difference here is: (1) we apply them to perturb the bottleneck features z instead of x ; and (2) the objective is to maximize the InfoNCE loss instead of a classiﬁcation loss such as cross entropy.
We call this method “identity-disentangled adversarial augmentation (IDAA)”, as illustrated in Fig.
In particular, the goal of IDAA is to generate an augmentation of x in the form of x (cid:48) in Theoremsuch that the InfoNCE loss is maximized using an (cid:15) -bounded perturbation δ to z
= E ( x ) when generating x (cid:48) .
For a batch of data x , this problem can be formulated as optimizing { δ i } Ni =1 by
[Equation]
To only perturb the positives and negatives to be harder for contrastive learning while keeping the anchors intact, we slightly modify L NCE ( x (cid:48) ) to be
[Equation]
[Figure]
Since the augmentation x (cid:48) can be a positive for its own identiﬁcation (i.e., when x is the anchor) and a negative for other samples’ identiﬁcation, the objective in Eq.
(9) modiﬁes a sample x to be both a hard positive and a hard negative.
In other words, IDAA perturbs the bottleneck features to move x (cid:48) away from x to other difference samples in the embedding space.
Besides InfoNCE loss, IDAA can be applied to other self-supervised losses, e.g., replacing L NCE ( · ) in Eq.
(9) with the consistency regularization L CS ( · ) in Eq (2).
IDAA is complementary to and can be applied to existing data augmentation techniques by replacing x i in Eq.
(9) with a pre-deﬁned data augmentation A ( x i ) .
It can also be applied together with other data augmentations, e.g., by combining multiple InfoNCE loss terms computed on different augmentations.
A primary advantage of IDAA is that the generated augmentations are adaptive to the training models f ( · ) and h ( · ) .
It aims at ﬁnding the weaknesses of the self-supervised models on the sample-identiﬁcation task and improve them without heavily distorting the original sample-identities, which is underexplored in previous literature.
There exists various off-the-shelf adversarial attack methods that can be directly applied to solve the problem in Eq.
In this paper, we adopt fast gradient sign method (FGSM)
(Goodfellow et al., 2014) for its computational efﬁciency.
FGSM perturbs z = E ( x ) for one step by adding noises along the gradient sign’s direction of the loss w.r.t.
δ and the augmentation x (cid:48) is generated based on the perturbed bottleneck features z + δ , i.e.,
[Equation]
E XPERIMENTS
In this section, we evaluate the training efﬁciency and the test performance of several state-of-the-art methods in (1) self-supervised learning and (2) semi-supervised learning when using (1) their default random augmentations; (2) default augmentations + CLAE (Ho & Vasconcelos, 2020); and (3) default augmentations + IDAA.
IDAA consistently improves both the training efﬁciency and the test accuracy of all the methods and outperforms the two baselines by large margins across several standard benchmark datasets.
We also observe that IDAA trained on one dataset can be transferred to other datasets and improve downstream tasks on them, implying promising generalization capability of the learnable augmentation model.
Moreover, we conduct a thorough sensitivity study of IDAA when changing (1) batch sizes; (2) network architectures; (3) training epochs; (4) regularization weight β in the VAE objective; (5) dimensions of the VAE bottleneck features; and (6) adversarial attack strength bounded by (cid:15) .
4. C ASE S TUDY OF A UGMENTATIONS GENERATED BY IDAA AND CLAE
We ﬁrstly present a case study of augmentations generated by IDAA, CLAE, and widely used random augmentations.
We compare their identity preserving, hardness to contrastive learning, and the patterns of perturbations created by them on natural images.
For random augmentations, we choose
RandomFlip, ColorJitter and GreyScale for their popularity in recent SSL methods (Ye et al., 2019;
Chen et al., 2020; Chen & He, 2021).
In Fig.
1 (b)-(c), we compare the distance of an anchor to a positive with its distance to the nearest negative in the embedding space we apply the contrastive learning.
If the latter is smaller than the former, i.e., the point is located below the “identity-preserving boundary” in the plots, its original identity is changed and cannot be preserved in the augmentation.
On the other hand, if the latter is much larger than the former, i.e., the point is located on the left region of the plots, the sample identiﬁcation task is too trivial and the contrastive learning is not efﬁcient.
From the plots, we can see that random augmentations are too easy while the CLAE augmentations are much harder but cannot preserve the original identity for some samples.
In contrast, IDAA (ours) produces sufﬁciently hard augmentations within the boundary of identity changing, which is ideal for contrastive learning.
We visualize the augmentations produced by the three methods in Fig.
4 for natural images from
ImageNet dataset.
Both IDAA and CLAE are applied to random augmentations A ( x ) and introduce further perturbations A (cid:48)
( x ) − A ( x ) .
IDAA generates more semantic perturbations to important regions of the images because its adversarial attack is conducted in the VAE bottleneck space, which is supposed to capture semantic attributes of the images.
On the contrary, CLAE generates the perturbations on the input pixels and produces unnatural artifacts, which may distort the original sample-identity in the embedding space.
In addition, we show the identity-relevant part R ( A ( x )) and the identity-disentangled part G ( A ( x )) produced by VAE.
The former well preserves the most important patterns to identity the original sample, e.g., the edge of the steamship (column 2), spots of the frog (column 5) and pattern of the leopard (column 7).
Since IDAA keep it intact in its augmentation model, the identity is well preserved in the augmentations.
4. S ELF -S UPERVISED L EARNING
We evaluate data augmentations on four self-supervised learning methods: Plain (InfoNCE in
Eq.
(1), UEL (Ye et al., 2019), SimSiam (Chen & He, 2021) and SimCLR (Chen et al., 2020) on three datasets, i.e., CIFAR10 (Krizhevsky et al., 2009), CIFAR100 (Krizhevsky et al., 2009), and miniImagenet (Vinyals et al., 2016).
The experimental details are given in Sec.
A.3 in the Appendix.
[Figure]
Performance on Downstream Classiﬁcation Tasks We ﬁrstly evaluate the representations of different SSL methods using different data augmentations on downstream classiﬁcation tasks.
In Table 1, we report the test accuracy of classiﬁers produced by two commonly used evaluation protocols, i.e., k-nearest neighbor (kNN) with k = 200 and linear evaluation that trains a linear classiﬁer on top of the learned representations.
IDAA consistently improves the original SSL methods with their default random augmentations and the improvement is usually much larger than that brought by CLAE.
In contrast, though
CLAE achieves better or comparable accuracy than the default augmentations on most contrastive learning methods, it results in signiﬁcant degradation when applied to SimSiam, indicating that consistency regularization without comparison to negatives is more prone to potential identity distortion caused by adversarial augmentations in CLAE, which may strengthen the collapse of the training objective.
Instead, IDAA exhibits better generalization to different types of SSL methods and evaluation protocols.
Transfer Learning Performance Different from random augmentations deﬁned by human priors, IDAA and CLAE depend on models trained on a speciﬁc dataset.
Hence, it is crucial to evaluate whether they can be transferred to other unseen datasets to improve downstream tasks.
Following the linear evaluation protocol of Chen et al.
, we train encoders by IDAA and
CLAE on ImageNet100 (i.e., 1oo classes subset of Imagenet) and then train linear classiﬁers ondatasets (Krizhevsky et al., 2009; Berg et al., 2014; Maji et al., 2013; Cimpoi et al., 2014;
Parkhi et al., 2012; Nilsback & Zisserman, 2006; Welinder et al., 2010) using the encoders’ output representations.
The test accuracy of these classiﬁers are reported in Table 2.
On Imagenet100, “SimCLR+IDAA”
[Table] [Table]
achieves 61.9% kNN accuracy, which outperforms 56.38% of SimCLR by a large margin.
This improvement can be transferred to the 8 datasets.
As shown in the Table, IDAA consistently outperforms
CLAE and improves the original SimCLR, while CLAE brings degeneration on one dataset,
i
[Table]
e., Pets.
We also compare their transfer learning results when applied to SimSiam in Sec.
A.4 of the Appendix.
4. S EMI -S UPERVISED L EARNING
[Table]
Data augmentation is also critical to state-of-theart semi-supervised learning algorithms such as
FixMatch (Sohn et al., 2020), which relies on accurate pseudo labeling and conﬁdence-based data selection and their quality heavily depends on data augmentations.
We apply IDAA to
FixMatch to train a a WideResNet-28-8 model on CIFAR100 with only { 400 , 2500 , 10000 } labeled samples and the rest unlabeled.
We train it for 2 × 10 5 steps with batch size of 512.
The test accuracy of the trained models are reported in
Table 3, where IDAA consistently improves FixMatch’s accuracy and the improvement is more signiﬁcant with fewer labeled data available.
Moreover, as shown in Fig. 5, IDAA signiﬁcantly improves the efﬁciency and convergence of FixMatch and can save a great amount of computation to reach a reasonable accuracy.
IDAA keeps outperforming CLAE given different amounts of labeled data.
4. S ENSITIVITY A NALYSIS OF H YPERPARAMETERS
Batch size We evaluate how SimCLR and SimCLR+IDAA perform using four different batch sizes and the results are shown in Fig.
6(a).
As veriﬁed by previous works (Chen et al., 2020; He et al., 2020), increasing the batch size can improve SimCLR and other contrastive learning methods’ performance, which is also reﬂected in our results.
However, IDAA can signiﬁcantly improve
SimCLR’s performance under small batch size, e.g., 64, because it effectively modiﬁes various samples to be hard positives/negatives without distorting their original identities.
Due to the same reason, SimCLR+IDAA is less sensitive to the change of batch size.
It is also worth noting that
SimCLR+IDAA using a very small batch size of 64 can surpass SimCLR with its best batch size of 512.
This demonstrates the advantage of IDAA on improving the data/memory efﬁciency: with
IDAA, contrastive learning is no longer limited to the usage of a large batch size.
[Figure] [Figure]
Model Architecture We evaluate the linear evaluation performance of SSL when training different
ResNet architectures in Fig. 6(b).
Increasing the model size improves the performance of both methods but SimCLR+IDAA always outperforms SimCLR by a large margin.
Hence, IDAA can improve SSL of different models and it can train a smaller model with less costs to match the performance of training a larger model by random augmentations.
Training epochs As shown in Fig. 6(c), the performance of both methods improves when investing more training epochs but the SimCLR+IDAA saturates much earlier and only spends 300 epochs to achieve comparable performance as SimCLR trained using 1000 epochs.
Hence, IDAA can greatly improve the training efﬁciency of SSL.
Regularization weight β in VAE As revealed by Theorem 1, β in the VAE objective controls the lower bound of the identity information preserved in IDAA augmentations: larger β enforces more identity preservation in x (cid:48) and stronger identity-disentanglement on z .
However, it also controls
I ( z ; x ) which reﬂects the proximity of z ’s distribution to the true data manifold of x : large β leads to small I ( z ; x ) and less information of x preserved in z that can be leveraged to produce stronger adversarial attacks for hard positives/negatives.
In Fig. 7(a), we observe that the trade-off reaches a sweet spot at β = 0 .
1 among all the four β values between 0 and 10 .
VAE bottleneck dimension As shown in Fig. 7(b), SSL performance with IDAA is not sensitive to the change of bottleneck dimension of VAE, though 512-dimension performs slightly better than other choices in the experiment.
Attack strength (cid:15) Stronger attacks may produce more hard positives/negatives but also increases the risk of identity distortion and unrealistic augmentations biased from the true data distribution/manifold.
Results in Fig.
7(c) shows this trade-off and its effects on the SSL performance.
Nevertheless, the performance of SimCLR+IDAA is still quite stable and only varies in a small range when changing (cid:15) since the identity-relevant information in R ( x ) stays intact.
C ONCLUSION
We propose a simple automatic data-augmentation method IDAA, which can generate more informative but identity-preserved augmentations to improve the efﬁciency and generalization of selfsupervised learning.
Motivated by an information theoretical analysis of VAE and identity preserving,
IDAA adds adversarial noise to an identity-disentangled space learned by VAE and combines the perturbed VAE outputs with an intact identity-relevant part to produce augmentations.
IDAA merely relies on a pretrained VAE without requiring any labeled data but consistently improves a diverse set of popular self-supervised/semi-supervised learning methods across multiple benchmarks.
It also enhances the transfer learning performance and signiﬁcantly improves the learning efﬁciency.
A A PPENDIX A. P ROOF FOR P ROPOSITION
Proposition 1.
(Sample-identiﬁcation likelihood as a lower bound of I ( x ; y ) ) .
If x is a random mini-batch of size N and the sample-identiﬁcation likelihood of x i on its correct identiﬁcation label y
= i to be p ( y = i
| x = x i ) , the mutual information I ( x ; y ) can be lower bounded by
I ( x ; y ) ≥ log N + E x (cid:34)
N
N (cid:88) i
= log p ( y = i | x = x i ) (cid:35) .
Proof.
To relate mutual information and sample-identiﬁcation task, we take the expectation by ﬁrst sample a mini-batch x = { x 1 , x 2 , · · · , x N } of size N and then sample pairs ( x, y ) from the mini-batch.
Then we have:
I ( x, y ) =
H ( y ) − H
( y | x )
= log N + E x (cid:34) N (cid:88)
y = (cid:90) dx p ( y, x ) log p ( y | x ) (cid:35) =
log N + E x (cid:34) N (cid:88)
y = (cid:90) dx p ( x ) p
( y | x ) log p ( y | x ) (cid:35) ≥ log N + E x (cid:34) N (cid:88)
y = (cid:90) dx p ( x ) p
( y | x ) log q ( y | x ) (cid:35) =
log N + E x (cid:34) N (cid:88)
y = (cid:90) dx p ( y, x ) log q ( y | x ) (cid:35) =
log N + E x (cid:34) N (cid:88)
y = (cid:90) dx p ( x
[Equation]
where p ( · ) denotes the true probability and q ( · ) denotes arbitrary estimation.
The inequality in
Equation (13) comes from D KL ( p ( y | x ) , q ( y | x )) ≥ 0 .
A. P ROOF FOR T HEOREM
We start by proving Lemma 2 and then we can prove Theorem 1 by combing Lemma 1 and Lemma 2.
Lemma 2.
(Identity-disentangled data generation) .
For a data generative model described above,
[Equation]
Proof.
Due to the Markov chain y
→ ( R ( x ) , z ) → x in the data generative model in Fig.
3, we have
[Equation]
By the deﬁnition of conditional mutual information, we have
[Equation] [Equation]
Since z ⊥⊥ R ( x ) in the generative model, the last two terms in the above equation are zeros and
I ( z ; y | R ( x )) =
I ( z ; y ) .
Substituting it to Eq.
(15) completes the proof.
[Table]
Theorem 1.
(Identity-disentangled data augmentation) .
If we use a VAE in the identity-disentangled data generative model for Lemma 2, and if we deﬁne an augmentation x (cid:48)
=
R ( x ) + G (cid:48) ( x ) with
G (cid:48)
( x ) ∼ q D ( x | z (cid:48) ) and z (cid:48) =
z
+ δ (a δ -perturbed z ), we can lower bound I ( x (cid:48) ; y ) as
[Equation]
Proof.
Applying the results from Lemma 2 and Lemma 1, we have
[Equation]
The ﬁrst inequality is due to the identiﬁability assumption extend from Theorem 1 in Robust PCA Candès et al.
(2011), i.e., the identity-disentangled part G ( x ) and the identity-revelant part R ( x ) are separable give x .
Under this assumption, we have the conditional entropy H ( R ( x )
| R ( x )+ G (cid:48)
( x ))
=
because R ( x ) + G (cid:48) ( x ) can be separated into R ( x ) and G (cid:48)
( x ) again in an unique and exact way using
VAE.
This is a natural extension because VAE is an extension of robust PCA, as pointed out by Dai et al.
A. E XPERIMENTAL DETAILS OF S ELF - SUPERVISED LEARNING
We evaluate data augmentations on four self-supervised learning methods: Plain (InfoNCE in
Eq.
(1), UEL (Ye et al., 2019), SimSiam (Chen & He, 2021) and SimCLR (Chen et al., 2020) on three datasets, i.e., CIFAR10 (Krizhevsky et al., 2009), CIFAR100 (Krizhevsky et al., 2009), and miniImagenet (Vinyals et al., 2016).
The four SSL methods cover different contrastive learning methods (Plain, UEL, SimCLR) and a consistency regularization based method (SimSiam).
The training/test splitting of miniImagenet follows Ebrahimi et al.
Unless otherwise noted, a
ResNet-18 with two separate BNs (Xie et al., 2020) for clean and adversarial samples is utilized as the representation encoder.
The regularization weight β in VAE is set to 0.1 and the bottleneck dimension is 512(3072) for CIFAR10/CIFAR100(miniImagenet).
A. T RANSFER LEARNING PERFORMANCE USING S IM S IAM AS BASELINE
We further add new experiments of SimSiam, SimSiam+CLAE, and SimSiam+IDAA, and report their results in Table 4 as an extension of Table 2.
Unlike SimCLR, SimSiam Chen & He (2021) adopts another popular idea of self-supervised learning based on the consistency regularization and
Siamese network.
We evaluate the three methods on ImageNet-100 as we did for SimCLR, i.e., by running each method for 100 epochs and evaluating their transfer learning performance on the 8 datasets as in Table 2.
Similar to SimCLR, IDAA consistenly improves the transfer learning performance of SimSiam and outperform CLAE on all the datasets.
We select SimCLR and SimSiam for this study because they represent the two most widely studied self-supervised learning strategies, i.e., contrastive learning and consistency learning respectively.
A. N EW B ASELINE : D ATA A UGMENTATION WITHOUT D ECOMPOSITION ( G ( z (cid:48) ) ONLY )
Here we compare with an additional baseline: simply attacking the z of a standard VAE (with beta=1) to produce an x (cid:48)
=
G ( z (cid:48) ) as augmentations for contrastive learning.
We report its results on
CIFAR in Table 5, denoted as “SimCLR+IDAA(w/o decomposition)”.
It shows that the performance signiﬁcantly declines once the decomposition removed and it performs even worse than the original
SimCLR.
Therefore, identity-disentanglement is critical and preserving the identity-relevant part R ( x ) in augmentations is essential to self-supervised learning.
This is illuestrated in Figure 1.
On the contrary, if we simply use the identity-distorted part G ( z (cid:48) ) for augmentation with R ( x ) removed, the positive and negative assignments in contrastive learning can be wrong.
With many wrong identity labels, the identiﬁcation task of contrastive learning can easily fail, resulting in poor representations and performance degradation on downstream tasks.
[Table]
A. E XPERIMENTS ON L OWER R ESOLUTION VERSION (64 × 64) OF I MAGE N ET -1 K
We add new experiments that train a ResNet-18 on ImageNet 64x64 by SimCLR, SimCLR+CLAE and SimCLR+IDAA.
We run each method for 100 epochs with a batch size of 1024.
We report the results in Table 6.
It shows that SimCLR+IDAA still outperforms SimCLR+CLAE and original
SimCLR on this more challenging dataset.
Moreover, IDAA can improve both the kNN and linear evaluation accuracy while CLAE can only improve the kNN accuracy.
[Table]
A. C OMPARISON WITH MORE B ASELINES
In this section, we compare IDAA with two additional baselines published recently, i.e., InfoMin (Tian et al., 2020b) and Debiased (Chuang et al., 2020).
InfoMin (Tian et al., 2020b) studies how to obtain good augmentation for contrastive learning, here we compare IDAA with InfoMin (Tian et al., 2020b) using their ofﬁcial code 1 on ImageNet they used in their paper (we reduced the resolution to 64 ×due to our limited computational resources).
We apply each method to train ResNet-18 forepochs and compare their linear evaluation accuracy on the test set.
The results are reported in the
Table 7, which shows that IDAA outperforms InfoMin (Tian et al., 2020b) by a large margin.
Debiased (Chuang et al., 2020)
proposed a new objective, which aims to avoid sampling negatives of the same class as the anchor in contrastive learning.
Our method as an augmentation strategy is complementary to Debiased .
We add new experiments comparing with Debiased using their ofﬁcial [Table] code 2 on CIFAR10, the dataset used in their original paper.
Speciﬁcally, we apply Debiased and “Debiased+IDAA” to train ResNet-18 for 300 epochs and compare their kNN an linear evaluation accuracy.
The results are reported in Table 8, which shows that Debiased outperforms SimCLR but
IDAA can further improve Debiased when applied on Debiased.
Moreover, only “SimCLR+IDAA” can outperform Debiased by a large margin, i.e., 1.6% linear evaluation accuracy, which clearly demonstrate the effectiveness of IDAA.
[Table]