S
ELF -C ONTRASTIVE L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper proposes a novel supervised contrastive learning framework, called
Self-Contrastive (SelfCon) Learning , that self-contrasts within multiple outputs from the different levels of a multi-exit network.
SelfCon learning does not require additional augmented samples, which resolves the concerns of multi-viewed batch (e g., high computational cost and generalization error).
Furthermore, we prove that SelfCon loss guarantees the lower bound of label-conditional mutual information between the intermediate and the last feature.
In our experiments including
ImageNet-100, SelfCon surpasses cross-entropy and Supervised Contrastive (SupCon) learning without the need for a multi-viewed batch.
We demonstrate that the success of SelfCon learning is related to the regularization effect associated with the single-view and sub-network
I NTRODUCTION
Recent studies have studied the success of deep neural networks by investigating how neural networks can encode representations with rich information (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Hjelm et al., 2018; Saxe et al., 2019a).
Among the various approaches suggested, contrastive loss functions, designed to maximize the lower bound of mutual information (MI) between the target and context, have achieved considerable success in self-supervised representation learning ﬁrst (Gutmann & Hyvärinen, 2010; Oord et al., 2018) and supervised learning recently (Khosla et al., 2020; Gunel et al., 2020; Wang et al., 2021).
The main objective of the contrastive loss functions in supervision is to make representations from the same class closer and representations from different classes farther.
To this end, they deﬁne positive samples, i.e., augmented samples from the same image or (augmented) images sharing the same class label, and negative samples, i.e., all other samples, for every data batch.
Contrasting two random augmented samples, often referred to as a multi-viewed batch, has shown impressive results in representation learning (Chen et al., 2020;
He et al., 2020; Grill et al., 2020;
Caron et al., 2020; Chen & He, 2020; Khosla et al., 2020), yet a multi-viewed batch causes the following issues.
(1) A multi-viewed batch doubles the batch size, which is a huge burden on memory and computation.
(2) Oracle needs to carefully choose the augmentation policies (Chen et al., 2020;
Tian et al., 2020; Caron et al., 2020; Kim et al., 2020).
(3) A multi-viewed contrastive task is domain-speciﬁc, because data-level augmentation requires speciﬁc domain knowledge such as image cropping and ﬂipping (Verma et al., 2021).
In spite of these concerns, a supervised contrastive learning framework (Khosla et al., 2020) still relies on multi-views, although they are not necessarily needed with the help of usable label information.
In this work, we propose a novel contrastive learning framework in supervision, called SelfContrastive (SelfCon) learning , that does not require additional augmented samples.
Instead,
SelfCon uses a multi-exit framework (Teerapittayanon et al., 2016; Zhang et al., 2019a;b) where sub-networks produce multiple outputs for the same input and self-contrasts within multiple outputs from the different levels of a single network.
For training, SelfCon learning can use the multi-viewed batch as well.
However, the multi-exit framework already generates positive pairs from a single image, which can replace the augmentation based multi-view.
In Figure 1, we compare
SelfCon learning and Supervised Contrastive (SupCon, Khosla et al.
(2020)) learning.
SelfCon learning improves the classiﬁcation performance of the encoder network, owing to (1) the increase of the lower bound of label-conditional MI between the intermediate and the last features, and (2) the regularization effect from the single-view and sub-network.
We theoretically demonstrate that SelfCon loss is the lower bound of label-conditional MI.
Furthermore, unlike
[Figure]
SupCon loss, it encourages the encoder to learn the label information that only intermediate features can explain.
Second, we empirically show that SelfCon learning reduces the generalization error with the single-view and sub-network.
The former prevents the encoder from overﬁtting to each instance, and the latter regularizes the intermediate feature to be similar to the last feature.
The contributions of our paper can be summarized as follows:
[S3]
We propose Self-Contrastive learning, a novel supervised contrastive framework between the multiple features from the different levels of a single network.
[S4] SelfCon loss guarantees the lower bound of label-conditional MI between the intermediate and the last features, and leads to increasing interaction information between the features and the label.
[S5.1, S5.2] SelfCon learning achieved higher classiﬁcation accuracy for various benchmarks and architectures than cross-entropy and SupCon loss.
Our empirical study of MI estimation provides evidence for superior performance.
[S5.3, S5.4] We investigate the advantages of single-viewed batch in terms of the generalization error and computation resources.
We also identify that SelfCon learning beneﬁts from the sub-network owing to the regularization effect, vanishing gradient, and ensemble prediction.
R ELATED W ORKS 2. C ONTRASTIVE L EARNING
As the cost of data labeling increases exponentially, there is an increased need for acquiring representation under unsupervised scenarios.
After Oord et al.
(2018) proposed an InfoNCE loss (also called a contrastive loss), contrastive learning-based algorithms showed a remarkable improvement in performance (Chen et al., 2020;
He et al., 2020; Grill et al., 2020; Caron et al., 2020; Chen &
He, 2020).
Most previous works use the structure of a Siamese network, which is a weight-sharing network applied on two or more inputs, with negative pairs (SimCLR (Chen et al., 2020), MoCo (He et al., 2020)), momentum encoders (MoCo (He et al., 2020), BYOL (Grill et al., 2020)), online clustering (SwAV (Caron et al., 2020)), or a stop-gradient operation (SimSiam (Chen & He, 2020)).
While the softmax form is frequently used for the contrastive loss (Chen et al., 2020; He et al., 2020), recent state-of-the-art algorithms utilize an MSE loss (Grill et al., 2020; Chen & He, 2020), or a cross-entropy loss (Caron et al., 2020; 2021).
Khosla et al.
(2020), inspired by the success in self-supervised learning, propose a label-based contrastive loss in supervision, named SupCon loss.
Supervised contrastive learning has also been extended to semantic segmentation (Wang et al.,
2021) and language tasks (Gunel et al., 2020).
While the SupCon loss utilizes the output features from two randomly augmented images, we also contrast the features from different network paths by introducing the multi-exit framework (Teerapittayanon et al., 2016; Zhang et al., 2019a;b).
We further propose a novel loss function that we can apply on the single-viewed batch.
2. M UTUAL I NFORMATION
Mutual information is a measure to quantify the information held in a random variable about the other variable, and it has been used as a powerful tool to open the black box of deep neural networks (Tishby & Zaslavsky, 2015; Shwartz-Ziv & Tishby, 2017; Saxe et al., 2019a).
As it is difﬁcult to compute the
MI exactly (Paninski, 2003), several works have proposed variational MI estimators based on neural networks, e.g., InfoNCE (Oord et al., 2018), MINE (Belghazi et al., 2018), NWJ (Nguyen et al., 2010),
ML-CPC (Song & Ermon, 2020), and SMILE (Song & Ermon, 2019).
Recently, MI estimator-based objectives have been proposed to improve the performance of contrastive learning (Hjelm et al., 2018;
Bachman et al., 2019; Song & Ermon, 2020; Wu et al., 2020b) and knowledge distillation (Tian et al., 2019b; Ahn et al., 2019).
Among previous approaches, we are highly motivated by those that aim to increase the MI between the intermediate and the last features.
DIM (Hjelm et al., 2018) and AMDIM (Bachman et al., 2019) propose novel contrastive losses between the global features and local features (i.e., all pixels or patches of the intermediate features), while we contrast the reﬁned local features via sub-networks.
VID (Ahn et al., 2019) makes the student learn the distribution of the activations in the auxiliary teacher’s intermediate features.
Our method differs from VID in that we self-distill within the network itself.
Zhang et al.
(2019a;b) share a similar idea with the aforementioned works, but they implicitly increase MI using Kullback–Leibler (KL) divergence loss.
S ELF -C ONTRASTIVE L EARNING
We propose a new supervised contrastive loss that maximizes the similarity of the outputs from different network paths by introducing the multi-exit framework.
We deﬁne the encoder structure with F as a backbone network and G as a sub-network that shares the backbone’s parameters up to some intermediate layer.
T denotes those sharing layers that produce the intermediate feature.
The sub-network has additional non-sharing parameters attached after T .
Note that F and G include the projection head on the encoder.
We highlight the positive and negative pairs with respect to an anchor sample, following Figure 1.
SupCon loss To mitigate the weaknesses of cross-entropy, such as the reduced generalization performance and the possibility of poor margins, Khosla et al.
(2020) proposed a supervised version of the contrastive loss that deﬁnes the positive pairs as every sample with the same ground-truth label.
We reformulate the SupCon loss function as follows:
[Equation]
where • denotes the inner product, B is the batch size, and τ is the temperature to soften or harden the softmax value.
I denotes a set of indices of the multi-viewed batch that concatenates the original
B images and the augmented ones, i.e., x B
+ i is an augmented pair of x i .
P i ∗ and N i are a set of positive and negative pair indices with respect to an anchor i , respectively.
Note that Eq.
1 is the same as NT-Xent loss (Chen et al., 2020) when P i ∗ ≡ { ( i + B ) mod 2 B } .
We dropped the dividing term of sums for brevity.
SelfCon loss Besides minimizing the above loss, we aim to maximize the similarity between the outputs from the backbone and the sub-network.
To this end, we deﬁne SelfCon loss, which forms a self-contrastive task for every output including the features from the sub-network.
In Section 4, we clarify the connection of this loss with the label-conditional MI between the intermediate and the last features.
[Table]
[Table] [Table]
. , 2 B }
[Table]
M
[Table]
p
[Table]
∈ P
[Table]
≡ { p ∈
[Table]
y
[Table]
= y
[Table]
n ∈ N
[Table]
≡ { n ∈
[Table]
y n (cid:54) =
y
[Table]
} ω , ω 1 , ω 2 ∈ Ω =
{ F , G } , a function set of the backbone network and the sub-network.
We exclude an anchor sample from the positive set to avoid contrasting the same feature, i.e., P i ∗ ← P i ∗ \ { i } when ω = ω ∗ .
While prevalent contrastive approaches (Khosla et al., 2020; Chen et al., 2020;
He et al., 2020; Grill et al., 2020; Caron et al., 2020) force a multi-viewed batch generated by data augmentation, in SelfCon learning, the sub-network plays a role as the augmentation and provides an alternative view on the feature space.
Therefore, we formulate our SelfCon loss function with s ingleviewed batch ( SelfCon-S ); I ≡ { 1 , . . .
, B } without the additional augmented samples.
Besides, we also deﬁne SelfCon with m ulti-viewed batch ( SelfCon-M ) loss, i.e., L self with I ≡ { 1 , . . .
, 2 B } .
In the development of our SelfCon loss formulation, we can further use multiple sub-networks, i.e., Ω = { F , G 1 , G 2 , . . . } .
Appendix C provides the classiﬁcation performance of the expanded network, which was comparable to that of the single sub-network in our experiments.
Thus, we simply use a single sub-network throughout our paper.
D ISCUSSIONS
Question 1: How does SelfCon loss maximize the lower bound of label-conditional MI between the intermediate and the last features?
We explain in the following propositions to answer Question 1.
All the proofs can be found in
Appendix A.
Proposition 4.1.
Let x and z be different samples that share the same class label c .
Then, with some discriminator function modeled by a neural network F and 2( K − 1) negative sample size, SupCon loss maximizes the lower bound of conditional MI between the output features of a positive pair,
[Equation]
Proposition 4.2.
Denote L self - s as SelfCon loss with single-viewed batch.
SelfCon-S loss maximizes the lower bound of MI between the output features from the backbone and the sub-network,
[Equation]
Proposition 4.3.
Let T ( x ) be the intermediate feature of the backbone network, which is also an input to the auxiliary network path.
Then the r.h.s.
of Eq.
4 is upper-bounded by
[Equation]
Proposition 4.1 and 4.2 can be derived from the exact bound of InfoNCE (Poole et al., 2019).
SupCon and SelfCon-S loss have 2( K − 1) negative sample size because of the augmented negative pairs for
SupCon and the sub-network features for SelfCon-S. Note that SelfCon-M loss has a similar upper bound as Eq.
3 and Eq. 4 (refer to Appendix A.2).
In Proposition 4.3, ( (cid:4) ) is an interaction information (Yeung, 1991) that measures the inﬂuence of T ( x ) on the amount of shared information between F ( x ) and c .
SelfCon-S loss is related to increasing this interaction information, so that the intermediate feature enhances the correlation between the last feature and the label, which may result in improving the downstream classiﬁcation tasks.
In addition, ( (cid:3) ) implies that SelfCon loss increases the MI between the intermediate and the last features in the backbone.
T ( x ) is distilled from F ( x ) that has richer class-related information and consequently, earlier layers are trained to produce better representation (Zhang et al., 2019a;
Ahn et al., 2019).
We believe this is theoretical evidence for the improved performance of SelfCon.
Also, refer to Appendix A.4 for the connection to the classiﬁcation performance of the SelfCon loss.
[Table]
Question 2: Is it applicable to unsupervised representation learning?
Good representations should get rid of redundant input information and extract meaningful features (Tishby & Zaslavsky, 2015).
However, in Eq. 2, the backbone network can be an anchored function where the contrastive features are those from the backbone as well as the sub-network.
Unfortunately, this might allow the last feature to follow the intermediate feature, learning redundant information about the input.
This could be why SelfCon learning might not work in an unsupervised environment where there is no label information (refer to Appendix D.1).
However, in Appendix D.2, we suggest that blocking the backbone from following the sub-network, i.e., removing the term in Eq.
2 where ω = F , ω ∗ =
G , can mitigate the problem.
Supervised contrastive framework, which is the main interest of our paper, is away from the above problem owing to the label information.
Note that the unsupervised version of SelfCon-S loss guarantees only the lower bound of ( (cid:3) ) in Proposition 4.3.
Meanwhile, ( (cid:4) ) , induced by the condition on label, encourages the backbone network to be in accordance with label information.
Thus, we believe that jointly maximizing the lower bound of ( (cid:3) + (cid:4) ) offers an evidence for the success of supervised SelfCon learning.
E XPERIMENT
We presented the image classiﬁcation accuracy for standard benchmarks, such as CIFAR-10, CIFAR100 (Krizhevsky et al., 2009), Tiny-ImageNet (Le & Yang, 2015), ImageNet-100 (Tian et al., 2019a), and ImageNet (Deng et al., 2009), and extensively analyzed the results.
We summarized the mean and standard deviation of top-1 accuracy over three random seeds for the reliability of the experimental results.
We used the optimal structure and position of the sub-network for all the network architectures (see
Appendix C).
We compared three types of the sub-network: the same structure with non-sharing layers of the backbone network ( same ), a structure with the number of non-sharing layers reduced in half ( small ), and a simple fully-connected layer ( fc ).
The overall performance was comparable or better than the baselines.
We observed a trend: in shallow networks the same structure was better, while fc was better in deeper networks.
Besides, the performance was consistently good when the exit path is attached after the midpoint of the encoder (e.g., 2 nd block in ResNet or 3 rd block in VGG architecture).
Complete implementation details are in Appendix B.
5. R EPRESENTATION L EARNING
We measured the classiﬁcation accuracy on the representation learning protocol (Chen et al., 2020), which consists of 2-stage training , (1) pretraining an encoder network and (2) ﬁne-tuning on a linear classiﬁer with the frozen encoder (called a linear evaluation).
[Table]
The classiﬁcation accuracy is summarized in Table 1.
Interestingly, the loss functions in the singleviewed batch, such as SupCon-S and SelfCon-S, outperform their multi-view counterparts in all settings.
Furthermore, our SelfCon learning, which trains with the sub-network, shows higher classiﬁcation accuracy than CE and SupCon.
The effects of the sub-network are analyzed in Section 5.4.
We also experimented on the ImageNet-100 benchmark of which 100 classes were randomly sampled (Tian et al., 2019a), to verify that SelfCon learning has the same effect on large scale datasets.
Table 2 presents the consistent performance improvement of SelfCon learning.
Refer to Appendix G for the ImageNet results.
5. M UTUAL I NFORMATION E STIMATION
Minimizing the SelfCon loss is highly related to maximizing the MI between the intermediate and the last features (Proposition 4.3).
To empirically conﬁrm this claim, we estimated MI using various estimators: InfoNCE (Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010).
Speciﬁcally, we extracted the features of the CIFAR-100 dataset from the pretrained ResNetencoders and computed the estimators.
Then we optimized a simple 3-layer Conv-ReLU network with the MI estimator objectives.
We consider the encoder output without the projection head, which differs from Section 3.
In Figure 2, SelfCon-M and SelfCon-S both show high I ( F ( x ); T ( x )) , which supports our claim, while the values of CE and SupCon are lower.
Recall that T ( x ) is the intermediate feature of the backbone network, which is an input to the auxiliary network path.
Although I ( F ( x ); T ( x )) does not explicitly guarantee an increase in accuracy, our results imply that it has a positive correlation with the test accuracy of the backbone network.
We believe that the richer information in earlier features makes the encoder output better representation because the intermediate feature is also the input for the subsequent layers.
Furthermore, we found that SelfCon learning increases the information between T ( x ) and the label, implying that the intermediate feature is imbued with class-related knowledge, while the information between T ( x ) and the input x is decreased (refer to Table 12 in Appendix E).
This suggests that SelfCon learning is in agreement with IB principle of training a deep neural network, i.e., ﬁtting-compression phase (Saxe et al., 2019a).
5. M ULTI - VIEW VS . S INGLE - VIEW
We compare the multi-view and single-view in terms of generalization error, efﬁciency, and batch size.
Single-view reduces generalization error.
In Figure 3, SupCon shows higher train accuracy but lower test accuracy than SupCon-S, and the same trend is observed with SelfCon-M and SelfCon-S.
Compared to single-view, multi-view from the augmented image makes the encoder amplify the memorization of data and result in overﬁtting to each instance.
Figure 4 shows that SelfCon-S gradually enhanced the generalization ability, while SelfCon-M or SupCon achieved little gain in test accuracy despite the fast convergence.
[Figure] [Figure]
[Figure] [Figure]
[Table]
Single-view is efﬁcient in terms of memory usage and computational cost.
In Table
[Table]
we compared SupCon and SelfCon-S to observe the efﬁciency of the single-viewed batch.
SelfCon-S requires a larger number of parameters owing to the extra sub-network but is more efﬁcient in memory and computation.
In both SupCon and SelfCon-S, the same batch size implies the same number of anchor features; however, they differ in memory consumption due to the data augmentation of the multi-viewed batch.
[Table]
Multi-view is advantageous for small batch size.
In supervised learning, large batch size reduces the generalization ability, which results in decreasing the performance (You et al., 2017; Luo et al., 2018;
Wu et al., 2020a).
We examined whether the performance in a supervised contrastive framework is also dependent on the batch size.
Table 4 summarizes the results.
The multi-viewed method, e.g., SupCon, which doubles the effective number of training data, outperformed the single-viewed counterpart in 64batch experiments; the opposite was observed in all other batch sizes.
For small batch size such as 64, doubling the effective batch size can make the learning stable, but as batch size gets larger the stabilizing effect from multi-views decreases and the necessity of regularization appears to be critical.
Hence, the optimal batch size of single-viewed methods was higher (SupCon-S vs. SupCon and
SelfCon-S vs. SelfCon-M) because single-view itself helps regularization (i.e., small batch with single-view might lead to under-ﬁtting).
Also, note that SelfCon learning with sub-network still surpasses the SupCon counterpart.
5. W HAT D OES THE S UB - NETWORK A CHIEVE ?
Regularization effect SelfCon loss regularizes the sub-network to output the similar features to the backbone network.
It prevents the encoder from overﬁtting to data, and it is effective in multi-viewed as well as single-viewed batch.
In Figure 3, we conﬁrmed the regularization effect (i.e., lower train accuracy, but higher test accuracy) by comparing each bar of the same color.
The strong regularization of the sub-network helped SelfCon (-M, -S) outperform the SupCon counterparts.
This trend is also observed in Table 4 and Figure 4.
[Figure]
[Figure] [Figure]
[Table]
Vanishing gradient SelfCon learning can send more abundant information to the earlier layers through the gradients ﬂowing from the sub-networks.
Previous works (Lee et al., 2015; Teerapittayanon et al., 2016; Zhang et al., 2019a) point out that the success of the multi-exit framework is owing to solving the vanishing gradient.
We showed that the same argument applies to our SelfCon learning.
Note that the sub-network is positioned after the 2 nd block of the backbone.
In Figure 5, a larger gradient ﬂows up to the earlier layer in the SelfCon-M, while a large amount of the SupCon loss gradient vanishes.
In particular, there is a signiﬁcant difference in the gradient norm in the 2 nd block.
Ensemble with sub-network The sub-network in SelfCon learning can also be used on downstream tasks such as image classiﬁcation.
In our previous experiments, we followed the linear evaluation protocol, ﬁne-tuning a classiﬁer with the frozen backbone network.
The network pretrained on
SelfCon learning has an exit path, which is similarly allowed to be frozen and used as linear evaluation.
We thus demonstrated two additional types of linear evaluation scenarios: (1) ﬁne-tuning a classiﬁer after the sub-network output and (2) ﬁne-tuning two classiﬁers after the backbone and the sub-network and ensembling two classiﬁers’ predictions.
Table 5 indicates that the sub-network could make appropriate, although not the best, predictions as the backbone did, while ensembling their predictions is found to be the most powerful trick we have come up with.
Feature-level multi-view One of the advantages of SelfCon learning is to relax the dependency on multi-viewed batch.
This is accomplished by the multi-views on the representation space made by the parameters of the sub-network.
In Figure 6, we visualized via Grad-CAM (Selvaraju et al., 2017) the gradient of SelfCon-S loss with respect to the intermediate layer of the backbone network, right before the exit path.
Both networks focus on similar, but clearly different pixels of the same input image, implying that the sub-network learns another view in the feature space.
As multi-view in contrastive learning requires domain-speciﬁc augmentation, recent studies have explored the domain-agnostic way of augmentation (Lee et al., 2020; Verma et al., 2021).
SelfCon might be an intriguing future work in that auxiliary networks could be an efﬁcient substitute for data augmentation.
5. A BLATION S TUDY
Different encoder architectures We experimented with other architectures: VGG-16 (Simonyan & Zisserman, 2014) with Batch Normalization (BN) (Ioffe & Szegedy, 2015) and WRN-16- (Zagoruyko & Komodakis, 2016), and the results are presented in Table 6.
The classiﬁcation accuracy
[Table]
for WRN-16-8 showed a similar trend as that of ResNet architectures.
However, for VGG-16 with
BN architecture, SupCon had lower performance than CE on every dataset.
Although the contrastive learning approach does not seem to result in signiﬁcant changes for the VGG-16 with BN encoders,
SelfCon-S was better than or comparable to CE.
[Table]
1-stage training 1-stage training framework, i.e., not decoupling the encoder pretraining and linear evaluation, on the single-viewed batch, is standard for supervised learning.
With a multi-exit framework, Zhang et al. (2019a) propose Self-Distillation (SD), which distills logit information within the network itself, i.e.,
L SD =
α L CE + (1 − α )
L KL .
We replaced the KL divergence term with L self - s in Eq. 2, and distilled the features from the projection head, instead of the logits.
From Table 7, we observed that adding cross-entropy loss to the sub-network (CE w/ Sub) improved the backbone network’s classiﬁcation performance.
However, the results of SD suggested that there is a saturation of increase even when the classiﬁer of sub-network better converged.
Meanwhile, we hypothesize that feature distillation of SelfCon-S makes the encoder learn better representation than logit distillation of classiﬁers.
We would like to highlight that SelfCon loss in 2-stage training still demonstrated the best classiﬁcation accuracy, as Khosla et al.
(2020) argued that representation learning mitigates the poor generalization performance of CE-based 1-stage training.
C ONCLUSION
We proposed SelfCon learning, which self-contrasts the features from the multiple levels of a network.
SelfCon learning is free from the issues that multi-viewed batch triggers.
We found that SelfCon loss maximizes the lower bound of label-conditional MI between the intermediate and the last features, and it is related to improving the classiﬁcation performance.
In addition, we analyzed why SelfCon is better by exploring the effect of single-view and sub-network.
We verify by extensive experiments, including ImageNet-100 and ImageNet, that SelfCon-S loss outperforms CE and SupCon.
Appendix A P ROOFS A. P ROOF OF P ROPOSITION 4.
Proof.
We simply extend the exact bound of InfoNCE (Poole et al., 2019;
Sordoni et al., 2021).
Here, we consider the supervised setting where there are C training classes.
Without loss of generality, choose a class c out of C classes, and let x and z be different samples that share the same class label c .
The derivation for the multi-view ( z being an augmented sample of x ) is similar.
For conciseness of the proof, we consider that no other image in a batch shares the same class.
We prove that minimizing the SupCon loss (Khosla et al., 2020) maximizes the lower bound of conditional MI between two samples x and z given the label c :
[Equation]
for some function F and hyperparameter K .
We start from Barber and Agakov’s variational lower bound on MI (Barber & Agakov, 2004).
[Equation]
where q is a variational distribution.
Since q is arbitrary, we can set the sampling strategy as follows.
First, sample z 1 from the proposal distribution π ( z | c ) where c is a class label of x .
Then, sample ( K − 1) negative samples { z 2 , · · · , z K } from the distribution (cid:80) c (cid:48) (cid:54) =
c π ( z , c (cid:48) ) , so that these negative samples do not share the class label with x .
We augment each negative sample by random augmentation and concatenate with the original samples, i.e., { z 2 , · · · , z K , z K +1 , · · · , z 2 K − 1 } , where z K
+ i − 1 is the augmented sample from z i for 2 ≤ i ≤ K .
We deﬁne the unnormalized density of z 1 given a speciﬁc set { z 2 , · · · , z 2 K − 1 } and x of label c is
[Equation]
where ψ is often called a discriminator function (Hjelm et al., 2018), deﬁned as ψ F ( u , v ) =
F ( u ) · F ( v ) for some vectors u
[Equation]
v .
By setting the proposal distribution as π ( z | c ) =
p ( z | c )
[Equation]
we obtain the MI bound:
[Equation]
13) where the second inequality is derived from Jensen’s inequality.
Because Eq.
12 is an expectation with respect to the sampled x and z 1 , the case where the anchor is swapped to z 1 is also being considered.
A neural network F (backbone in our framework) with L layers are formulated as F =
f L ◦ f L − 1 ◦ · · · ◦ f 1 .
Then, ψ F ( u , v ) =
F ( u ) · F ( v ) =
f 1: L ( u ) · f 1: L ( v ) .
We deﬁne another discriminator function as ψ † F ( u , v ) =
f ( (cid:96) +1): L ( u ) · f ( (cid:96) +1): L ( v ) .
Obviously, the following equivalence holds:
[Equation]
Note that f 1: (cid:96) ( u ) is the (cid:96) -th intermediate feature of input u .
Following the same procedure as in Eq.
[Equation]
From above, as the intermediate feature is arbitrary to the position, we can obtain a similar inequality:
[Equation]
A. P ROOF OF P ROPOSITION 4.
Proof.
In Section 4.1, we proved that SupCon loss maximizes the lower bound of conditional MI between the output features of a positive pair.
We can think of another scenario where the network F now has a sub-network G .
Assume that the sub-network has M > (cid:96) layers: G = g M ◦ g M − 1 ◦· · ·◦ g 1 .
As we discussed in the paper, the exit path is placed after the (cid:96) -th layer, so regarding our deﬁnition of the sub-network, G shares the same parameters with F up to (cid:96) -th layer, i.e., g 1 = f 1 , g 2 = f 2 , · · · , g (cid:96) =
f (cid:96) .
Deﬁne ψ G ( u , v ) =
G ( u ) · G ( v )
We introduce a discriminator function that measures the similarity between the outputs from the backbone and the sub-network, ψ FG ( u , v ) =
F ( u ) · G ( v ) .
Similarly, ψ GF ( u , v ) =
G ( u ) · F ( v ) .
Considering that the SelfCon-S loss has the anchored function of F and G , we obtain an upper bound of two symmetric mutual information.
Here, z 1 = x because SelfCon-S loss is deﬁned on the single-viewed batch and we assume that other images in a batch (i.e., z 2 , · · · , z K ) are sampled from the different class label with x .
[Equation] [Equation]
[Equation] [Equation]
Due to the symmetry of mutual information,
[Equation]
[Equation]
In addition, we can similarly bound the SelfCon-M loss.
As the derivation of SupCon loss bound, only consider the anchor x and its positive pair z 1 .
When the anchored feature is F ( x ) , the contrastive features are: G ( x ) , G ( z ) , and F ( z ) .
By symmetry, when the anchored feature is G ( x ) , the contrastive features are: F ( x ) , F ( z ) , and G ( z ) .
As the derivation of the SupCon loss bound, we assume the augmented negative samples,
i
[Equation]
e., { z 2 , · · · , z K , z K +1 , · · · , z 2 K − 1 } .
[Equation]
A. P ROOF OF P ROPOSITION 4.
F ( x ) and G ( x ) are the output features from the backbone network and the sub-network, respectively.
Recall that T denotes the sharing layers between F and G .
T ( x ) is the intermediate feature of the backbone which is also an input to the auxiliary network path.
Before proving the following Lemma, we would like to note that the usefulness of mutual information should be carefully discussed on the stochastic mapping of a neural network.
If a mapping
T ( x ) (cid:55)→ F ( x ) is a deterministic mapping, then the MI between T ( x ) and F ( x ) is degnerate because
I ( T ( x ); F ( x )) is either inﬁnite for continuous T ( x ) (conditional differential entropy is −∞ ) or a constant for discrete T ( x ) which is independent on the network’s parameters (equal to H ( T ( x )) ).
However, for studying the usefulness of mutual information in a deep neural network, the map
T ( x ) (cid:55)→ F ( x ) is considered as a stochastic parameterized channel.
In many recent works about information theory with DNN, they estimate the mutual information via assuming a noisy system in the network (Shwartz-Ziv & Tishby, 2017; Goldfeld et al., 2019; Saxe et al., 2019b; Goldfeld et al., 2019).
By adopting the noisy DNN framework, it is possible to provide theoretical guarantees that are not degenerate.
Also, information theory in deep learning, especially in contrastive learning, is based on the InfoMax principle (Linsker, 1989) which is about learning a neural network that maps a set of input to a set of output to maximize the average mutual information between the input and output of a neural network, subject to a speciﬁed constraints and/or noise processes.
This InfoMax principle is nowadays widely used for analyzing and optimizing DNNs.
Most works for contrastive learning based on maximizing mutual information grounds on the InfoMax principle, and they are grounding on the stochastic mapping of an encoder.
Moreover, Poole et al.
(2019) rigorously discussed the mutual information with respect to a stochastic encoder.
This is common practice in representation learning context where x is data and z is a learned stochastic representation.
Lemma A.1.
[Equation] [Equation]
Proof.
As F ( x ) and G ( x ) are conditionally independent given the intermediate representation T ( x ) , they formulate a Markov chain as follows: G ← T → F .
Under this relation, the following is satisﬁed:
[Equation]
Eq. 30 is from the property of conditional entropy, and Eq.
32 is due to the conditional independence and marginalization of g .
From Lemma A.1 and Eq.
24, we can prove the Proposition 4.3.
Proof.
[Equation] [Equation]
Strictly speaking, SelfCon-S loss does not guarantee the lower bound of either ( (cid:4) ) or ( (cid:3) ) in Eq. 39.
However, SelfCon-S loss guarantees the label-conditional MI between the intermediate and the last feature, which is ( (cid:4) + (cid:3) ).
A. C ONNECTION B ETWEEN S ELF C ON L OSS AND C LASSIFICATION P ERFORMANCE
We used the variational inference to prove that SelfCon loss is the lower bound of label-conditional MI between the intermediate and the last features; thus, we assumed a probabilistic model as Eq. 8.
When the anchor feature is similar to the negative pairs (i.e., different class representations, z 2:(2 K − 1) ), this model becomes a distribution with random mapping, and SelfCon loss cannot be optimized.
Therefore, representations of other classes should be farther to decrease the gap between SelfCon loss and MI.
After all, SelfCon loss has improved performance because it aims to increase the lower bound of the label-conditional MI between F and G while increasing the distinction between different class representations.
B I MPLEMENTATION D ETAILS
Network architectures We modiﬁed the architecture of networks according to the benchmarks.
For smaller scale of benchmarks (e.g., CIFAR-10, CIFAR-100, and Tiny-ImageNet) and the residual networks (e.g., ResNet-18, ResNet-50, and WRN-16-8), we changed the kernel size and stride of a convolution head to 3 and 1, respectively.
We also excluded Max-Pooling on the top of the
ResNet architecture for the CIFAR datasets.
Moreover, for VGG-16 with BN, the dimension of the fully-connected layer was changed from 4096 to 512 for CIFAR and Tiny-ImageNet.
MLP projection head for contrastive learning consisted of two convolution layers with 128 dimensions and one ReLU activation.
For the architectures of sub-networks, refer to Appendix C.
Representation learning We refer to the technical improvements used in SupCon, i.e., a cosine learning rate scheduler (Loshchilov & Hutter, 2016), an MLP projection head (Chen et al., 2020), and the augmentation strategies (Cubuk et al., 2019): {ResizedCrop, HorizontalFlip, ColorJitter,
GrayScale}.
ColorJitter and GrayScale are only used in the pretraining stage.
We used 8 RTX
Ti GPUs and set the batch size to 1024 for the pretraining and 512 for the linear evaluation.
We used the batch size of 512 for ResNet-18 and 256 for ResNet-50 when pretraining on the ImageNetdataset.
We trained the encoder and the linear classiﬁer for 400 epochs and 40 epochs, respectively, in ImageNet-100 benchmark.
Meanwhile, we trained 1000 epochs and 100 epochs, respectively, in all other benchmarks.
Every experiment used SGD with 0.9 momentum and weight decay of 1e-4 without Nesterov momentum.
All contrastive loss functions used temperature τ of 0.1.
For a fair comparison to (Khosla et al., 2020), we set the same learning rate of the encoder network as 0.5.
Then, we linearly scaled the learning rate according to the batch size (Goyal et al., 2017).
For ImageNet-100 dataset, we used the small learning rate of 0.0625.
We used 5.0 as a learning rate of the linear classiﬁer for the residual architecture, but it was robust to any value and converged in nearly 20 epochs.
Meanwhile, for VGG architecture, only a small learning rate of 0.1 converged.
1-stage training In the 1-stage training protocol, we trained the encoder network jointly with a linear classiﬁer on the single-viewed batch.
Most of the experimental settings were same as those of representation learning, but we trained the encoder for 500 epochs on CIFAR and Tiny-ImageNet dataset.
We used the batch size of 512 and the learning rate of 0.8.
For the cross-entropy result of ImageNet-100 dataset, we trained for 90 epochs with learning rate decay of 0.1 after 30 andepochs.
Here, we used the batch size 512 and the learning rate 0.2.
In the multi-exit framework, we used linear combination of loss functions for the backbone and sub-network.
We used only cross-entropy loss for the backbone network, and weighted linear combinations of various loss functions (e.g., KL divergence and SelfCon-S) for the sub-network.
Self-Distillation (Zhang et al., 2019a) used the interpolation coefﬁcient α of 0.5.
For the 1-stage version of SelfCon loss, we follow the coefﬁcient form in (Tian et al., 2019b):
L = L CE + β L self .
We set the coefﬁcient β = 1 . 0 for ResNet-18 and β = 0 .
8 for ResNet-50.
Note that we used the outputs from the projection head instead of the logits.
We used temperature τ = 3 . 0 for SD and τ = 0 .
1 for SelfCon loss.
C A BLATION S TUDY ON S UB - NETWORK
The structure, position, and the number of sub-networks are important to the performance of SelfCon learning.
First, in order to ﬁnd a suitable structure of the sub-network, the following three structures were attached after the 2 nd block of an encoder: (1) a simple fc , fully-connected, layer, (2) small structure which reduced by half the number of layers in the non-sharing blocks, (3) same structure which is same as the backbone’s non-sharing block structure.
After we found the optimal structure, we ﬁxed the structure of sub-network and found which position is the best.
For ResNet architectures, there are three positions to attach; after the 1 st , 2 nd , and 3 rd block.
For VGG-16 with BN, there are four positions and for WRN-16-8, there are two positions possible.
Note that blocks are divided based on the Max-Pooling layer in VGG-16 with BN.
Table 8 presents the ablation study results for ResNet-18 and ResNet-50, and Table 9 presents the results for WRN-16-8 and VGG-16 with BN.
We highlighted the selected structure and position in
Table 8 and Table 9.
For the ImageNet-100 dataset, we used same structure positioned after the 2 nd block of ResNet-18.
Obviously, there are many combinations of placing sub-networks, and Table 8 and Table 9 presented an interesting result that some performance was the best when sub-networks are attached to all blocks.
It seems that increasing the number of positive and negative pairs by the various views improves the performance.
It is consistent with the argument of CMC (Tian et al., 2019a)
that the more views, the better the representation, but our SelfCon learning is much more efﬁcient in terms of the computational cost and GPU memory usage.
However, for the efﬁciency of the experiments and better understanding of the framework, we stuck to a single sub-network in all experimental settings.
[Table]
[Table]
D E XTENSIONS OF S ELF C ON L EARNING D. S ELF C ON IN U NSUPERVISED L EARNING
Although we have experimented only in the supervision, our motivation of contrastive learning with a multi-exit framework can also be extended to unsupervised learning.
We propose a SelfCon loss function for the unsupervised scenario and present the linear evaluation performance of ResNetarchitecture on CIFAR-100 dataset.
Loss function Under the unsupervised setting, Chen et al.
(2020) proposed a simple framework for contrastive learning of visual representations (SimCLR) with NT-Xent loss.
SimCLR suggests a contrastive task that contrasts the augmented pair among other images.
Therefore, the objective of
SimCLR is exactly same as Eq.
1, while each sample has only one positive pair of its own augmented image, i.e., P ≡ { ( i + B ) mod 2 B } .
We denote this loss as L sim .
We fomulate SelfCon loss in unsupervised setting as SimCLR, using a positive set without label information.
We formulate SelfCon loss with the m ulti-viewed and u nlabeled batch ( SelfCon-MU ) as follows:
[Equation]
i
[Equation]
As SelfCon loss in supervised setting, we exclude anchor sample from P i ∗
and N i to avoid contrasting the same feature, i.e., P i ∗ ← P i ∗ \ { i } when ω = ω ∗ and N i ← N i \ { i } when ω = ω 2 .
Here, we used τ = 0 .
5 for the unsupervised SelfCon loss and L sim .
[Table]
We also formulate SelfCon loss with the s ingle-viewed and u nlabeled batch ( SelfCon-SU ) as follows:
[Equation]
Similarly, N i ← N i \ { i } when ω
= ω 2 .
For the positive set, since this loss is based on the single-viewed batch, we have an empty positive set when ω = ω ∗ .
Experimental results
All implementation details for unsupervised representation learning are identical with those of supervised representation learning in Appendix B, except for temperature τ of 0.5 and linear evaluation learning rate of 1.0.
We used a small sub-network attached after the 2 nd block.
Table 10 shows the linear evaluation performance of unsupervised learning on ResNetin CIFAR-100 dataset.
However, we empirically found that the encoder failed to converge with
SelfCon-MU and SelfCon-SU loss.
D. S ELF C ON WITH A NCHORS ONLY FROM THE S UB - NETWORK
We suspect that Eq.
40 and 41 allow the backbone network to follow the sub-network, which makes the last feature learn more redundant information about the input variable, without any label information.
Thus, unsupervised loss function under the SelfCon framework needs to modiﬁed.
When the anchor feature is from the backbone network, we remove the loss term which contrasts the features of sub-network.
Strictly speaking, it does not perfectly prevent the backbone from following the sub-network, since there is no stop-gradient operation on the outputs of backbone network when the outputs of sub-network are the anchors.
However, we hypothesize that it helps prevent the encoder from collapsing to the trivial solution by the contradiction of IB principle.
We conﬁrmed the performance of revised loss functions in both unsupervised and supervised scenarios.
Loss function
[Equation]
We exclude anchor sample from P i ∗ and N i , and all notations are same as Eq. 40, except for the coefﬁcient α where we used 1.0.
For the supervised setting, simply change P i ∗ to { p ∈
I | y p = y i } and L sim to L sup .
Note that P i ∗ ← P i ∗ \ { i } when ω ∗ =
G and N i ← N i \ { i } when ω
2 =
G .
We get rid of the situation that the anchor F ( x ) contrasts the positive pair in sub-network G ( x p ∗ ) .
Still,
[Table] [Table]
the proposed loss function includes SimCLR loss ( L sim ) or SupCon loss ( L sup ) in the unsupervised or supervised setting, respectively.
[Equation]
For the supervised setting, we change the above equation as the equally-weighted linear combination of SupCon-S loss and Eq.
43 with P i ∗ ≡ { p ∈
I | y p =
y i } .
Note that we also exclude contrasting the anchor itself in SupCon-S loss term.
Experimental results In Table 10, we also reported the accuracy of SelfCon-MU and SelfCon-SU loss according to Eq.
42 and 43.
Surprisingly, in this case, SelfCon-MU outperformed SimCLR loss (Chen et al., 2020), improving 1.3%p.
Unfortunately, SelfCon-SU had not converged again, although it improved the result in a small amount compared to Eq.
While SelfCon-MU has SimCLR loss term which makes the backbone encoder still learn meaningful features, SelfCon-SU loss does not have the anchor features from the backbone, which makes the backbone hard to be trained.
Table 11 summarizes SelfCon-M and SelfCon-S loss, removing the anchors from the backbone in the supervised setting, i.e., supervised version of Eq.
42 and 43.
As we expected, these variants of
SelfCon-M and SelfCon-S further improved the classiﬁcation performance.
E D ETAILS OF MI E STIMATION
Table 12 summarizes the full details of mutual information estimation, measured on CIFARwith the pretrained ResNet-50 encoders.
Note that we considered the encoder output without the projection head.
We used three types of estimators: InfoNCE (Oord et al., 2018), MINE (Belghazi et al., 2018), and NWJ (Nguyen et al., 2010).
As expected, SelfCon-M and SelfCon-S loss exhibited larger MI between the intermediate and the last feature of the backbone network than CE and SupCon loss.
F Q UALITATIVE E XAMPLES FOR V ANISHING G RADIENT
In Figure [Table] we have already showed that the sub-network solves the vanishing gradient problem through the visualization for gradient norms of each layer.
In Figure [Table] we also visualized qualitative examples using Grad-CAM (Selvaraju et al., 2017).
We used the gradient measured on the last layer in the 2 nd block when the sub-network is attached after the 2 nd block.
In order to compare the absolute magnitude of the gradient, it is normalized by the maximum and minimum values of the two methods,
SelfCon-M and SupCon.
As in Figure 7, SelfCon learning led to a larger gradient via sub-networks, and more clearly Grad-CAM highlighted the pixels containing important information in the images [Table]
G I MAGE N ET R ESULTS
[Table]
We experimented with the full-scale ImageNet benchmark to enhance the reliability of SelfCon learning on the large-scale dataset.
Due to the limited computational resources, we used
ResNet-18 architecture for both the pretraining and linear evaluation.
For the pretraining stage, we experimented with various batch sizes and epochs. Besides, we tuned the learning rate as 0.125 for 1024 batch size and 0.25 forbatch size. In the linear evaluation stage, we trained the linear classiﬁer with a batch size of 256, 40 epochs, and a learning rate of 6.0.
Other hyperparameters are the same as Appendix B. For the cross-entropy result, we used the batch size of 256 and the learning rate of 0.1.
The settings of epochs and the learning rate rule are the same as the ImageNet-100 experiments.
We summarized the experimental results in Table 13.
For the experiment with 1024 batch size and 400 epochs, it is still consistent that SelfCon-M showed better performance than SupCon, but the multi-viewed methods outperformed the singleviewed counterparts.
We consider the possible reasons as follows.
(1) Under-ﬁtting issue due to the tremendous number of samples and relatively small size of architecture.
(2) Insufﬁcient number of training epochs (also refer to Figure 4).
(3) Relatively small batch size with respect to the number of classes (1000) in ImageNet (also refer to Table 4).
Therefore, we did two more ablation studies: (1) pretraining with 800 epochs and (2) pretraining with larger batch size ( B = 2048 ).
In the experiments with 800 epochs, the trend that multi-viewed methods outperformed the singleviewed counterparts did not change.
However, the classiﬁcation accuracy of each algorithm signiﬁcantly increased as the pretraining epochs became longer.
On the other hand, for the larger batch
[Table]
size (i.e., B = 2048 ), we observed that SelfCon-S is better than SupCon. The difference seems to be marginal but it is noteworthy that the training time for SelfCon-S is more than twice shorter than SupCon and memory consumption is much lower. It is an important result in terms of the recent research ﬂow for the compression in contrastive learning (Koohpayegani et al., 2020; Fang et al., 2021).
From the result of SelfCon-S in 2048 batch size, we are highly convinced that the trend in
ImageNet is the same as the other benchmarks.
The experiments for the larger architecture (e.g.,
ResNet-50, ResNet-101) or the larger batch size (e.g., 4096, 6144) are left for the future work.
H M EMORY U SAGE AND C OMPUTATIONAL C OST
In Table 14, we reported the computational cost for pretraining with ResNet-50.
ImageNetdataset result is not reported because ResNet-50 with batch size over 256 exceed the GPU limit.
The overall trend is similar to that in Table 3.