M
ULTI -D OMAIN S ELF -S UPERVISED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Contrastive self-supervised learning has recently gained signiﬁcant attention owing to its ability to learn improved feature representations without the use of label information.
Current contrastive learning approaches, however, are only effective when trained on a particular dataset, limiting their utility in diverse multi-domain settings.
In fact, training these methods on a combination of several domains often degrades the quality of learned representations compared to the models trained on a single domain.
In this paper, we propose a Multi-Domain Self-Supervised
Learning (MDSSL) approach that can effectively perform representation learning on multiple, diverse datasets.
In MDSSL, we propose a three-level hierarchical loss for measuring the agreement between augmented views of a given sample, agreement between samples within a dataset and agreement between samples across datasets.
We show that MDSSL when trained on a mixture of CIFAR-10,
STL-10, SVHN and CIFAR-100 produces powerful representations, achieving up to a 25% increase in top-1 accuracy on a linear classiﬁer compared to singledomain self-supervised encoders.
Moreover, MDSSL encoders can generalize more effectively to unseen datasets compared to both single-domain and multidomain baselines.
MDSSL is also highly efﬁcient in terms of the resource usage as it stores and trains a single model for multiple datasets leading up to 17% reduction in training time.
Finally, for multi-domain datasets where domain labels are unknown, we propose a modiﬁed approach that alternates between clustering and
MDSSL.
Thus, for diverse multi-domain datasets (even without domain labels),
MDSSL provides an efﬁcient and generalizable self-supervised encoder without sacriﬁcing the quality of representations in individual domains.
I NTRODUCTION
Self-supervised contrastive training (Chen et al., 2020; He et al., 2020; Misra & van der Maaten, 2020; Caron et al., 2020b) has become a popular paradigm for unsupervised representation learning as it shows impressive results on linear classiﬁcation tasks, almost matching the performance of a supervised model trained from scratch.
However, we ﬁnd that current self-supervised models are only effective when trained on a single-domain.
This can hinder their deployment in large scale realworld settings where data almost always comes from multiple diverse domains.
We illustrate this issue in Table 1, where we show that a popular self-supervised model, SimCLR (Chen et al., 2020), trained on CIFAR-10 (Krizhevsky et al., a) does not generalize to other domains at the test time.
We observe that the top-1 accuracy of a linear classiﬁer signiﬁcantly drops on unseen datasets.
This means that a different self-supervised model needs to be trained for every new dataset, which can add signiﬁcant computational overheads given that training these models often require large batch sizes and a large number of training epochs (Chen et al., 2020;
He et al., 2020; Wu et al., 2018).
One potential solution for self-supervised learning on multi-domain datasets is to train the models on the union of all input domains.
Unfortunately, this solution performs poorly and fails to obtain a good performance on every individual dataset and does not generalize well to unseen domains.
To illustrate this, we trained SimCLR on the union of multiple datasets including CIFAR-
(Krizhevsky et al., a), CIFAR-100 (Krizhevsky et al., b), SVHN (Netzer et al., 2011) and STL- (Coates et al.).
The trained model is unfavorable as it signiﬁcantly decreases the top-1 accuracy in all training datasets compared to the single-domain baselines (see Table 1).
To tackle these issues, we propose Multi-Domain Self-Supervised Learning (MDSSL) , a technique for obtaining a uniﬁed embedder that can be trained on multiple domains.
In MDSSL, we train the model over the union of multiple datasets using a three-level hierarchical loss involving:
Embedding similarities of two views of a sample
: In the ﬁrst level, we
maximize
agreement (i
e. the cosine similarity between l 2 -normalized vectors) between embeddings of two augmented views of a given sample.
Embedding similarities of samples from a given dataset: In the second level, we minimize the pairwise agreements between embeddings of all samples within a dataset.
Embedding similarities of samples from different datasets: In the third level, we minimize the pairwise agreement between samples across all training datasets.
The ﬁrst two levels ensure that the model learns high quality representations for each individual domain.
The third level of the MDSSL loss encourages the model to learn distinguishable representations between domains.
This approach assumes that domain labels are known during training.
We also extend MDSSL to more realistic multi-domain setups where domain labels are unknown .
In such scenarios, we present an iterative approach that alternates between clustering and MDSSL at ﬁxed intervals.
We use clustering to detect pseudo-domain-labels for each training dataset and use these labels in the MDSSL loss.
We also propose a robust version of clustering by reducing outlier noise which further improves the performance of MDSSL in an entirely unsupervised setup.
In summary, the goal of MDSSL is to compute improved latent representations of samples from multiple diverse datasets using a single self-supervised model (See Figure 1).
We summarize our contributions as follows:
• We show that current self-supervised learning techniques such as SimCLR, under multidomain setups, show degraded performance on downstream linear classiﬁcation tasks and do not generalize well to unseen domains.
• We propose Multi-Domain Self-Supervised Learning (MDSSL) that uses a new loss function for self-supervised learning that supports training over multiple domains at once and pushes the model to learn distinguishable representations across datasets.
• We show that MDSSL trained on a mixture of CIFAR-10, STL-10, SVHN and CIFAR-100, shows a 25% increase in top-1 accuracy and is more efﬁcient (See Table 1).
• We also experiment over DTD and Tiny-ImageNet and show that MDSSL generalizes better to unseen domains of varying diversity compared to both single-domain SimCLR and multi-domain SimCLR.
• We propose an iterative approach combining MDSSL with clustering to train over multidomain datasets without the use of domain labels.
• We further improve our clustering approach by introducing robust clustering that prevents outlier noise from affecting domain labels.
[Table]
R ELATED W ORK
Supervised classiﬁcation techniques involve minimizing a loss function (e.g. the cross-entropy loss) to match model predictions to true labels.
Unsupervised classiﬁcation methods, on the other hand, learn to classify data without the use of training labels, usually with the use of clustering techniques (Bojanowski & Joulin, 2017; Dosovitskiy et al., 2014; YM.
et al., 2020; Bautista et al., 2016; Caron et al., 2018; 2019; Huang et al., 2019).
More recently, new unsupervised techniques called self-supervised representation learning have been proposed. A self-supervised model learns by observing every instance of the given data and assigns its own labels, and then performs a classiﬁcation task (Bojanowski & Joulin, 2017; Dosovitskiy et al., 2014; Wu et al., 2018;
Dosovitskiy et al., 2016).
To simplify the complexity of instancelevel classiﬁcation, a memory bank (Wu et al., 2018; He et al., 2020) can be used with the help of contrastive learning (Gutmann & Hyv¨arinen, 2010;
Hjelm et al., 2019; van den Oord et al., 2019; Grill et al., 2020).
Contrastive learning (Arora et al., 2019; Tosh et al., 2021; Bachman et al., 2019) is a temperature-controlled cross-entropy loss between positive pairs of similar samples and negative pairs of dissimilar samples.
Positive pairs are usually considered as multiple transformations (views) (Tian et al., 2020) of a given sample using stochastic data augmentation.
SimCLR (Chen et al., 2020) shows that contrastive learning can be done without the use of a memory bank, using the samples within a batch, if we have large enough batches.
SwAV (Caron et al., 2020a) uses a mixture of contrastive learning and clustering to form a swapped prediction problem that can learn even with very small batch sizes.
Finally, contrastive learning can beneﬁt from training labels, if available with a simple modiﬁcation of contrasting between samples within a class and taking samples of other classes as negatives (Khosla et al., 2020).
Each of these approaches show remarkable linear classiﬁcation accuracy on single-domain setups.
Extending self-supervised learning to multiple diverse domains, other than ImageNet (Russakovsky et al., 2015), is a relatively less explored topic (Wallace & Hariharan, 2020).
When multiple related domains are available during training, a possible approach is to use mutual information to simultaneously encode common invariant information and domain-speciﬁc information of each image (Feng et al., 2019).
In our paper, we focus on a general setup where we combine diverse unrelated domains and evaluate individual domain-speciﬁc tasks.
M ULTI -D OMAIN S ELF -S UPERVISED L EARNING WITH D OMAIN L ABELS
In this section, we deﬁne the Multi-Domain Self-Supervised Learning (MDSSL) paradigm for D training datasets where domain labels are known.
We deﬁne x di ∈ R r and ˜ x id ∈ R r as two transformed views of the i th sample from the d th dataset, d ∈ { 1 , ..., D } .
Similar to SimCLR, we use a base encoder f ( . ) and a two-layer MLP projection head g ( . ) to map a given sample into the latent space.
We deﬁne the latent representations of the two views of the i th sample from the d th dataset as z di = f
( g ( x di ))
∈ R r (cid:48) and ˜ z id = f ( g ( ˜ x id ))
∈ R r (cid:48) where r (cid:48) is the size of each latent representation.
We represent mini-batches containing 2 N samples (2 views per sample) from D datasets as a matrix
X ∈ R 2 ND × r , whose corresponding latent representation is denoted by Z ∈ R 2 ND × r (cid:48) .
We then calculate a similarity matrix S ∈ R (2 ND ) × (2 ND ) that contains the exponential cosine similarity scaled by a temperature parameter τ , between all the latent representations in a given batch.
The ( i, j ) th element of S is:
[Equation]
where z i ∈ R r (cid:48) and z j ∈ R r (cid:48) are the
i th and j th row of Z , respectively.
S di represents the cosine similarity between z di and ˜
z id .
S d,d (cid:48) ij represents the cross-dataset cosine similarity between z di and z d (cid:48)
j where d, d (cid:48) ∈ { 1 , ..., D } .
MDSSL aims to solve the following optimization problem:
[Equation]
where θ is the set of model parameters and 0 < λ 1 ≤ 1 and λ 2 ≥ 0 are tunable regularization parameters.
This is a three-level hierarchical loss.
(2) maximizes the similarity between two transformed views ( x di and ˜ x id ) in the latent space.
(3) minimizes the similarity between every pair of samples ( x di and x dj ) within a dataset.
(4) minimizes the similarity between pairs of samples ( x di and x d (cid:48) j ) across different datasets.
When λ 1 = 1 and λ 2 = 0 , this optimization is simpliﬁed to SimCLR
(Chen et al., 2020).
Therefore, we use λ 1 = 1 and λ 2 = 0 as the baseline in all our experiments.
λ 1 helps us control the extent to which we want to minimize the similarity within a domain. We empirically observe that relaxing λ 1 from the SimCLR baseline ( λ 1 = 0 ) to a value slightly less than 1, in fact generates better structure in the latent space by clustering samples within a domain relatively closer compared to samples outside a domain (See Appendix Section A.3).
λ 2 should always be non-negative as we always want to minimize the agreement between samples of different datasets.
We can implement the MDSSL loss efﬁciently as it only involves calculating the similarity matrix S once and then selecting elements according to each term mask.
As the number of datasets increases, the size of S increases, gradually increasing the running time of MDSSL.
3. E XPERIMENTAL S ETUP
We use ResNet-50 (He et al., 2016) as the base encoder ( f ( . ) ) and a 2-layer MLP projection head ( g ( . ) ) for all of our experiments.
For data augmentation, we use a combination of random crop , random horizontal ﬂip , random color distortion and random Gaussian blur .
In all experiments, the latent representations are in a 128 -dimensional space and τ = 0 .
We optimize our loss using LARS optimizer (You et al., 2017) with a learning rate of 4 and weight decay of 10 − 6 .
We train with a batch size of 1024 and train over 48 , 000 iterations.
We experiment with the following datasets: CIFAR10 (Krizhevsky et al., a), CIFAR-100 (Krizhevsky et al., b), STL-10 (Coates et al.), SVHN (Netzer et al., 2011), Tiny-ImageNet (Le & Yang, 2015) and DTD (Describable Textures) (Cimpoi et al., 2014).
We resize all images to 32x32 in all our experiments.
We use Nvidia GeForce RTX GPUs.
We measure the quality of representations using the linear evaluation protocol (Kolesnikov et al., 2019; Bachman et al., 2019; van den Oord et al., 2019) where we train a linear classiﬁer on top of frozen MDSSL representations and compute the top-1 accuracy of each domain-speciﬁc classiﬁcation task.
Since we train over multiple domains, we compute the top-1 accuracy over each domain to evaluate the overall model performance.
[Table]
3. MDSSL P ERFORMANCE COMPARED TO S IM CLR BASELINE
In this section, we analyze the performance of MDSSL and compare it to SimCLR trained on single domains (referred to as the single-domain SimCLR) and multiple domains (referred to as the multidomain SimCLR).
Table 1 summarizes our results on CIFAR-10, STL-10, SVHN and CIFAR-100.
We observe that, single-domain SimCLR models generalize poorly on unseen datasets.
For example,
SimCLR trained on CIFAR-10 achieves 92 .
35% top-1 accuracy on CIFAR-10 samples but only 55 . 97% on SVHN samples.
We also observe that the multi-domain SimCLR model shows a degraded performance when evaluated on each individual training domain.
For example, SimCLR trained on the union of samples from CIFAR-10, STL-10, SVHN and CIFAR-100 achieves 82 .
30% top-1 accuracy on CIFAR-10, signiﬁcantly lower than the performance of the single-domain SimCLR model trained on CIFAR10.
Our method, MDSSL, shows a signiﬁcant improvement compared to the multi-domain SimCLR and almost matches the baseline accuracy of single-domain SimCLR models on some of the training domains.
Among the average top-1 accuracy, we observe up to 25% improvement from the singledomain SimCLR and a 10% improvement from the multi-domain SimCLR (See Table 1).
SimCLR would require us to train 4 different single-domain models for these datasets and therefore requires more compute, memory and time.
MDSSL, being a uniﬁed model, signiﬁcantly outperforms SimCLR in terms of resource utilization as shown in Table 2.
This makes MDSSL an efﬁcient solution in limited resource environments.
3. G ENERALIZATION TO U NSEEN D ATASETS
In this section we evaluate the generalization capacity of MDSSL to unseen domains.
We consider two setups: in the ﬁrst case, we use CIFAR-10, STL-10 and SVHN as training datasets (domains containing less diverse datasets as their number of classes are ≤ 10 ) and evaluate the model performance on unseen datasets of CIFAR-100, DTD and Tiny ImageNet (highly diverse datasets whose number of classes are > 10 ).
In the second case, we use CIFAR-100, DTD and Tiny ImageNet as [Table] our training datasets and assess the performances on CIFAR-10, STL-10 and SVHN.
We also add results on ImageNet (250K) which contains 1000 classes, each including 250 samples resized to 32x32.
Table 3 summarizes our results.
Among the single-domain SimCLR models, we observe that SimCLR trained on Tiny-ImageNet generalizes relatively better than other single-domain models since Tiny-ImageNet is comparatively larger and most diverse.
However, the drop in top-1 accuracy of unseen domains from the baseline is very signiﬁcant even for the single-domain SimCLR trained on Tiny-ImageNet ( 42% drop for SVHN).
Similarly, ImageNet (250K) also poorly generalizes to unseen domains.
In our ﬁrst multi-domain setup (with training datasets of CIFAR-10, STL-10 and SVHN), we observe that although these training datasets are relatively less diverse, MDSSL generalizes remarkably well on more diverse datasets like CIFAR-100, DTD and Tiny-ImageNet.
MDSSL also outperforms the multi-domain SimCLR in all domains (training and unseen).
We observe a similar improvement when we train MDSSL on CIFAR-100, DTD and Tiny-ImageNet and on ImageNet (250K), CIFAR-100 and SVHN.
MDSSL outperforms both single and multi-domain SimCLR in terms of generalization capacity.
These results highlight that MDSSL is a favorable solution that achieves good accuracy on training domains and generalizes well to unseen domains.
3. E FFECT OF N UMBER OF T RAINING D ATASETS
In this section, we discuss the behavior of MDSSL as we increase the number of training domains.
We train MDSSL on CIFAR-10 and CIFAR-100 (2-domain baseline).
We then add STL-10, SVHN, Tiny-ImageNet and DTD datasets one by one and train MDSSL. [Figure]
In Figure 2, we observe that as the number of datasets increases, the top-1 accuracy also increases and ﬁnally beats the single-domain baseline.
Therefore, MDSSL beneﬁts from training over a large number of datasets.
3. H YPERPARAMETER S ELECTION
The MDSSL loss is controlled by two regularizers λ 1 and λ 2 , as shown in Section 3.
When λ 1 = 1 and λ 2 = 0 , MDSSL boils down to our baseline, SimCLR (Chen et al., 2020).
As we decrease λ 1 while ﬁxing λ 2 = 0 , we observe that the indistribution similarity increases (See Appendix Section A.3) and eventually, all samples show a mutual similarity of 1 .
Consequently, the top-1 accuracy quickly degrades from the baseline as shown in the ﬁrst plot in Figure 3.
This behavior can be explained by Term 3 of the MDSSL loss in Section 3 which measures the mutual similarity between all samples within a dataset.
Therefore, we ﬁx λ 1 ≥ 0 .
9 so that it mildly increases in-distribution similarity without signiﬁcantly affecting the top-1 accuracy.
We utilize Term 4 of the MDSSL loss by controlling λ 2 , to ensure that domains are more distinguishable in the latent space.
In Figure 3, the second plot shows the top-1 accuracy as we increase λ 2 .
The top-1 accuracy rises steadily at ﬁrst, and then drops at around λ 2 = 0 .
This is because after a certain threshold, the in-distribution representations become too similar which makes them harder to classify.
Therefore, a good balance should be found between λ 1 and λ 2 such that we achieve favourable top-1 accuracy.
[Figure]
M ULTI -D OMAIN S ELF -S UPERVISED L EARNING WITHOUT D OMAIN L ABELS
Most real-world multi-domain datasets are unlabelled (i.e., domain label information is not available).
In this section, we develop an extension of MDSSL for such setups by identifying pseudo domain labels via a clustering approach in the latent space.
As it is common in clustering, we assume the number of domains (denoted by M ) is known.
In the MDSSL loss (especially in Term 4), we need domain labels to compute pairwise similarities of samples from two different domains.
To achieve this, we ﬁrst treat the problem as single-domain self-supervised learning and warm up the MDSSL encoder for the ﬁrst few training iterations using the optimization described in Section 3 with λ 1 = 1 , λ 2 = 0 and D = 1 (i.e., SimCLR training on one domain).
This warm up helps us get somewhat distinguishable representations for samples between M domains and the number of iterations to warm up is determined empirically.
At the end of the warm up, we cluster the latent representations of the entire multi-domain dataset into
M clusters using K-Means clustering (Hartigan & Wong, 1979).
Using these clusters as pseudodomain-labels, we continue training the encoder under the MDSSL loss with λ 1 ≤ 1 , λ 2 >and D = M .
As the training progresses, MDSSL improves the latent structure and therefore, we recompute clusters multiple times (determined empirically) as the training progresses to ensure that improved domain labels are used.
In practice, we observe that clustering does not provide 100% accurate domain labels, especially for datasets that are distributionally similar such as CIFAR-10 and STL-10.
In such cases, we propose to use a robust clustering approach coupled with MDSSL to prevent outlier clustering noise from affecting the MDSSL training.
Let us consider a MDSSL encoder that is warmed up on a multidomain dataset containing M domains.
We cluster the representations of this dataset into M clusters with centroids c 1 , c 2 , . . .
, c M .
Before assigning pseudo-domain-labels to each representation, we ﬁrst determine if they are outliers or not.
If so, we ignore these samples in training MDSSL in the next round.
We say a latent sample
z i is not an outlier if it is signiﬁcantly closer to one of the clustering centroids compared to another.
Concretely, z i is not an outlier if
[Equation]
where (cid:15) ≥ 0 is deﬁned as an outlier threshold.
When (cid:15) is high, it means that the given sample is close to its respective centroid.
When (cid:15) approaches 0 , it indicates that the sample is almost equidistant from at least two centroids and therefore, may not be reliably clustered into one.
We ignore such samples in MDSSL training.
When we perform clustering for the ﬁrst time, we start with (cid:15) =
and each time we repeat clustering, we decay its value exponentially such that it approaches 0 by the end of training to ensure that at the end, all samples contribute to the MDSSL training.
[Figure]
4. P ERFORMANCE OF MDSSL TRAINED WITHOUT D OMAIN L ABELS
Our experimental setup for training MDSSL without domain labels remains the same as the one we explain in Section 3.1.
We perform the SimCLR warmup for 480 iterations and update the clusters every 2 , 400 iterations going forward.
In this section, we consider two mixtures for training datasets: (i) CIFAR-10 and SVHN (containing visually dissimilar samples), (ii) CIFAR-10, STL- (containing visually similar samples).
In Figure 4, we plot the TSNE of CIFAR-10 and SVHN sample embeddings while using either clustering or robust clustering approaches.
We observe that with clustering, we achieve a reasonable separation between domains although there are several outliers.
These outliers are signiﬁcantly reduced while using robust clustering as described in Equation 5.
As a result, the clusters are quite well deﬁned and easily distinguishable.
In Figure 5, we plot the top-1 accuracy of 5 training setups: single-domain SimCLR, multi-domain
SimCLR, MDSSL (with domain labels), MDSSL (with clustering) and MDSSL (with robust clustering).
When trained on CIFAR-10 and SVHN, we observe that MDSSL with clustering outperforms
SimCLR on both datasets and on CIFAR-100 which is an unseen domain.
We also observe that
MDSSL with clustering seems to generalize better to CIFAR-100 compared to MDSSL.
MDSSL with clustering also outperforms SimCLR when trained on CIFAR-10 and STL-10 which are more visually similar.
We also observe that applying robust clustering shows an improvement on all domains including unseen domains (CIFAR-100).
These results highlight that clustering is a useful approach to identify pseudo-domain-labels and when coupled with MDSSL, it helps us learn better representations for seen and unseen domains.
[Figure]
(left) and CIFAR-10 and STL-10 (right).
We observe that, although domain labels are not used in clustering, we are able to improve the performance of MDSSL compared to SimCLR.
D ISCUSSION
We propose Multi-Domain Self-Supervised Learning (MDSSL), as a unifying approach to compute self-supervised representations for a range of datasets.
We support training MDSSL under two setups: with domain labels and without domain labels.
We show that MDSSL achieves up to a 25% increase in top-1 accuracy with linear evaluation compared to the SimCLR baseline on a combination of CIFAR-10, STL-10, SVHN and CIFAR-100.
We also show that MDSSL is signiﬁcantly more efﬁcient than SimCLR in terms of resource (time, compute and memory) utilization, generalizes better than SimCLR in multi-domain setting, and beneﬁts from an increase in the number of training datasets.
In addition, we propose two versions of clustering that can be coupled with
MDSSL when training over multiple domains without the use of domain labels.
MDSSL achieves good performance even under these entirely unsupervised setups.
Our uniﬁed approach, MDSSL, is general-purpose, enables training on diverse multi-domain settings, and can obtain meaningful embeddings achieving state-of-the-art results both on seen (training) and unseen benchmark datasets.
R EPRODUCIBILITY S TATEMENT
We share our code in the supplementary materials.
We also provide several implementation details to ensure reproducibility of all our experiments.
In Sections 3.1 and A.5, we provide a detailed explanation of our training setup including the architecture of our encoder, optimizers, learning rate schedule and training hyperparameters.
We explain the process of hyperparameter selection in
Sections 3.5 and A.3.
E THICS S TATEMENT
We use only publicly available datasets which involve classiﬁcation tasks on general objects, vehicles, animals, etc.
To the best of our knowledge, our work does not have a negative impact on our society or any societal group.
However, as with all machine learning models, MDSSL should not be used on datasets that are inherently biased or involve harmful tasks that target or affect any particular regional, cultural or societal group.
Therefore, before running MDSSL, one must select datasets and downstream tasks such that they are safe and do not amplify any social biases.
A A PPENDIX A. E XPERIMENTS ON N ON -O BJECT -F OCUSED D ATASETS
In this section we discuss the results on 3 datasets that are not object-focused i
[Table]
e, EuroSAT (Helber et al., 2019), Chest X-Ray (Pneumonia) (Kermany et al., 2018) and DTD (Cimpoi et al., 2014)
[Table]
These datasets are understandably not generalizable to unseen domains as shown in Table 4.
However, under a multi-domain setup where we combine all of these domains, MDSSL shows a modest improvement compared to multi-domain SimCLR and almost matches the single-domain baselines
[Table]
[Table]
λ
[Table]
9 , λ 2 =
[Table]
A. C OMPARING MDSSL WITH S IM CLR P RE - TRAINED ON I MAGE N ET
In this section
[Table]
we use a pre-trained SimCLR encoder from Pytorch Lightning Bolts (Falcon et al., 2019) and train a linear classiﬁer on several unseen datasets.
We resize all images to 32x during linear classiﬁcation to maintain consistency with the rest of our experiments.
We realize this may be an unfair comparison since the encoder is pre-trained on 224x224 images.
Nevertheless, we observe a signiﬁcant improvement in generalization of MDSSL over SimCLR pre-trained on full-sized ImageNet on all datasets (See Table 5)
[Table]
[Table]
A. H YPERPARAMETER S ELECTION
In Figure A.1, we plot the similarity matrices of class-averaged samples of MDSSL trained on
CIFAR-10 and SVHN by ﬁxing λ 2 = 0 and varying λ 1 .
As explained in Section 3, higher λ 1 increases the similarity of samples within a domain.
We observe that when λ 1 = − 1 and λ 2 =(SimCLR), the similarity within a domain is comparable with the similarity across domains, meaning that, domains are indistinguishable.
As we increase λ 1 , we see that the similarity within domains increases and eventually, all samples show a mutual similarity of 1 .
In Figure A.1, although similarity within domains increases, we still cannot distinguish between domains.
To achieve this, we utilize Term 4 of the MDSSL loss by controlling λ 2 .
In Figure A.2, we vary λ 2 and ﬁx λ 1 = − 1 (ﬁrst row) and λ 1 = − 0 .
9 (second row).
As λ 2 increases, the similarity across domains decreases and each domain become clearly distinguishable.
When λ 1 = − 0 .
9 , the effect is seen even at lower values of λ 2 , as MDSSL learns to simultaneously increase similarity within domains while decreasing similarity across domains. [Figure]
A. R UNNING T IME OF MDSSL
The MDSSL optimization, as discussed in Section 3, is solved by iterating over the number of training datasets ( D ) in each term.
Therefore, as the number of datasets increases, the running time of MDSSL will increase accordingly.
We use a large batch size of 1024 which also accounts for increased running time for larger datasets.
In Figure A.3, we see that as the number of training datasets increases, number of training hours of MDSSL also increases.
A. I MPLEMENTATION D ETAILS
Table 6 summarizes the entire architecture of each component of MDSSL with the ﬁlter and output dimensions for input image size 3 × 32 × 32 .
Our implementation of MDSSL is on PyTorch. We
[Figure] [Figure] R unn i n g T i m e ( h o u r s ) use ResNet-50 (He et al., 2016) as the base encoder for all our experiments.
Since we have multiple datasets during training, we prepare a DataLoader for each dataset and load batches of sizefrom each dataset.
We refer to these as dataset batches .
When the number of training datasets is low, we concatenate all dataset batches ( X ) and pass it through the encoder ( f ( . ) ) and projection head ( g ( . ) ) to get Z .
However, when the number of training datasets increases, X becomes too large and may require more memory to encode.
In this case, we ﬁrst separately encode every dataset batch and then concatenate all dataset embeddings to get Z .
This trick helps us efﬁciently train MDSSL on 2 GPUs with 4 training datasets and a high batch size of 1024 .
In Section 3.1, we discuss the experimental setup with hyperparameters for MDSSL training.
We summarize these parameters in Table 7.
We evaluate MDSSL using the linear evaluation protocol [Table] [Table] (Kolesnikov et al., 2019; Bachman et al., 2019; van den Oord et al., 2019).
At test time, we discard the projection head ( g ( . ) ) and keep only the ResNet encoder ( f ( . ) ).
We freeze the encoder and deﬁne a trainable linear layer that maps 128 -dimensional features from the encoder to class probabilities.
This is our linear classiﬁer.
We train this classiﬁer over the frozen embeddings from the ResNet encoder for 100 epochs with a batch size of 1024 .
We use the SGD optimizer with an initial learning rate of 0 . 1 .
We summarize all of these parameters in Table 7.
We optimize the linear classiﬁer using the cross-entropy loss and calculate the top-1 accuracy at the end of training.