H OW D OES C ONTRASTIVE P RE - TRAINING C ONNECT D ISPARATE D OMAINS ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Pre-training on massive unlabeled datasets greatly improves accuracy under distribution shifts.
As a ﬁrst step toward understanding this, we study a popular pre-training method, contrastive learning, in the unsupervised domain adaptation (UDA) setting where we only have labeled data from a source domain and unlabeled data from a target domain.
We begin by showing on 4 benchmark vision datasets that out-of-the-box contrastive pre-training (even without large-scale unlabeled data) is competitive with other UDA methods.
Intuitions from classical UDA methods such as domain adversarial training focus on bringing the domains together in feature space to improve generalization from source to target.
Surprisingly, we ﬁnd that contrastive pre-training learns features that are very far apart between the source and target domains.
How then does contrastive learning improve robustness to distribution shift?
We develop a conceptual model for contrastive learning under domain shifts, where data augmentations form connections between classes and domains that can be far apart.
We propose a new measure of connectivity —the relative connection strengths between same and different classes across domains—that governs the success of contrastive pre-training for domain adaptation in a simple example and strongly correlates with our results on benchmark vision datasets.
I NTRODUCTION
In applications such as image recognition for self-driving cars (Yu et al., 2020; Sun et al., 2020) or medical image diagnosis (AlBadawy et al., 2018; Dai & Gool, 2018), machine learning models often fail on examples drawn from a different distribution than training.
Pre-training on large-scale unlabeled data (Chen et al., 2020;
He et al., 2020; Radford et al., 2021) can substantially boost accuracy under distribution shift (Radford et al., 2021;
Hendrycks et al., 2019; Fisch et al., 2019; Yogatama et al., 2019;
Devlin et al., 2019).
For example, CLIP (Radford et al., 2021), which is trained on internet-scale image and text data, has shown impressive robustness beneﬁts on ImageNet variants such as ImageNetV2.
Similarly, pre-training methods such as BERT (Devlin et al., 2019) have improved robustness in NLP — for example, multilingual pre-training signiﬁcantly improves performance on unseen languages (Liu et al., 2020).
While other robustness methods that use unlabeled data such as self-training (Prabhu et al., 2021; Sohn et al., 2020; Berthelot et al., 2021) and domain adversarial training (Shu et al., 2018;
Ganin et al., 2016) jointly optimize a labeled and unlabeled objective, pre-training is particularly suitable for large unlabeled datasets since we can generically pre-train once on unlabeled data and then specialize the model to downstream labeled data via ﬁne-tuning.
How does pre-training improve robustness to distribution shift when the pre-training method is not tailored to any downstream task?
In this paper, we run controlled experiments to understand the robustness beneﬁts of a popular pre-training method, contrastive pre-training
(Chen et al., 2020;
He et al., 2020; Caron et al., 2020), even without large-scale unlabeled data.
We directly apply contrastive pre-training to unsupervised visual domain adaptation (UDA), where unlabeled data from a shifted target domain is used to adapt models trained on labeled source dataset.
We compare contrastive pre-training (SwAV (Caron et al., 2020)) to UDA methods such as SENTRY (Prabhu et al., 2021)
(self-training) and DIRT-T (Shu et al., 2018) (domain adversarial training) where all methods use the same unlabeled data (source and target).
We ﬁnd that SwAV achieves comparable or better results on the DomainNet, BREEDS Living-17, and BREEDS Entity-30 benchmarks.
Furthermore, we show that pre-training once on unlabeled data from many domains further improves the average target accuracy over the consituent source-target pairs, showing that extra unlabeled data improves contrastive pre-training for domain adaptation.
Given the strong performance of contrastive pre-training, we seek to understand why contrastive pre-training works for domain adaptation.
Domain adversarial training and self-supervision algorithms are commonly based on the intuition that domains get merged in feature space (Tzeng et al., 2014;
Ganin et al., 2016; Tzeng et al., 2017; Sun et al., 2019; Ben-David et al., 2010; Shu et al., et al., 2021; Thota & Leontidis, 2021).
This is support by theoretical notions such as H ∆ H -divergence, where domains that can be discriminated well and results in worse generalization bounds for target accuracy (Ben-David et al., 2010).
A natural hypothesis is that contrastive pre-training brings domains together by bringing the representations of strong augmentations of the same image together (Sun et al., 2019).
Surprisingly, we ﬁnd that the source and target domains are still far apart in the feature space learned by contrastive pre-training.
A domain classiﬁer trained on the pre-trained representations of strongly augmented images from the sketch and painting domains in DomainNet has 94% accuracy, which is even higher than the accuracy of a domain classiﬁer on the original (augmented) input space (75%).
We study this phenomenon from the lens of the recently proposed augmentation graph (HaoChen et al., 2021), which forms connections between examples if they have similar augmentations (e.g., a small crop + grayscale can connect sketch and real photos, Figure 1).
These connections are central to the contrastive objective, which seeks to bring neighbors in the graph closer together while pushing apart non-neighbors.
From both simulations and real datasets, we deﬁne a critical connectivity quantity which seems to govern whether domain adaptation will be successful (Figure 1).
In particular, the connection strength between the same classes across different domains should be relatively greater than the connection between different classes across different domains.
Importantly, the absolute connectivity between domains can be small and the domains can be far apart in input space.
This suggests that contrastive learning circumvents the separation of domains through the connectivity structure of the (general-purpose) augmentations used.
Using our understanding, we ablate contrastive learning by removing subsets of unlabeled data that contribute most to connecting the two domains.
We do this by training a domain classiﬁer to distinguish between the source and target and removing examples which are most uncertain under the classiﬁer.
This consistently worsens the target accuracy in comparison to randomly removing the same number of data points (by 2–3% on DomainNet), showing the importance of domain connectivity for adaptation.
S ETUP
We consider a classiﬁcation problem from an input space X ⊆
R
d to a label space Y =[ K ] .
Data.
Let P S and P T denote the source and target distributions of (¯ x,y ) pairs, respectively, where ¯ x ∈ X and y ∈ Y .
The labeled source dataset S consists of n S input-output pairs from P S .
We also have access to an unlabeled target dataset T with n T unlabeled inputs from P T , and we assume the label sets for the source and target are identical.
In some cases, we also consider an additional related dataset R with n R unlabeled inputs from a related distribution R .
Model and metrics.
The goal is to learn a classiﬁer h : X →Y with low classiﬁcation error on the target distribution L 0 − 1 ( h )=
E ¯ x,y ∼ P T [ 1 [ h (¯ x ) (cid:54) =
y ]] .
The pre-training algorithms we consider consist of two steps: ﬁrst, we train an encoder function f : X →Z into a representation space Z ⊆ R d (cid:48)
(where d (cid:48) denotes the representation dimension), then we train a classiﬁcation head g : Z → Y .
The ﬁnal classiﬁer h
= g ◦ f composes the encoder with the classiﬁcation head.
2. C ONTRASTIVE L EARNING
Contrastive learning (He et al., 2020; Chen et al., 2020; Caron et al., 2020) trains representations to be similar between augmented views of the same example and dissimilar between augmented views of random examples.
Example image augmentations include random cropping, color jitter, and Gaussian blur (Chen et al., 2020).
The basic assumption is that the semantic content of the data is invariant under the augmentations.
Though there are several variants of contrastive learning, we focus on the following objective from HaoChen et al.
(2021) for our conceptual understanding.
Let X denote the “augmented” space containing all data-augmented views of natural examples (e.g., all random crops of natural ImageNet images).
Data augmentation is deﬁned as a random function A : X →X , where P [ A ( x )= x ] denotes the probability that x ∈X augments to x ∈X .
A pair of augmented views is called a positive pair and denoted x,x
+ if a natural point x is ﬁrst drawn and then x,x + are the results of two calls to A ( x ) .
On the other hand, a negative pair x,x − arises from two natural data points x,x (cid:48)
when x = A ( x ) , x − =
A ( x (cid:48) ) .
Minimizing the following loss function simultaneously brings together the representations of positive pairs x,x + and repels the representations of negative pairs x,x − : [Equation]
[Equation]
Here the ﬁrst expectations are taken over the marginal distribution over X and the randomness of A .
2. C ONSTRASTIVE P RE - TRAINING FOR D OMAIN A DAPTATION
Let D denote the unlabeled input data used for contrastive pre-training (e.g., some combination of the source unlabeled data S , target unlabeled data T , and/or additional unlabeled data R ).
We use the following simple algorithm to apply contrastive pre-training to domain adaptation:
(Pre-train)
Learn an encoder function f : X →Z via contrastive learning on D . 2.
(Fine-tune)
On labeled source data S , train a classiﬁer h (cid:44) g ◦ f : X →Y as the composition of the pre-trained encoder f and a classiﬁer head g .
In this paper, we take “ﬁne-tuning” to mean updating both the parameters of f and g .
C ONTRASTIVE P RE - TRAINING IS A STRONG DOMAIN ADAPTATION METHOD
We evaluate contrastive learning and relevant baselines on 4 benchmark vision datasets and observe that contrastive learning achieves comparable or better performance in all cases.
3. D ATASETS
BREEDS (Santurkar et al., 2020).
BREEDS is a subpopulation shift benchmark derived from
ImageNet by constructing a hierarchical tree structure of classes from WordNet.
Nodes at a speciﬁed depth of the tree become the labels for the classiﬁcation task, and descendant nodes are treated as subpopulations that can be randomly partitioned into source and target domains.
DomainNet (Peng et al., 2019).
DomainNet is a large unsupervised domain adaptation task, consisting of approximately 600,000 images and 345 classes in 6 domains.
For our experiments we utilize the same ﬁltered version of DomainNet from Prabhu et al.
(2021), which uses 40 of theclasses and the sketch, painting, photograph, and clipart domains.
[Table]
STL-10 → CIFAR-10.
STL-10 and CIFAR-10 are two classic image classiﬁcation datasets, each consisting of 10 classes (Coates et al., 2011; Krizhevsky, 2009).
Each dataset has a class label that is not present in the other, and hence we follow the procedure of French et al.
and ﬁlter out the examples in the non-overlapping classes, resulting in a 9-class classiﬁcation problem.
3. M ETHODS
Baselines.
We run empirical risk minimization (ERM) on the source dataset for each task, i.e., a “source-only” baseline.
In addition, for BREEDS and DomainNet we use these ERM models as weight initialization for SENTRY, a self-training algorithm for domain adaptation that achieved state of the art results on several domains of DomainNet when initialized with ImageNet-pretrained models (Prabhu et al., 2021).
For STL → CIFAR, we use Dirt-T (Shu et al., 2018), the current state of the art algorithm for this task.
Contrastive pre
training.
For BREEDS and DomainNet we use SwAV (Caron et al., 2020), a contrastive learning algorithm that achieved state-of-the-art linear evaluation accuracy on ImageNet.
SwAV contrasts examples in feature space rather than input space and uses a novel multi-crop data augmentation strategy using several crops of different sizes.
For the lower-resolution task
STL → CIFAR we use SimCLR (Chen et al., 2020), which uses only 2 crops and therefore scales straightforwardly to smaller input dimensions.
3. R ESULTS
Main comparison.
For all 4 tasks, contrastive pre-training is competitive when compared to ERM and SOTA unsupervised domain adaptation baselines.
On DomainNet (top of Table 1), contrastive pre-training on source and target unlabeled data achieves higher target accuracy than all joint-training baselines both on 5 of 12 domain pairs and on average over all pairs (44.91% vs. the second best, 43.99%).
On Living-17 (middle of Table 1), contrastive learning on source and target unlabeled data improves over SENTRY over 4%, while on Entity-30, contrastive learning (source + target) is within 0.3% of the target accuracy of SENTRY.
On STL → CIFAR, SimCLR on the target unlabeled data achieves 0.8% higher target accuracy than Dirt-T.
Extra unlabeled data from related domains.
We also consider adding extra unlabeled data from other (related) domains in DomainNet, Living-17, and Entity-30.
In particular, we can pre-train once on an unlabeled dataset that consists of a superset of the domains we want to adapt to, and ﬁne-tune on the available source data.
While this is not a fair comparison to other domain adaptation methods, the ability to scale to large unlabeled datasets is a natural advantage of pre-training.
[Table]
On DomainNet, we consider contrastive pre-training with SwAV on unlabeled data from all the domains at once, which we denote as SwAV + extra.
The top Table 1 shows that adding extra unlabeled data improves the target accuracy of contrastive pre-training on all pairs and improves the average by almost 6% (44.91% to 50.75%).
On Living-17 and Entity-30 (middle of Table 1), we additionally consider contrastive pre-training with SwAV with 1) only source, 2) only target, and 3) with all of ImageNet as the unlabeled data.
We ﬁnd that increasing the amount of unlabeled data steadily increases the target accuracy, and pre-training on all of ImageNet improves over source + target by 8% and 4% respectively for Living-17 and Entity-30.
Ease of hyperparmeter tuning
The SENTRY self-training baseline requires extensive hyperparameter tuning (e.g., ﬁnding optimal balancing coefﬁcients for the joint losses, or learning rate schedules) for every transfer task and is difﬁcult with access only to source labels.
In contrast to the baseline methods, for contrastive pre-training we did not need to sweep over hyperparameters or use target labels for model selection.
We used nearly identical hyperparmeters for both BREEDS datasets (Living-17 and Entity-30) and all DomainNet pairs (details in Appendix A).
No explicit knowledge of distribution shift.
We note that SENTRY uses explicit knowledge of label imbalance within its algorithm, on top of being designed for the domain adaptation task of adapting from source to target.
In contrast, we used off-the-shelf contrastive pre-training, which was designed for general representation learning, with no additional information about distribution shift.
C ONNECTIVITY M ODEL FOR D OMAIN A DAPTATION
How does contrastive pre-training improve target accuracy with unlabeled data?
Conventional intuitions and theory for domain adaptation are based on reducing the distance between the source and target domains (Ben-David et al., 2010; Ganin et al., 2016).However, we see empirically that contrastive pre-training keep the domains very separated in representation space.
Towards building an understanding for how contrastive pre-training connects domains without merging them, we extend a conceptual framework based on connectivity via data augmentations (HaoChen et al., 2021) for distribution shifts.
We deﬁne a crucial measure of domain connectivity that relates properties of the data distribution and contrastive augmentations to the success of contrastive learning for domain shifts on a simulated data model, DomainNet, Living-17, and Entity-30.
4. C ONTRASTIVE PRE - TRAINING KEEPS DOMAINS APART
Contrary to common intuitions for domain adaptation methods, we ﬁnd that contrastive learning does not bring the domains together in feature space.
Table 2 shows the average test error of classiﬁers trained to discriminate between examples from the same class but different domains on DomainNet.
We train and test the classiﬁers on images using the same augmentations as SwAV and ﬁnd that contrastive pre-training learns features that are actually more discriminable between domains than in the original input space (7.5% vs. 27.3% error on average over all pairs).
These estimates for the baselines and for individual pairs are provided in table 3.
Next, we measure the separability in feature space and input space between classes within a single domain (e.g., 3% vs. 24% error in the real domain, table 3).
We note that the separability between domains in feature space is comparable to the separability between classes , suggesting that contrastive learning may cluster both domains and classes in separate axes.
4. U SING C ONNECTIVITY TO C HARACTERIZE D OMAIN S HIFT
In this section, we set up our conceptual model of connectivity used to study contrastive learning.
Connectivity via augmentations in domain adaptation.
HaoChen et al.
(2021) proposed viewing contrastive learning as operating on an augmentation graph.
The augmented inputs from X constitute the vertices and the edge weight between x,x (cid:48) represents how likely they are to be generated from the same natural example (i.e., how likely they are to be chosen as a positive example during contrastive pre-training).
HaoChen et al.
(2021) show that when examples from different classes x,x (cid:48) ∈
X are sparsely connected via the augmentation graph (i.e., rarely augment into the same augmented example), and the connectivity within a class is strong, the contrastive pre-training leads to features that obtain good accuracy on the downstream task.
Extension to domain adaptation.
Motivated by the augmentation graph framework, we deﬁne a notion of connectivity for domain adaptation on a graph where class-domain pairs are vertices.
We denote the set of examples belonging to class c and domain d (which we refer to as the class-domain pair ( c,d ) ) as P ( c,d ) .
We formalize this notion in the following deﬁnition:
Deﬁnition 1 (Connectivity) .
The connectivity between two class-domain pairs ( i,d ) and ( j,d (cid:48) ) is conn(( i,d ) , ( j,d (cid:48) ))
(cid:44) P
[ A ( x )=
A ( x (cid:48) )] , where x ∼ P ( i,d ) and x (cid:48) ∼ P ( j,d (cid:48) ) .
Intuitively, the connectivity between two class-domain pairs is the probability that two randomly sampled points from each class-domain pair are augmented to the same point.
Deﬁnition 1 is deﬁned for two arbitrary class-domain pairs, which then allows us to build further deﬁnitions of connectivity involving multiple class-domain pairs.
Such connectivity deﬁnitions would allow us to more rigorously describe (for instance)
our observation in the DomainNet dataset that points of the same class but different domains are more discriminable than points in different classes within the same domain.
We deﬁne these instances of connectivity using Deﬁnition 1 below:
Deﬁnition 2 ( ( α,β,γ ) (cid:57) connectivity) .
A source-target pair of distributions S and T over X (with domain labels d S , d T , respectively) satisﬁes ( α,β,γ ) (cid:57) connectivity if
E i ∈
[ K ] [conn(( i,d S ) , ( i,d T ))]= α .
(same-class-different-domain) 2.
E i,j ∈ [ K ] [conn(( i,d ) , ( j,d ))]= β .
(different-class-same-domain, d ∈{ d S ,d T } ) 3.
E i,j ∈ [ K ] [conn(( i,d S ) , ( j,d T ))
]= γ .
(different-class-different-domain)
4. I LLUSTRATIVE T OY E XAMPLE : B INARY C LASSIFICATION
As an illustrative example, consider the simple case of binary classiﬁcation where each class-domain pair consists of a single data point.
This toy setting pinpoints a simple connectivity condition such that linear classiﬁcation on the pre-trained features results in good target accuracy.
Setting (Figure 2).
Assume a uniform marginal distribution over the natural inputs, and suppose data augmentation transforms each data point directly into another with a certain probability, so that the entire augmentation graph consists of the 4 points (2 classes, 2 domains) each identiﬁed by their class and domain.
Formally, the example ( i,d ) (of class i and domain d ) has augmentation given by
A (( i,d ))=
   ( i,d ) with probability ζ ( i,d (cid:48) ) with probability ( α/ 2 ζ ) ( j,d ) with probability ( β/ 2 ζ ) ( j,d (cid:48) ) with probability ( γ/ 2 ζ ) where j (cid:54)
=
i , d (cid:48) (cid:54) =
d .
We also assume ζ > max { α,β,γ } (i.e., data augmentation is most likely to map each data point to itself) and appropriately constrain α + β + γ = 2 ζ (1 − ζ ) so that the probabilities sum to 1.
Note that the speciﬁc transition probabilites are selected so that the distribution shift satisﬁes ( α,β,γ ) (cid:57) connectivity.
We also number the points 1 − 4 in the following order: class 1 (source), class 1 (target), class 2 (source), class 2 (target). [Figure]
Pre-training.
HaoChen et al.
(2021) show that minimizing the spectral contrastive loss (given in
Eq. 1) is equivalent to computing the spectral decomposition of the adjacency matrix of the population augmentationgraph.
Theleadingeigenvectorsoftheresultingdecompositionarethenusedasthelearned representations, so that the representation of example i is the concatenation of the i th components of those eigenvectors.
Because the population here is ﬁnite we compute the spectral decomposition algebraically and express the learned representations as a function of connectivity.
Agnostic of the ordering of the relevant parameters, the unordered eigenvectors of this toy example are:
[Equation]
with associated eigenvalues
[Equation]
We discard the eigenvector v 1 , as it assigns the same feature to every point, and use the top 2 remaining eigenvectors as the learned features.
For example, if v 2 and v 3 are the top remaining eigenvectors, then the features for data point 2 would be the 2 nd component of v 2 and v 3 (namely, ( − 1 , 1) ).
Fine-tuning.
We consider learning a linear classiﬁer upon the pre-trained features.
The middle panel of Figure 2 illustrates the fortunate case, in which α>γ and β >γ (but possibly α ≤ β ).
In this case, λ 2 >λ 4 and λ 3 >λ 4 and therefore the learned features use v 2 and v 3 and are (1 , 1) , (1 , − 1) for class 1 and ( − 1 , 1) , ( − 1 , − 1) for class 2 .
Then, a linear classiﬁer trained on the source uses only the class information, which is contained in the second index, and labels the target accurately.
Note that the source classiﬁer labels the target correctly if and only if it uses solely the features given by v 3 for prediction.
Two possible failure modes occur when α is smallest (and therefore v 3 is discarded) and when β is smallest (and therefore the source classiﬁer can use features from either v 3 or v 4 ).
Importantly in this toy setting, whether or not the features of the target examples are oriented correctly depends only on whether α and β are both > γ .
Consequently, the classes within each domain can be very connected–more so than across domains–and target accuracy can be high.
To illustrate the importance of this connectivity ratio on a more ﬁne-grained setting, we run simulations and observe sharp thresholding behavior when γ surpasses α (Figure 3, simulation details in Appendix C).
4. C ONNECTIVITY G OVERNS THE S UCCESS OF P RE - TRAINING FOR D OMAIN A DAPTATION
Connectivity on benchmark datasets.
We estimate the input space domain connectivity on
DomainNet, Living-17, and Entity-30 by training classiﬁers between various pairs of class-domain pairs (test errors provided in Table 2, full details provided in Section B).
We ﬁnd that all the datasets satisfy α>γ –the necessary condition for the good case in the toy example–but α<β for DomainNet while α>β for BREEDS. Intuitively, on natural distribution shifts α is likely to be higher than γ , since the data augmentation should be more likely to connect class-domain pairs where only the domain differs than to bridge differences across both the domain and the class.
[Figure]
T a r g e t a cc u r a c y g a i n ( o v e r w o r s t p a i r w i t
h s a m e t a r g e t )
[Figure]
[Figure]
Domain and class classiﬁers are nearly orthogonal.
We check the extent to which the domain and class information are separated by training 3 linear classiﬁers on the pre-trained representation space: 1) Source classiﬁer (which discriminates classes in source) 2) Target classiﬁer (which discriminates classes in target) 2 3) Domain classiﬁer (discriminates between source or target domains).
We compare the linear weights of these classiﬁers by computing the cosine similarity.
We ﬁnd that
The source and target classiﬁers are very similar; on average over the classes, the cosine similarity of linear weights for the same class from the source and target classiﬁers is high (around 0.20 for DomainNet, 0.34 for Living-17).
The domain classiﬁer is nearly orthogonal to the source and target classiﬁers.
The average cosine similarity between the weights of any class from the source or target classiﬁers and the weights of the domain classiﬁer is 0.01.
This suggests that the class information and domain information are contained in different directions of the feature space, and gives an explanation for how both classes and domains can be easily separated in feature space while maintaining the ability to generalize from source to target.
Learning to discriminate classes results in a classiﬁer with weights that are nearly orthogonal to the weights for domain classiﬁcation, suggesting a natural amount of ‘domain agnosticity” despite not removing domain information from the features.
Detailed results on individual pairs are provided in Appendix D.
Connectivity ratios correlate with target accuracy.
We construct a scalar connectivity quantity and ﬁnd that it correlates with target accuracy of contrastive learning on DomainNet.
In the simple example, contrastive pre-training for domain adaptation works when α > γ and β > γ .
The ﬁrst condition ( α > γ ) is relatively more important since if β (connectivity between different classes in the same domain) is very high, then the classes within a domain are hard to discriminate and the classiﬁcation task itself will be hard.
Thus, we deﬁne the quantity
[Equation]
where we take (cid:15) = 0 .
1 to “discount” the importance of the second ratio.
We multiply the two ratios to express the logical “and”.
The larger this quantity is, the better the target accuracy should be.
Figure 4 shows a plot of the quantity with target accuracy of contrastive learning on DomainNet pairs.
Because different target domains can have different intrinsic errors, we “debias” the target accuracy of each source-target pair by subtracting the worst accuracy over all pairs with the same target domain.
We ﬁnd a strong correlation between the connectivity and target accuracy (Pearson r of 0.87 for S+T pre-training) (Figure 4).
Ablating connectivity.
We ﬁnd that the connectivity between examples of the same class but different domain is also very important, as those examples “bridge” the domains.
We verify this intuition by systematically ablating the connectivity on a subset of Living-17 by removing those examples that contribute most to domain connectivity.
Speciﬁcally, for each class we trained discriminators between the source and target examples of that class and removed the n pre-training data points from each class-domain pair on which the discriminator was least conﬁdent; these examples can be seen as the ones contributing most to domain connectivity.
For subsample sizes of { 1176 , 2352 , 5880 , 11760 } we consistently reduce target accuracy as compared to random subset removal (Figure 5; conﬁdence subsampling increases error relatively by 6% on average compared to random).
R ELATED W ORK
Domain adaptation.
Ben-David et al.
(2010) prove generalization bounds in the distribution shift setting.
Their bounds rely on a notion of distance between two data distributions called H ∆ H divergence being small, which many methods such as domain adversarial training are inspired by (Ganin et al., 2016;
Tzeng et al., 2014; 2017).
Our conceptual model based around connectivity provides a way to explain how contrastive pre-training can work for domain adaptation even when the H ∆
H divergence between domains is large.
Domain adaptation with self-supervision.
Prior works have explored the application of selfsupervision to domain adaptation.
Sun et al.
(2019) propose optimizing on hand-crafted self-supervised tasks such as predicting the angle of rotation of a rotated image.
These tasks are jointly optimized on the source and target data along with the labeled source loss.
As part of their comparisons, they found that ﬁrst pre-training on target (or source and target) data
and then ﬁne-tuning on source data separately (instead of jointly) was highly non-competitive with their other methods.
In contrast, we ﬁnd that pre-training with a general, off-the-shelf contrastive learning objective is competitive with other state-of-the-art domain adaptation techniques.
Contrastive objective for domain adaptation.
Contrastive objectives have also been applied to domain adaptation problems.
Wang et al.
(2021) jointly train the contrastive loss on source and target data.
These methods rely on explicit anchor pairs that encourage domain alignment (Wang et al., 2021), or still employ some form of explicit domain alignment (Thota & Leontidis, 2021).
We consider pre-training instead of joint training and do not use any domain knowledge about the distribution shift problem.
Self-training.
Berthelot et al.
(2021) use self-training and consistency regularization with a distribution alignment method, but require estimating the target label distribution and tuning a conﬁdence thresholding parameter (tuned using target test labels ).
Cai et al.
(2021) give a label propagation analysis for self-training in domain adaptation.
They have only one parameter which governs both the different-class-same-domain and different-class-different-domain connections, requiring both to be small.
In our framework, we allow the different-class-same-domain connectivity to not be small, in accordance with our observation that domains can be well separated in contrastive feature space.
They give experiments on self-training when operating on a contrastive pre-trained representation space, but their focus is on self-training while the pre-trained representations are an empirical detail.
One inﬂexible characteristic of standard self-training is that it relies on the unlabeled data having the same classes as the labeled data (in order to pseudolabel unlabeled points), which may not allow them to utilize diverse sources of extra unlabled data like many pre-training methods can.
C ONCLUSION
In this work we focused on evaluating and understanding contrastive pre-training under domain adaptation.
Pre-training has the advantage of amortizing the cost of processing a large unlabeled dataset into a pre-training step that is done only once.
While pre-training loses the advantage of seeing labeled data (and thus the downstream task) to adapt its usage of unlabeled data, we ﬁnd that
contrastive pre-training is still competitive with other domain adaptation algorithms.
We hope that our connectivity model can give insights to improve pre-training data selection, develop better augmentations for pre-training with distribution shift in mind, and improve ﬁne-tuning methods that exploit the properties of the pre-trained feature space.
E THICS
Our experiments use standard benchmark vision datasets in domain adaptation, which is publicly available data.
In general, however, large-scale unsupervised learning can involve data scraped from the internet which may lead to privacy concerns.
Care should be taken when curating datasets for large scale contrastive pre-training in practice.
R EPRODUCIBILITY
We detail the hyperparameters and experimental details in the paper and appendix.
We have uploaded a zipped code archive and will publish the code on Github upon acceptance.
A A DDITIONAL E XPERIMENTAL D ETAILS A. D ATASETS
BREEDS.
We use the dataset creation functions deﬁned in the robustness Python library to generate the Living-17 and Entity-30 tasks from the original ImageNet dataset (Engstrom et al., 2019;
Russakovsky et al., 2015).
The Living-17 dataset is an animal classiﬁcation task which consists of nodes in the subtree rooted at the “living thing” node in the WordNet hierarchy.
An example of a label is “ape” with subpopulations of gibbons, orangutans, gorillas, and chimpanzees.
The Entity-30 dataset is a much more general task, incorporating nodes in the “entity” subtree.
Labels include “building” and “home appliance”.
The trailing number in the dataset name is the total number of classes in that dataset.
DomainNet.
We use the ofﬁcial SENTRY repo at https://github.com/virajprabhu/
SENTRY , which ﬁlters the original DomainNet dataset automatically as described in Section 3.
This reﬁnement is done to eliminate much of the noise present in the original DomainNet dataset (Tan et al., 2020).
STL → CIFAR.
CIFAR-10 consists of 32 × 32 images from the former TinyImages dataset, and
STL-10 is derived from ImageNet and contains images with resolution 96 × 96 .
We resize the STLimages to 32 × 32 to match the resolution of CIFAR10.
The two non-overlapping classes (“monkey” in CIFAR-10 and “frog” in STL10) are removed from both datasets before training.
A. B ASELINES AND H YPERPARAMETER T UNING
ERM (source-only)
• BREEDS:
We use the same hyperparameters as Santurkar et al.
For Entity30, we train for 300 epochs and divided the learning rate by 10 every 100 epochs.
On Living17, we train for 450 epochs and divided the learning rate by 10 every 150 epochs.
For both datasets we use a ResNet50 with a learning rate of 0.1, a weight decay of 10 − 4 , and a batch size of 128. • DomainNet: The SENTRY algorithm runs ERM with class balancing (starting with ImageNet-pretrained initialization) prior to beginning entropy minimization, and therefore the SENTRY repository contains an ERM implementation and hyperparameters for DomainNet.
We run ERM for 150 epochs and multiply the initial learning rate by 10x, keeping all other hyperparameters constant.
• STL
→ CIFAR10: We report the exact results from Shu et al.
DIRT-T. DIRT- T (Shu et al., 2018) is
a
domain
adaptation
method that addresses two
ﬂaws
of domain
adversarial neural networks (Ganin et al., 2016): 1) distribution matching is a weak constraint, and 2) in some domain adaptation settings there does not exist a good joint classiﬁer on both source and target.
The authors address the ﬁrst shortcoming by adding a conditional entropy regularization term so that the model’s decision boundaries do not overlap high-density regions of data.
This is inspired by the cluster assumption , which states that the input space is divided into well-separated clusters, one for each class in the label space.
The lack of a good classiﬁer on both source and target is then addressed via self-training on the unlabeled target data.
We report the STL10 → CIFAR10 results from Shu et al.
SENTRY.
For each pair of domains on DomainNet we conduct a hyperparameter search through λ src ∈{ 0 . 5 , 1 . 0 , 1 . 5 } (the weight on the supervised classiﬁcation loss) and learning rates ∈{ 0 . 01 , 0 . 001 } and report the model results with the highest source evaluation accuracy.
On BREEDS we keep the same default hyperparameters from the SENTRY repo and search over 3 learning rates { 0 . 001 , 0 . 01 , 0 . 1 } , choosing the model with the highest target evaluation accuracy to report.
SENTRY underperforms the ERM models when the source domain is clipart images (from DomainNet), which could be the result of not using ImageNet-pretrained classiﬁers for initialization as Prabhu et al.
(2021) do.
Therefore, as the joint-training baseline for DomainNet we take the maximum of ERM and SENTRY.
A. S IM CLR
We use the ofﬁcial SimCLR repository from Google for our experiments ( https: //github.com/google-research/simclr ).
We use a ResNet18 with a batch size of 256, a learning rate of 0.2, and weight decay 10 − 4 .
The projection head has two layers and an output dimension of 64.
We pre-train the model for 400 epochs with square-root learning rate scaling and we train the linear probe for 100 epochs on batches of size 512 and a learning rate of 0.1.
A. S W AV
We use the public SwAV repository available at https://github.com/facebookresearch/ swav and kept almost all of the hyperparameters provided by the original paper for 400 epoch, 256-batch-size training on ImageNet.
However, we used a batch size of 512 and additionally made the following changes based on the Github issues answered by the original authors:
We avoid using a queue on DomainNet and the subsampled variant of Living-17 in order to stabilize training.
For pre-training on the full Living-17 and Entity-30 datasets, we introduce the queue at epoch 60.
We set the number of prototypes to be 10 times the number of classes (170, 300, andfor Living-17, Entity-30 and DomainNet, respectively).
We set (cid:15) =0 .
We set the base learning rate to 0.6, following a linear scaling rule based on batch size.
We always ﬁne-tuned from the ﬁnal iterate of SwAV pre-training (400 epochs).
On DomainNet, we ﬁne-tuned SwAV models for 50 epochs using the ERM implementation in the SENTRY repository (without running any joint-training algorithm), keeping all hyperparameters other than the number of epochs constant.
We report the target test accuracy of the ﬁnal model (after 50 epochs).
On BREEDS Living-17, we ﬁne-tuned SwAV models for 100 epochs with a cosine learning rate schedule without restarts.
We use SGD with initial learning rate 0.1 for the classiﬁer head and 0. for the encoder, momentum 0.9, and weight decay 0.0001.
We use a batch size of 96, and once again report the target test accuracy of the ﬁnal model (after 100 epochs).
On BREEDS Entity-30, which contains 300K examples, we instead linear probe.
B R ESULTS AND P ROTOCOL FOR E STIMATING ( α,β,γ ) (cid:57) CONNECTIVITY
Table 3 reports the connectivity estimates for the input space and the feature spaces learned by ERM,
DANN, and SwAV for all pairs of DomainNet domains.
To estimate the connectivity between 2 class-domain pairs ( i,d ) and ( j,d (cid:48) ) , we use the following algorithm:
Label all training examples of class and domain ( i,d ) as 0 and all training examples of ( j,d (cid:48) ) as 1.
Discard the remaining examples.
Train a ResNet50 for 100 epochs using SwAV augmentations and cosine learning rate.
We did not exhaustively tune this training step, but we kept the hyperparameters constant for all the pairs.
Collect the test set analogously to step 1, and evaluate the classiﬁer on augmented data.
Each domain in DomainNet-S has a unique label distribution (all of which are far from uniform), and therefore in computing the average connectivity we compute the weighted mean, where each pair of (class, domain) pairs is weighted by the ratio of the less to more frequent label (0 or 1).
C S IMULATION D ETAILS
To obtain an even more ﬁne-grained picture, we run simulations on toy data settings, where we can precisely vary each type of connectivity, and observe:
Sharp thresholding behavior when α and γ cross over.
α must be nontrivial.
The simulation is on a ﬁnite population so that we can study the spectral decomposition directly; however, later we show empirically that these ideas still hold in real-world, continuous-population settings.
For each domain-class pair dc , we generate N points in augmentation space.
These augmented points inherit the domain-class assignments of their corresponding natural points (we have to ensure that all [Table] the augmented points are distinct).
Then for any new natural point x ∈X that we want to augment, we simply choose one of our existing augmented points x at random and designate x as the augmentation of x .
Since we have assigned domain-class memberships to this base set of augmentations, we can deﬁne the connectivity of two domain-class pairs ( d 1 c 1 ,d 2 c 2 ) to be the probability that a natural point from d 1 c 1 is “augmented into” an augmented point in d 2 c 2 (and vice-versa).
This probability is something we can easily change in the simulations to vary this notion of connectivity.
D A DDITIONAL D ETAILS FOR S ECTION 4.
As another method for evaluating the connectivity measures (in addition to Figure 4), we inspect the weights learned from a linear regression model from 3 features [ α,β,γ ] to the target accuracy (de-biased).
The weights are [ − 6 . 87] , which gives a large negative weight to α (different- class-different-domain connecitivity), large positive weight to γ (same-class-different-domain), and a small positive weight to β (different-class-same-domain) as expected from our intuitions.
The following table contains more detailed results on the cosine similarity between domain and class classiﬁers.
[Table]