U NSUPERVISED D OMAIN A DAPTATION THROUGH S ELF -S UPERVISION
Anonymous authors
Paper under double-blind review
A BSTRACT
We develop a very simple but surprisingly effective method for unsupervised domain adaptation – when labeled training data is available from a source domain, but the goal is to have good performance on a target domain with only unlabeled data.
We hope to align the learned representations of the source and target domains while preserving discriminability, by learning to perform self-supervised tasks on both domains simultaneously, and jointly training a source classiﬁer for the main task.
The presented objective is straightforward to implement, easy to optimize, and motivates a simple heuristic for early stopping without a labeled target validation set.
We achieve state-of-the-art results on four out of seven standard benchmarks, and competitive results on segmentation adaptation, demonstrating that our method is effective across a variety of applications, and composes well with another popular pixel-level adaptation method.
I NTRODUCTION
Visual distribution shifts are fundamental to our constantly evolving world.
We humans face them all the time when we navigate a foreign city, read text in a new font, or recognize objects in an environment we have never encountered before.
All of these real-world challenges in human visual perception have direct parallels in computer vision, such as object recognition with backgrounds or lighting conditions different from what was trained on.
Formally, a distribution shift happens when we train on data from one distribution (source), but want to make good predictions on some other distribution (target) that shares the label space with the source.
Classic supervised methods often struggle even for pairs of distributions that humans ﬁnd intuitively similar.
Our paper studies the setting of unsupervised domain adaptation – when we have from the source domain labeled data, and from the target only unlabeled data, which are much easier to obtain, and should still contain useful information about the target.
The general philosophy of the ﬁeld is to induce alignment of the source and target domains through some transformation.
In the context of deep learning, a convolutional neural network maps images to learned representations in some feature space, and to induce alignment is to make the distribution shifts small between the source and target in this shared feature space (Csurka, 2017; Wang & Deng, 2018; Gopalan et al., 2011).
If, in addition, the representations preserve discriminability on the source domain, then we can learn to have good performance on the source representations, which now generalizes to the target under the reasonable assumption that the representations of the two domains have the same ground truth.
Most previous approaches implement this philosophy of alignment by optimizing for a measurement of distributional discrepancy in the feature space (details in section 2), often some form of maximum mean discrepancy (MMD), or a learned discriminator of the source and target as an approximation to the total variation distance.
Both measurements lead to the formulation of the training objective as a minimax optimization problem a.k.a.
adversarial learning, which is known to be very difﬁcult to solve; unless carefully balanced, the push and pull in opposite directions can often cause wild ﬂuctuations in the discrepancy loss and lead to sudden divergence (details in section 2).
Therefore, we propose to avoid minimax optimization altogether through a very different approach.
Our main idea is to train a model on the same task on both domains simultaneously, thus aligning the features needed for this task on both the source and target domains, provided that the model
[Table]
[Table]
/uf1Uwxv/UyoJEWu2PKjMJUEY5LfT0ZCc4ZyZgllWtishE2opgxtSxVbgrd68jrpXNU9t+7dX9cazaKOMpzBOVyCBzfQgCa0oA0MYniGV3hz0Hlx3p2P5WjJKXZO4Q+czx+NdpFv</latexit>
[Figure]
does not overﬁt 1 .
Indeed, if we had labels on both domains, we could simply use these labels for the intended main task and the problem would trivial.
Since we do not have labels on the domain, we propose to use a self-supervised task, which creates its own labels directly from the data by transforming the data and predicting the transformation (details in section 3). In fact, we can use multiple self-supervised tasks, each aligning the two domains along the feature dimensions relevant to that task, as shown in Figure 1. To preserve the labeling information necessary for the intended main task, we jointly train all the self-supervised tasks on both domains together with the main task on the source.
Thanks to the alignment induced by the self-supervised tasks, labeling information can then transfer from the source domain to the target.
Our main contribution is to combine the ﬁelds of unsupervised domain adaptation and selfsupervision.
Among the related works from both ﬁelds discussed in section 2, for unsupervised domain adaptation there is no previous work on alignment through self-supervision, and previous works on self-supervision, to the best of our knowledge, study different problem settings.
We do not propose new self-supervised tasks, but offer insights in section 3 on how to select the right ones for domain adaptation, and propose in section 4 a novel training algorithm on those tasks, using batches of samples from both domains. Our proposed method also brings these beneﬁts: 1)
In section 5, we demonstrate state-of-the-art results on some standard benchmarks for object recognition, and competitive results for segmentation, in the setting of unsupervised domain adaptation. The smooth and regular convergence behavior of our method, together with its simplicity, make it easier to incorporate into research and applications. Conceptually, it shows that adversarial learning is not the only way to implement the philosophy of alignment; this existence proof of a different way might encourage new directions in the unsupervised domain adaptation community.
R ELATED WORK
In this section we provide a brief overview of the two ﬁelds that we are combining.
Unsupervised domain adaptation in computer vision can be divided into, broadly speaking, three classes of methods (Saenko et al., 2010; Torralba & Efros, 2011).
The dominant class, which our work belongs to, follows the philosophy of alignment as described in section 1 .
Previous works in this class simply optimize for some measurement of distributional discrepancy.
One popular measurement is the maximum mean discrepancy (MMD) – the distance between the mean of the two domains in some reproducing kernel hilbert space, where the kernel is chosen to maximize the distance (Bousmalis et al., 2016; Long et al., 2015; 2017).
Another is a learned adversarial discriminator trained to distinguish between the two domains (Ganin & Lempitsky, 2014).
As mentioned in section 1, these minimax optimization problems are widely known, both in theory and practice, to be very difﬁcult (Fedus et al., 2017; Duchi et al., 2016; Liang, 2017; Jin et al., 2019).
Since the optimization landscape is much more complex than in standard supervised learning, training often does not converge or converges to a bad local minimum (Goodfellow, 2016; Nagarajan & Kolter, 2017; Li et al., 2017), and requires carefully balancing the two sets of parameters (for minimization and maximization) so one does not dominate the other (Salimans et al., 2016; Neyshabur et al., 2017a).
To make minimax optimization easier, researchers have proposed numerous modiﬁcations to the loss function, network design, and training procedure (Arjovsky et al., 2017; Gulrajani et al., 2017;
Karras et al., 2017; Courty et al., 2017; Sun & Saenko, 2016; Shu et al., 2018; Sener et al., 2016).
Over the years, these modiﬁcations have yielded practical improvements on many standard benchmarks, but have also made the state-of-the-art algorithms very complicated.
Often practioners are not sure which tricks are necessary for which applications, and implementing these tricks can be bug-prone and frustrating.
To make matters worse, since there is no labeled target data available for a validation set, practioners no way to perform hyper-parameter tuning or early stopping.
The second class of methods directly transforms the source images to resemble the target images with generative models (Taigman et al., 2016; Hoffman et al., 2017; Bousmalis et al., 2017).
While similar to the ﬁrst class in the philosophy of alignment, these methods operate on image pixels instead of an intermediate representation space, and therefore can beneﬁt from an additional round of adaptation in some representation space.
In subsection 5.2 we demonstrate that composing our method with a popular pixel-level method yields stronger performance than either alone.
The third class of methods uses a model trained on the labeled source data to estimate labels on the target data, then trains on some of those estimated pseudo-labels (e.g. the most conﬁdent ones), therefore bootstrapping through the unlabled target data.
Sometimes called self-ensembling (French et al., 2017), this technique is borrowed from semi-supervised learning, where it is called co-training (Saito et al., 2017; Zou et al., 2018; Chen et al., 2018; 2011).
Note that our method uses joint training (of the main and self-supervised tasks), which is completely different from co-training in every aspect except the name.
While a few works study self-supervision in the context of semi-supervised learning, there is one work (Ghifary et al., 2016) superﬁcially similar to ours; in Appendix A we explain how our method differs in both philosophy and algorithm, and also achieves better results.
Self-supervision. Like in unsupervised learning, the goal of self-supervised learning is often to learn semantically useful features; this is achieved with a well designed self-supervised task that can only be solved with a good understanding of high-level visual concepts.
There are many successful designs of self-supervised tasks such as context prediction (Doersch et al., 2015), colorization (Zhang et al., 2016b), solving jigsaw puzzles (Noroozi & Favaro, 2016), noise prediction (Bojanowski & Joulin, 2017), feature clustering (Caron et al., 2018), and contrastive predictive coding (Oord et al., 2018), besides rotation prediction (Gidaris et al., 2018) as used in our method.
Doersch & Zisserman (2017) experiment with learning from multiple self-supervised tasks.
All these works use self-supervision as a pre-training step on unlabeled data (e.g. the ImageNet training set without labels) to initialize a deep learning model, before ﬁne-tuning on a labeled training set (e.g. PASCAL VOC) and evaluating on the corresponding test set.
Their setting, for which the source domain (ImageNet) is unlabeled and target (PASCAL) is labeled, is much easier and very different from ours, for which the source domain is labeled and target is unlabeled; therefore our algorithm is also very different 2 .
In Appendix A we explain in details three works using self-supervision, how their settings are different from ours, therefore also have different training algorithms and evaluations.
D ESIGNING SELF - SUPERVISED TASKS FOR ADAPTATION
[Table]
The design of self-supervised tasks is an exciting area of research by itself, with many successful examples covered in section 2.
However, not all of them are suitable for unsupervised domain adaptation.
In order to induce alignment between the source and target, the labels created by selfsupervision should not require information on the very factors where the domains are meaninglessly different, i.e. the factors of variation that we are trying to eliminate through adaptation.
To make our discussion concrete, here are the three used in our work, as examples of suitable tasks:
Rotation: (Gidaris et al., 2018)
An input image is rotated in 90-degree increments i.e. 0 ◦ , 90 ◦ , 180 ◦ , and 270 ◦ ; the task is to predict the angle of rotation as a four-way classiﬁcation problem.
Flip: An input image is randomly ﬂipped vertically; the task is to predict if it is ﬂipped or not 3 .
Location: Patches are randomly cropped out of an input image; the task is to predict where the patches come from 4 .
Consider a trivial illustrative example where the source and target are exactly the same except that the target pixels are all scaled down by a constant factor i.e. darker.
All three of the aforementioned forms of self-supervision are suitable for this example, because pixel scaling i.e. brightness is “orthogonal” to the prediction of rotation, ﬂip and location.
In contrast, a particularly unsuitable task would be reconstructing (maybe a corrupted version of) the original image, as is used by the denoising autoencoder (Vincent et al., 2008).
The success of pixel-wise reconstruction depends strongly on brightness information, or other factors of variation in overall appearance that are irrelevant to high-level visual concepts.
Thus, instead of inducing alignment, learning a task such as reconstruction would instead serve to further separate the domains.
For this reason, cross-channel prediction (Zhang et al., 2017), inpainting (Pathak et al., 2016) and colorization (Zhang et al., 2016b) are also unsuitable tasks for our particular training algorithm which implements the philosophy of alignment.
In general, classiﬁcation tasks that predict structural labels are better for our purpose than reconstruction tasks that predict pixels.
We have experimented with using the colorization task and the denoising autoencoder for our training algorithm, and found their performance little better than source only, sometimes even worse. It is interesting to note that works borrowing from semi-supervised learning, such as those discussed in Appendix A, can use the denoising autoencoder for their training algorithm and obtain performance better than source only (but still not competitive with ours). As explained in Appendix A, this difference in suitability of a particular form of self-supervision is due to the difference in training algorithms, namely we use self-supervision on both domains but they only use on the target. This further reﬂects the difference in philosophy, namely we use target data for alignment but they use simply as extra data like in semi-supervised learning.
The best known self-supervised tasks today were all originally designed for pre-training; they were evaluated in their respective papers in terms of accuracy gains on a downstream recognition, localization or detection task.
It will be interesting to see if new tasks arise from the motivation of adaptation, for which alignment is more important.
Moreover, domain experts can incorporate their dataset speciﬁc knowledge into the design of a self-supervised task speciﬁcally for their application.
[Figure]
M ETHOD
Our training algorithm is simple once a set of K self-supervised tasks are selected.
We already have the loss function on the main prediction task, denoted L 0 , that we do not have target labels for.
Each self-supervised task corresponds to a loss function L k for k = 1 ...K .
So altogether, our optimization problem is set up as a combination of these K + 1 loss functions, as is done for standard multi-task learning (see Figure 2).
Implementation details are included in Appendix B.
Each loss function corresponds to a different “head” h k for k =
K , which produces predictions in the respective label space.
All the task-speciﬁc heads (including h 0 for the actual prediction task) share a common feature extractor φ .
Altogether, the parameters of φ and h k , k = 0 , ..
k are the learned variables i.e. the free parameters of our optimization problem.
In our paper, φ is a deep convolutional neural network, and every h k is simply a linear layer i.e. multiplication by a matrix of size output space dimension × feature space dimension.
If k th task is classiﬁcation in nature, then h k also has a softmax or sigmoid (for multi-class or binary) following the linear layer.
The output space dimension is only four for rotation and location classiﬁcation, and two for ﬂip and location regression.
Depending on the network architecture used, the feature space dimension ranges between 64 and 512.
The point is to make every h k low capacity, so the heads are forced to share high-level features, as is desirable for inducing alignment.
A linear map from the highest-level features to the output is the smallest possible head and performs well empirically.
Let S =
{ ( x i , y i ) , i = 1 ...m } contain the labeled source data, and T = { ( x i ) , i = 1 ...n } contain the unlabeled target data.
L 0 for the main prediction task takes in the labeled source data, and produces the following term in our objective:
[Equation]
Each self-supervised task F k for k = 1 ...K modiﬁes the input samples with some transformation f k and creates labels ˜ y .
Denote F k ( S )
= { ( f k ( x i ) , ˜ y i ) , i = 1 ...m } as the self-supervised samples generated from the source samples (with the original labels discarded), and F k ( T ) =
{ ( f k ( x i ) , ˜ y i ) , i = 1 ...n } from the target.
Then the loss L k of each task k
= 1 ...
K produces the following term:
[Equation]
[Figure] Note that L k ( S, T ; φ, h k ) for k = 1 ...K , unlike L 0 ( S ; φ, h 0 ) , take in both the source and target data ; as we emphasize for many times throughout the paper, this is critical for inducing alignment.
Altogether, our optimization problem can be formalized as in multi-task learning 5 :
[Equation]
At test-time, we discard the self-supervised heads and use h 0 ( φ ( x )) .
4. A HEURISTIC FOR HYPER - PARAMETER TUNING AND EARLY STOPPING
As previously mentioned, in the unsupervised domain adaptation setting, there is no target label available and therefore no target validation set, so typical strategies for hyper-parameter tuning and early stopping that require a validation set cannot be applied.
This problem remains underappreciated; in fact, it is often unclear how previous works select their hyper-parameters or detemine when training is ﬁnished, both of which can be important factors that impact performance, especially for complex algorithms using adversarial learning.
In this subsection we describe a simple heuristic, merely as rule-of-thumb to make things work instead of a technical innovation.
Like almost all of deep learning, there is no statistical guarantee on this heuristic, but it is shown to be practically effective in our experiments (see Figure 3).
We only hope that it serves as the ﬁrst guess of a solution towards this underappreciated problem.
The main idea is that, because our method never explicitly optimizes for any measurement of distributional discrepancy, these measurements can instead be used for hyper-parameter tuning and early stopping.
Since it would be counterproductive to introduce additional parameters in order to perform hyper-parameter tuning, we simply use the distance between the mean of the source and target samples in the learned representation space, as produced by φ .
Formally, this can be expressed as
[Equation]
where S (cid:48) and T (cid:48) are unlabeled source and target validation sets 7 .
Appendix C contains some additional explanation of why the mean distance is suitable for our heuristic, even though it is a speciﬁc form of MMD,
which we [Table] Our heuristic combines D ( S (cid:48) , T (cid:48) ; φ ) and the main task error on the (labeled) source validation set.
Denote v = ( v 1 , ..., v T ) and w = ( w 1 , ..., w T ) the measurement vectors of those two quantities respectively over T epochs.
The ﬁnal measurement vector is u = v / min( v ) + w / min( w ) i.e. a normalized sum of the two vectors; the epoch at which we perform early stopping is then simply arg min t ∈{ 1 ...T } u t .
Intuitively, this heuristic roughly corresponds to our goal of inducing alignment while preserving discriminability.
E XPERIMENTS 5. S EVEN BENCHMARKS FOR OBJECT RECOGNITION
The seven benchmarks are based on the six datasets described in Appendix D, each with a predeﬁned training set / test set split, and labels are available on both splits.
Previous works (cited in Table 2) have created those seven benchmarks by picking pairs of datasets with the same label space, treating one as the source and the other as the target, and training on the training set of the source with labels revealed and of the target with labels hidden.
Following the standard setup of the ﬁeld, labels on the target test set should only be used for evaluation, not for hyper-parameter tuning or early stopping; therefore we apply the heuristic described in subsection 4.1.
For the two natural scene benchmarks, we use all three tasks described in section 3: rotation, location and ﬂip prediction.
For the ﬁve benchmarks on digits we do not use location because it yields trivial solutions that do not encourage the learning of semantic concepts.
Given the image of a digit cropped into the four quadrants, location prediction can be trivially solved by looking at the four corners where a white stroke determines the category.
Adding ﬂip to the digits does not hurt performance, but does not improve signiﬁcantly either, so we do not report those results separately.
As shown in Table 2, despite the simplicity of our method, we achieve state-of-the-art accuracy on four out of the seven benchmarks.
In addition, we show the source only results from our closest competitor (VADA and DIRT-T), and note that our source only results are in fact lower than theirs on those benchmarks that we perform the best on; this indicates that our success is indeed due to effectiveness in adaptation instead of the base architecture.
Our method fails on the pair of benchmarks with SVHN, on which rotation yields trivial solutions.
Because SVHN digits are cropped from house numbers with multiple digits, majority of the images
[Table]
s i
[Table]
a l
[Table]
f
[Table]
t
[Table]
t r a
[Table]
ﬁ
[Table]
s i
[Table]
t
[Table]
r
[Table]
our training algorithm further aligns the domains
have parts of the adjacent digits on the side. The main task head needs to look at the center, but the rotation head learns to look at the periphery and cheat. This failure case shows that the success of our method is tied to how well the self-supervised task ﬁts the application. Practioners should use their domain knowledge evaluate how well the task ﬁts, instead of blithely apply it to everything.
Also seen in Table 2 is that our method excels at object recognition in natural scenes, especially with all three tasks together.
For STL-10 → CIFAR-10, our base model is considerably worse than that of
VADA and DIRT-T, but still beats all the baselines.
Adding location and ﬂip gives an improvement of 8%, on top of the 8% already over source only.
This is not surprising since those tasks were originally developed for ImageNet pre-training i.e. object recognition in natural scenes – our method is very successful when the task ﬁts the application well.
5. B ENCHMARK FOR SEMANTIC SEGMENTATION
To experiment with our method in more diverse applications, we also evaluate on a challenging simulation-to-real benchmark for semantic segmentation – GTA5 → Cityscapes.
GTA5 (Richter et al., 2016) contains 24,966 video frames taken from the computer game, where dense segmentation labels are automatically given by the game engine.
Cityscapes (Cordts et al., 2016) contains 5, video frames taken from real-world dash-cams.
The main task is to classify every pixel in an image as one of the 19 classes shared across both datasets, and accuracy is measured by intersection over union (IoU).
The best possible results are given as the oracle, when the labels on Cityscapes are available for training, so typical supervised learning methods are applicable.
Our results are shown in Table 3, and implementation details in Appendix F.
Our self-supervised tasks were designed for classiﬁcation, where the label on an image depends on its global content, while in segmentation the labels tend to be highly local.
Nevertheless, with very little modiﬁcation, we see signiﬁcant improvements over the source only baseline.
We also experiment with combining our method with another popular unsupervised domain adaptation method – CyCADA (Hoffman et al., 2017), designed speciﬁcally for segmentation.
Surprisingly, when operating on top of images produced by this already very strong baseline, our method further improves performance.
This demonstrates that pixel-level adaptation methods might still beneﬁt from an additional round of adaptation by inducing alignment through self-supervision.
We emphasize that these results are obtained with a very simple instantiation of our method, as a start towards the development of self-supervised tasks more suitable for semantic segmentation.
F UTURE WORK
Through our paper, we hope to encourage future researchers in unsupervised domain adaptation to consider the study of self-supervision as an alternative to adversarial learning, and researchers in self-supervision to consider designing tasks and evaluating them in our problem setting.
One additional advantage of our method, not considered in this paper, is that it might be particularly amenable to very small target sample size, when those other methods based on adversarial learning cannot accurately estimate the target distribution; we leave this topic for the future.
A PPENDIX A A DDITIONAL DISCUSSION ON SPECIFIC PAPERS RELATED TO OURS
In this section we discuss the four papers mentioned in section 2 that are related to ours, but different in training algorithm, philosophy or problem setting.
Deep Reconstruction-Classiﬁcation Networks (Ghifary et al., 2016).
This method learns a denoising autoencoder (instead of, say, rotation prediction head) on the target, together with a labeling function on the source.
However, they use only the target domain for reconstruction, claiming both empirically and theoretically that it is better than using both domains together.
In contrast, we use both the source and target for self-supervision.
These two algorithmic differences really reﬂect our fundamental difference in philosophy.
They take the target data as analogous to the unlabeled (source) data in semi-supervised learning, so any form of self-supervision suitable for semi-supervised learning is good enough; their theoretical analysis is also directly borrowed from that of semi-supervised learning, which concludes that it is necessary and sufﬁcient to only use the target data for selfsupervision.
We take data from both domains for self-supervision in order to align their feature distributions; also, both conceptually and empirically, we cannot use reconstruction tasks because they are unsuitable for inducing alignment (see section 3).
Empirical experiments further support our arguments in three ways: 1) If we were to only use the target for our self-supervision tasks, we would perform barely better than the source only baseline, even on MNIST → MNIST-M, the easiest benchmark where we would otherwise observe a huge improvement.
If we were to use the reconstruction task i.e. a denoising autoencoder, we would again perform barely better than the source only baseline, and sometimes even worse, as described in section 3. 3)
As shown in Table 2, our results are much better than theirs on all of the benchmarks by a large margin.
This shows that implementing the philosophy of alignment, developed for unsupervised domain adaptation, is much superior in our problem setting to blithely borrowing from semi-supervised learning.
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty (Hendrycks et al., 2019).
This paper studies the setting of robustness, where there is no sample provided, even unlabeled, from the target domain.
Their method jointly trains for the main task and the self-supervised task only on the source domain , really because there is no target data of any kind.
Because their problem setting is very challenging (with so little information provided on the target), their primary evaluation is on a dataset of CIFAR-10 images with the addition of 15 types of common corruptions (e.g. impulse noise, JPEG compression, snow weather, motion blur and pixelation), simulated by an algorithm.
No idea on unsupervised domain adaptation is mentioned.
Domain Generalization by Solving Jigsaw Puzzles (Carlucci et al., 2019).
This paper studies two setting.
The ﬁrst is the robustness setting, exactly the same as in Hendrycks et al.
(2019), except that evaluation is done on MNIST → MNIST-M and MNIST → SVHN.
Their baseline is also a method from the robustness community that trains on adversarial examples and uses no target data.
Again, because their problem setting is very challenging, the accuracy is low for both their proposed method and the baseline.
The second setting is called domain generalization, which is very similar to meta-learning.
The goal is to perform well simultaenously on multiple distributions, all labeled.
Evaluation is done using the mean accuracy on all the domains.
Beside the name, there is little similarity between the setting of unsupervised domain adaptation and domain generalization, which has no unsupervised component.
Boosting Supervision with Self-Supervision for Few-shot Learning (Su et al., 2019).
As evident from the title, this paper studies the setting of few-shot learning, where the goal is to perform well on a very small dataset.
Again, there is no unsupervised component and little connection to our setting.
Most of our results have already been produced when Carlucci et al.
and Su et al.
(2019) were presented at a conference.
Hendrycks et al.
(2019) was submitted to NeurIPS 2019 and should be considered concurrent work.
In addition to the three classes of methods covered in section 2, unsupervised domain adaptation has also been studied using sample-to-sample correspondence through graph matching (Das & George
Lee, 2018; Das & Lee, 2018); these methods are less relevant to ours and not discussed in detail.
B A DDITIONAL ALGORITHMIC DETAILS
In practice and for our experiments, the source and target datasets are often imbalanced in size.
If we were to blithely solve for the objective in Equation 2, the domain with a larger dataset would carry more weight for every L k ( S, T ; φ, h k ) , k = 1 ...K because we are summing over all samples in both datasets.
We would like to keep the two sums inside L k roughly balanced such that in terms of features produced by φ , neither of the two domains would dominate the other.
This is easy to achieve in our implementation through balanced batches.
When we need to sample a batch for task k , we simply sample half the batch from the source, and another half from the target, then put them together.
In the end, our implementation requires little change on top of an existing supervised learning codebase.
Each self-supervised task is deﬁned as a module, and adding a new one only amounts to deﬁning the structural modiﬁcations and the loss function.
We optimize the objective in Equation 2 with stochastic gradient descent (SGD).
One can simply take a joint step on Equation 2, which covers all the losses for k = 0 and k = 1 ...K .
However, the implementation would then have to store the gradients with respect to all K + 1 losses together.
For memory efﬁciency, our implementation loops over k = 1 ...
K ; for each self-supervised task k , it samples a batch of combined source and target data (without their original labels), structurally modiﬁes the images by f k , creates new labels according to the modiﬁcation, and obtains a loss for h k and φ on this batch.
So a gradient step is taken for each k = 1 ...K , before ﬁnally a batch of original source images and labels are sampled for a gradient step on h 0 and φ .
In terms of test accuracy, these two implementations make little difference (usually less than 1%), and the choice simply comes down to a time-memory trade-off.
Because our source only baseline on STL-10 → CIFAR-10 is considerably worse than the closest competitor DIRT-T, we borrow from their implementation and use dropout regularization (Srivastava et al., 2014) with p = 0 .
5 , only on this benchmark.
This makes our source only baseline closer to theirs, so to allow for a fair comparison of the adaptation methods.
Without this modiﬁcation, our accuracy for source only, R, and R+L+F are respectively 56.1, 65.6 and 74.0%.
C A DDITIONAL DISCUSSION ON THE MEAN DISTANCE
In this section we answer some potential questions about our selection rule, which uses the mean distance, from the perspective of a possibly confused reader.
Q : The motivation of the paper is that methods based on minimax optimization, such as those using kernel-MMD, are difﬁcult to optimize.
Since you are using the mean distance (MMD under the linear kernel) for hyper-parameter tuning and early stopping, which is a form of model selection, how are you different from those other methods and why is optimization easy for you?
A : Our method is fundamentally different from those other methods based on MMD, and therefore more amenable to optimization in the following two ways: 1) Even though we use the mean distance, which is a form of MMD, we never pose a minimax optimization problem.
The model parameters minimize the loss functions of the tasks, which we hope makes the mean distance small (see Figure 3).
Model selection using subsection 4.1 also minimizes the mean distance.
In our method, all the loss functions, for model parameters and hyper-parameters, work towards the same goal of inducing alignment, so optimization is easier.
Even though hyper-parameter tuning and early stopping are a forms of model selection just like training, there is a qualitative difference in the degrees of freedom involved.
For subsection 4.1, when performing early stopping for example, optimization amounts to a grid search over the epochs after training is ﬁnished, and there is only one degree of freedom.
Q : Why is the mean distance suitable for model selection when it comes to hyper-parameter tuning and early stopping, but not regular training?
A : Again, we agree that hyper-parameter tuning and early stopping are a forms of model selection.
However, unlike model parameters ranging in the hundred of thousands, there are at most a few hyper-parameters and only one parameter for early stopping.
Model parameters can easily overﬁt to the mean distance while those few degrees of freedom we use cannot.
This is precisely also the reason why previous works using MMD resort to minimax optimization.
D D ETAILS OF THE SIX DATASETS USED FOR OBJECT RECOGNITION
1) MNIST (LeCun et al., 1998): greyscale images of handwritten digits 0 to 9; 60,000 samples in the training set and 10,000 in the test set.
2) MNIST-M (Ganin et al., 2016): constructed by blending MNIST digits with random color patches from the BSDS500 dataset Arbelaez et al.
(2011); same training / test set size as MNIST.
3) SVHN (Netzer et al., 2011): colored images of cropped out house numbers from Google Street View; the task is to classify the digit at the center; 73,257 samples in the training set, 26,032 in the test set and 531,131 easier samples for additional training.
4) USPS: greyscale images of handwritten digits only slightly different from MNIST; 7291 samples in the training set and 2007 in the test set.
5) CIFAR-10 (Krizhevsky & Hinton, 2009): colored images of 10 classes of centered natural scene objects; 50,000 samples in the training set and 10,000 in the test set.
6) STL-10 (Coates et al., 2011): colored images of objects only slightly different from CIFAR-10; 5000 samples in the training set and 8000 in the test set. Because CIFAR-10 and STL-10 differ in one class category, we follow common practice (Shu et al., 2018; French et
al. Ghifary et al., 2016) and delete the offending categories, so each of these
two datasets actually only has 9 classes E I MPLEMENTATION DETAILS ON THE OBJECT RECOGNITION BENCHMARKS
We use a 26-layer pre-activation ResNet (He et al., 2016) as our test-time model h 0 ( φ ( x )) , where his the last linear layer that makes the predictions, and φ is everything before that.
For unsupervised domain adaptation, there is no consensus on what base architecture to use among the previous works.
Our choice is simply base on the widespread adoption of the ResNet architecture and the ease of implementation.
In Table 2 we provide the source only results using our base architecture, and the ones from Shu et al.
(2018), our closest competitor.
Our source only results are in fact worse than theirs, indicating that our improvements are indeed made through adaptation.
At training time, the self-supervised heads h k , k = 1 ...
K are simply linear layers connected to the end of φ as discussed in section 4.
There is no other modiﬁcation on the standard ResNet.
For all experiments on the object recognition benchmarks, we optimize our model with SGD using weight decay 5e-4 and momentum 0.9, with a batch size of 128.
We use an initial learning rate of 0.1 and a two milestone schedule, where the learning rate drops by a factor of 10 at each milestone.
All these optimization hyper-parameters are taken directly from the standard literature (He et al., 2016;
Huang et al., 2016; Guo et al., 2017) without any modiﬁcation for our problem setting.
We select the total number of epochs and the two milestones based on convergence of the source classiﬁer h 0 and unsupervised classiﬁers h 1 , ..., h K .
Early stopping is done using the selection heuristic discussed in subsection 4.1.
For fair comparison with our baselines, we do not perform data augmentation, following previous works (Hoffman et al., 2017; Sener et al., 2016; Ghifary et al., 2016).
F I MPLEMENTATION DETAILS ON GTA5 → C ITYSCAPES
For our experiments, we initialize our model from the DeepLab-v3 architecture (Chen et al., 2017), pre-trained on ImageNet, as commonly done in the ﬁeld.
Each self-supervised head consists of a global average pooling layer on the pre-logit layer, followed by a single linear layer.
To take advantage of the large size of the natural scene images, we use the continuous i.e. regression version of location prediction.
The self-supervised head is trained on the square loss, to regress the coordinates (in two dimensions) that the patch is cropped from.
Natural for the regression version, instead of cropping from the quadrants like for the classiﬁcation version on the small datasets, we instead crop out 400 × 400 patches taken at random from the segmentation scenes.
We optimize our model with
SGD using a learning rate of 0.007 for 15,000 iterations, with a batch size of 48.
Once again, we use the selection heuristic in subsection 4.1 for early-stopping. [Table]
G Q UALITATIVE COMPARISON ON GTA5 → C ITYSCAPES
We thank the anonymous Reviewer 3 for suggeting us to add these visualizations.
This section shows segmentation results produced before and after adaptation(as shown in Table 4), alongside the original input image and the ground truth label.
Qualitatively, from visual inspection of these images, we can see that the quality of produced segmentations improves drastically after adaptation with our method.
In particular, we note that location prediction as a self-supervised task helps to prevent nonsensical label conﬁgurations, which are often produced by the baseline.
For example, the baseline model sometimes predicts “building” or “terrain” for pixels directly in front of a car.
These errors are largely corrected for by our proposed adaptation method, leading to more accurate “road” predictions.