A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Domain adaptation seeks to mitigate the shift between training on the source data and testing on the target data.
Most adaptation methods rely on the source data by joint optimization over source and target.
Source-free methods replace the source data with source parameters by ﬁne-tuning the model on target.
Either way, the majority of the parameter updates for the model representation and the classiﬁer are derived from the source, and not the target.
However, target accuracy is the goal, and so we argue for optimizing as much as possible on target.
We show signiﬁcant improvement by on-target adaptation , which learns the representation purely on target data, with only source predictions for supervision (without source data or parameter ﬁne-tuning).
In the long-tailed classiﬁcation setting, we demonstrate ontarget class distribution learning, which learns the (im)balance of classes on target data.
On-target adaptation achieves state-of-the-art accuracy and computational efﬁciency on VisDA-C and ImageNet-Sketch.
Learning more on target can deliver better models for target.
I NTRODUCTION
Deep networks achieve tremendous success on various visual tasks at the expense of massive data collection and annotation efforts.
Even more data is needed when training (source) and testing (target) data differ, as the model must be adapted on the new data to maintain accuracy.
To reduce the annotation effort on new data, unsupervised domain adaptation (UDA) approaches transfer knowledge from labeled source data to unlabeled target data.
Standard UDA requires simultaneous optimization on the source and target data to do so.
However, this requirement may not be entirely practical, in that shifted or future target data may not be available during training.
Furthermore, (re-)processing source data during testing may be limited by computation, bandwidth, and privacy.
Most importantly, it is the target data that ultimately matters for testing.
In this work, we therefore turn our attention from source to target, and how to learn more from it.
Recent work adapts to the target data without the source data or even adapts during testing.
However, these “source-free” and “test-time” approaches still rely heavily on the source parameters for ﬁnetuning.
Source-free adaptation initializes from source parameters then optimizes on target data without the joint use of source data (Liang et al., 2020; Li et al., 2020; Kundu et al., 2020).
Test-time adaptation partially updates source parameters on the target data while testing (Sun et al., 2019;
Schneider et al., 2020; Wang et al., 2021).
Such approaches reduce reliance on the source data, and can even improve accuracy, but have they made full use of the target data?
Many of the model parameters are ﬁxed (Liang et al., 2020; Schneider et al., 2020; Wang et al., 2021) or regularized toward the source parameters (Li et al., 2020;
Kundu et al., 2020).
We investigate whether more can be learned from target, and more accuracy gained, by not transferring the source parameters.
We propose on-target adaptation to unshackle the target representation from the source representation.
To do so, we (1) factorize the representation from the classiﬁer and (2) separate the source parameters from the source predictions.
By factorizing the representation from the classiﬁer, we can train the representation entirely on the target data by self-supervision.
Given this on-target representation, we can then supervise a new classiﬁer from source predictions by distillation (Hinton et al., 2015), without transferring the source parameters.
Not transferring parameters frees our target model from the constraints of the source architecture, so that we can experiment with distinct target architectures.
In this way, we can even change the model size to optimize a target-speciﬁc model that is more accurate and more efﬁcient.
In contrast to prior work on adaptation, this uniquely allows for learning 100% of the target model parameters on target data, as illustrated by Figure 1.
[Figure]
To realize our proposed factorization and separation, we employ contrastive learning, source-free adaptation, and teacher-student distillation.
We initialize the target representation by self-supervision with contrastive learning.
We turn the source model into a teacher model by source-free adaptation, and then generate pseudo-labels to supervise distillation.
We lastly train the student model on the teacher supervision, starting from the target representation and new classiﬁer parameters, and repeat this teacher-student cycle by resetting the student classiﬁer parameters between epochs.
Contrastive learning has recently enabled self-supervised representations to compete with or even surpass supervised representations (Chen & He, 2020; Caron et al., 2020;
He et al., 2020; Chen et al., 2020; Grill et al., 2020; Zbontar et al., 2021).
We show it provides a sufﬁcient target representation.
Our experiments show on-target adaptation achieves state-of-the-art accuracy and computational efﬁciency on common domain adaptation benchmarks.
For model accuracy, our method brings ∼ 3% absolute improvement compared to state-of-the-art unsupervised and source-free domain adaptation methods on VisDA-C
(Peng et al., 2017) and ImageNet Sketch (Wang et al., 2019a) while reducing 50%+ of parameters.
For computation, our method reduces FLOPs by 50+% and memory by 75+% for each forward pass of the target model.
In the long-tailed classiﬁcation setting, on-target class distribution learning equals the state-of-the-art learnable weight scaling (Kang et al., 2019) without needing source data.
Ablation experiments support the generality of on-target representation learning across architectures, contrastive learning methods, losses, and amount of optimization.
Our contribution is to investigate whether the source data should be the primary source of target model parameters, and to propose an alternative: on-target adaptation.
Our insight is that the source representation can be fully decoupled from source supervision.
Domain adaptation normally emphasizes the representation of source data, by either jointly optimizing on source data or transferring source parameters.
On-target adaptation emphasizes the representation of target data instead, by distilling source predictions into a self-supervised target representation.
We are the ﬁrst to show this is feasible, as a new kind of source-free adaptation.
Furthermore we show it improves accuracy and reduces computation on standard benchmarks like VisDA-C.
R ELATED WORK
Adaptation On-target adaptation is unique in its decoupling of the target representation from the source representation.
Prior adaptation approaches transfer the source representation to the target, either by joint optimization or by initialization.
To transfer the source model to a visually different target domain, unsupervised domain adaptation (UDA) learns a joint representation for both domains for visual recognition tasks, such as image classiﬁcation (Tzeng et al., 2014), object detection (Chen et al., 2018), semantic segmentation (Hoffman et al., 2016).
Some of the most representative unsupervised domain adaptation ideas are 1) maximum mean discrepancy (Long et al., 2015; 2017); 2) moment/correlation matching (Sun et al., 2016; Zellinger et al., 2017); 3) domain confusion (Ganin & Lempitsky, 2015; Tzeng et al., 2017); 4) GAN-based alignment (Liu et al., 2017; Hoffman et al., 2018).
All these UDA methods need simultaneous access to both source and target data.
In practice, it might be impossible to meet this requirement due to limited bandwidth, computational power, or privacy concerns.
Therefore, test-time training (Sun et al., 2019), source-free adaptation (Liang et al., 2020), and fully test-time adaptation (Wang et al., 2021) settings focus on adapting a source model by ﬁne-tuning on the target data without source data. Exciting concurrent work even adapts without the
[Table]
[Table] [Table]
+SlRwnWngcaTgKro9rvSrH6CNTJMLLDIIYz5J5FgKjo4atiGYnTHQkyi0ftffqbsMEH6i/Z4aU24FU462KG9wh9o5v+8zxqqAzw72q+ThYW+X9crixgaZljFQLMvtYbsz87qibwGrQYfUdTZs3wWjVOQxJCgUN2bA/AxDyzVKoaBsBbmBjItbPoGBgwmPwYR2NlpJvzlmRMepdp0gnbEvE5bHxhRx5Jwxx6mZ1yryPW2Q47gXWplkOUIini8a54piSqvd0pHUIFAVDnChpZuViinXXKD7gZZbApt/8ltwudtl+112vtc5OqnX0SQbZJNsEUYOyBE5JWekTwT5RR7IX/LP++398R69/8/Whldn1smrajSfALbCsIU=</latexit> Loss(ˆ y t ,y 0 t ) [Table] [Table]
These “black-box” adaptation methods exclusively optimize teacher and student predictions, with distillation losses and output regularizers.
While we likewise apply teacher-student learning, our work is complementary in using contrastive learning as a loss on the input for the student representation.
Semi-supervised learning Many UDA methods follow the practice of semi-supervised learning, especially pseudo labeling (Lee, 2013) which is to utilize the model prediction to generate supervision for the unlabeled images.
The typical setup of unsupervised domain adaptation methods is to jointly optimize with ground truths on the source and pseudo labels on the target (Zhang et al., 2018; Choi et al., 2019;
Long et al., 2017; Zou et al., 2018).
When source data annotations are not available,
DeepCluster (Caron et al., 2018) and SHOT (Liang et al., 2020) further leverage weighted k-means clustering to reduce the side effects on noisy pseudo labels.
Similarly, our method does not require access to labeled source data, while only relying on the target images with generated pseudo labels.
In addition, our method heavily beneﬁts from the contrastive learned target domain representation, which is treated as initialization to overcome the misleading of noisy pseudo labels.
Long-tailed recognition Long-tailed recognition tackles imbalanced class distributions in realworld data.
Existing work divides into three groups: 1) re-balancing the data distribution (Chawla et al., 2002; Han et al., 2005;
Shen et al., 2016; Mahajan et al., 2018); 2) designing class-balanced losses (Cui et al., 2019; Khan et al., 2017; Cao et al., 2019; Khan et al., 2019; Huang et al., 2019;
Lin et al., 2017; Shu et al., 2019; Ren et al., 2018; Hayat et al., 2019); 3) transfer learning across classes (Yin et al., 2019;
Liu et al., 2019).
All of these methods address imbalance by altering training, so that the model may learn more balanced features, and a classiﬁer that covers common (head) and rare (tail) classes.
We instead adapt the classiﬁer for long-tailed recognition during testing.
M ETHOD : ON - TARGET ADAPTATION
The goal of the proposed on-target adaptation is to tackle domain shift during the test-time with only a source model, without the access of annotation and source data.
Speciﬁcally, the supervised model with source parameter f ( · ; θ s ) trained on source images x s and labels y s needs to generalize on unlabeled target data x t when an unneglectable domain shift happened.
Our on-target adaptation (Figure 2) is proposed to obtain target model parameter θ t purely during test-time.
Stage 0 (source): train model with labeled source data We train a deep ConvNet and learn source parameter θ s by minimizing vanilla cross-entropy loss L (ˆ y s , y s ) on labeled source data ( x s , y s ) .
Speciﬁcally, L (ˆ y s , y s )
= − Σ c p ( y sc ) log( p (ˆ y sc )) for the predicted probability ˆ y sc of class c , where target probability y sgt is 1 for the ground truth class gt and 0 for the rest.
Stage 1 (teacher): adapt source model without source data We update the source parameter θ s during testing to minimize information maximization (InfoMax) loss (Gomes et al., 2010).
Specifically, InfoMax loss augment entropy loss L ent = − Σ c p (ˆ y tc ) log( p (ˆ y tc ) with diversity objective
L div = D KL (ˆ y t || 1 C 1 C ) − log( C ) .
where D KL indicates the Kullback–Leibler divergence, 1 C is an all-one vector with C dimensions.
Here 1 C 1 C indicates the target label vector with evenly distributed 1 C probabilities, where L div is propose to enforce the global diversity over classes. [Figure]
As for the parameters to optimize over, we follow the motivation of decoupling the representation and classiﬁer.
When the classiﬁer is frozen, the goal of optimization is to mitigate domain shift by deriving proper target features from the source model.
In particular, we keep the classiﬁer the same on both source and target domain, and obtain ∆ by the gradient of the test-time objective (InfoMax), to update the representation part of model parameter θ s .
Stage 2 (student): initialize target model with contrastive learning Instead of ﬁne-tuning from source model, we choose to initialize the target feature purely from target data.
Beneﬁting from the recent advances in contrastive learning methods, we train an unsupervised model with purely unlabeled target images.
Speciﬁcally, we initialized target representation via improved momentum contrast learning (MoCo v2)
(He et al., 2020; Chen et al., 2020).
It is worth noting that our method does not require a speciﬁc contrastive learning method.
In other words, the default MoCo v2 could be easily replaced by a more recent self-supervised learning model, such as SwAV (Caron et al., 2020), SimSiam (Chen & He, 2020), Barlow Twins (Zbontar et al., 2021).
Such a modular design makes it easier to beneﬁt from the latest advance in contrastive learning.
We have performed an ablation study on the choice of contrastive learning method in Section 4.5.
Stage 3 (teacher-student): transfer knowledge from teacher to student We use the adapted source model f ( · ; θ s + ∆) as the initial teacher model to generate pseudo labels y (cid:48) t on unannotated target images x t .
Then we ﬁne-tune the student model f ( · ; θ t ) initialized by contrastive learning on target data with cross-entropy loss L (ˆ y t , y (cid:48) t ) =
− Σ c p ( y (cid:48) tc ) log( p (ˆ y tc )) .
The teacher would be replaced with the latest student to gradually denoise pseudo labels for the subsequent phase.
Meanwhile, the contrastive learned model would re-initialize the student feature to eliminate the accumulated errors from imperfect pseudo labels.
In other words, the student model would start over one more time with only newer pseudo labels for the next transferring phase.
Figure 3 illustrates the procedure of transferring the knowledge from teacher to student.
Speciﬁcally, the interaction between teacher and student models beneﬁts from consistency regularization and pseudo-labeling, inspired by a recent semi-supervised learning approach called FixMatch (Sohn et al., 2020).
During the transferring, we augment the target images with random cropping, random ﬂipping, and AutoAugment with ImageNet policy, as “strong” augmentation, while the “weak” augmentation is the combination of resizing and center cropping when generating pseudo labels.
Relying on the assumption that the model should generate similar predictions on data-augmented versions of the same image (Bachman et al., 2014; Sajjadi et al., 2016; Laine & Aila, 2016), consistency regularization enforces the cross-entropy loss between student output on strongly-augmented images and teacher output on weakly-augmented images.
E XPERIMENTS 4. S ETUP
Datasets We evaluate our method on both domain adaptation and long-tailed recognition benchmarks, including VisDA-C
(Peng et al., 2017), Ofﬁce Home (Venkateswara et al., 2017), Sketch (Wang et al., 2019a), ImageNet-LT (Liu et al., 2019), and iNaturalist-18 (Van Horn et al., 2018).
Figure 5 presents some of example images to illustrate domain shifts.
Metric We report top-1 accuracy (denoted as acc.)
on the whole dataset for all datasets.
On VisDA-C, we additionally report the percentage accuracy of each category and the corresponding average of all categorical accuracies (denoted as avg.), due to the imbalanced distributed label space.
On Ofﬁce
Home, we calculate the average for all kinds of domain shifts as the summary number for each method.
On long-tailed recognition benchmarks, we additionally report percentage accuracy on many-shot (more than 100 samples), medium-shot (20-100 samples), and few-shot (less thansamples) following the evaluation protocol from Liu et al.
; Kang et al. Baselines We choose the most recent fully test-time adaptation method, TENT (Wang et al., 2021), and source-free adaptation framework, SHOT (Liang et al., 2020), as our “online” and “ofﬂine” adaptation baselines.
TENT does not alter training, while SHOT minimally customizes the source architecture and training.
Entropy minimization is the target optimization objective for both TENT and SHOT.
SHOT additionally regularizes optimization by the information maximization (InfoMax) loss (Gomes et al., 2010; Shi & Sha, 2012;
Hu et al., 2017) to augment entropy minimization on each sample with diversity maximization across samples.
Following SHOT, we therefore augment TENT into TENT-IM by including this regularization.
Alongside their role as baselines, these methods can serve as teacher models for our stage 1.
We also compare with unsupervised domain adaptation (UDA) baselines, including DANN (Ganin & Lempitsky, 2015), DAN (Long et al., 2015), ADR (Saito et al., 2018), CDAN+E (Long et al., 2018), CDAN+BSP
(Chen et al., 2019), CDAN+TN (Wang et al., 2019b), SAFN (Xu et al., 2019), SWD (Lee et al., 2019), DSBN+MSTN (Chang et al., 2019), STAR (Lu et al., 2020).
It is worth noting that all these UDA methods are ﬁne-tuning from the ImageNet pretrained ResNet-101 source model, with access to both source and target data.
TENT-IM and SHOT likewise initialize their representation from the source model.
In contrast, our method trains ResNet-18 from scratch, and entirely on the target data, by contrastive learning for the representation and teacher-student learning for the classiﬁcation.
For long-tailed recognition, we choose learnable weight scaling (LWS)
(Kang et al., 2019) as the train-time baseline.
LWS ﬁrst decouples the full model into representation and classiﬁcation, and then only re-scales the classiﬁer parameters with class-balanced sampling.
Our on-target class distribution learning extends LWS to test-time, by re-scaling the parameters on unlabeled target data.
Architecture For comparability with state-of-the-art models, we choose 18/50/101-layer ResNet models (He et al., 2016) for both main results and ablation studies.
When reproducing the prior works, we keep the architecture the same, for example, the weight-normalization (Salimans & Kingma, 2016) augmented ImageNet pretrained ResNet-101 for SHOT (Liang et al., 2020).
4. I MPLEMENTATION
Our implementation is in PyTorch (Paszke et al., 2019) and depends on the VISSL (Goyal et al., 2021), MMClassiﬁcation (Contributors, 2020), and Weights & Biases (Biewald, 2020) libraries.
The code is attached and will be released for publication.
Stage 0 (source) We train residual networks (He et al., 2016) with various depths (including 18, 50, 101), and initializations (ImageNet pretraining or Kaiming init (He et al., 2015) when training from scratch).
We optimize cross-entropy loss by SGD with an initial learning rate 0.1, momentum 0.9, weight decay 0.0001, batch size 256.
We do not apply label smoothing (Müller et al., 2019) except for SHOT (Liang et al., 2020), as it speciﬁcally includes it.
We adopt the standard data augmentation pipeline from ImageNet training, such as random cropping, random ﬂipping, and color jitter.
We choose ImageNet statistics as the default input mean and variance for all models.
Stage 1 (teacher)
We experiment with three types of teacher models: 1) source-only, 2) TENTIM (Wang et al., 2021), 3) SHOT (Liang et al., 2020).
To optimize this altered loss, we choose SGD with learning rate 0.0001, momentum 0.9, and weight decay 0.0001.
In addition to batch normalization (Ioffe & Szegedy, 2015), we also update convolutional layers except the ﬁnal classiﬁcation layer.
As for SHOT, We execute the authors’ open-sourced codebase with the same hyper-parameters for various architectures (ResNet-50 and ResNet-101), initialization (from scratch and ImageNet pretrain), and domain shifts (train to val/test splits on VisDA-C).
Stage 2 (student) We experiment with two designs as students: 1) source-only, 2) contrastive learning.
Speciﬁcally, we leverage some of off-the-shelf contrastive learning methods to initialize target-domain representation, such as MoCo v2 (Chen et al., 2020), SimSiam (Chen & He, 2020), SwAV (Caron et al., 2020), Barlow Twins (Zbontar et al., 2021).
Compared to their training recipes on ImageNet, we have more epochs on VisDA-C val/test with the same batch size, learning rate, data augmentation, and model architecture, to make the training procedure longer with the smaller amount of images. [Figure] [Figure] [Figure]
[Table] [Figure] [Table] Stage 3 (teacher-student)
By default, the whole knowledge distillation consists of three phases, where each phase has 10 epochs to train the student model with the hard pseudo label.
The student would be reset to the contrastive model to avoid error accumulation at the beginning of every phase.
The teacher would be replaced with the latest student before starting the next phase, so that the quality of pseudo-labeling could be improved gradually.
We utilize SGD with an initial learning rate 0.01, momentum 0.9, weight decay 0.0001, batch size 256, and cosine annealing scheduler (Loshchilov & Hutter, 2016).
4. O N - TARGET ADAPTATION
Table 1 reports reports the change in target accuracy for each stage of on-target adaptation on VisDAC.
The source model is ResNet-50 trained from scratch. The target model is ResNet-18.
Source-free adaptation is done by TENT-IM. In teacher-student learning, the student, with either the source (teacher) or our on-target (contrastive) representation, is trained on the predictions of the teacher. This is repeated for multiple epochs.
The on-target representation (stage #2) improves accuracy with and without source-free adaptation of the teacher, as it only depends on the target data.
VisDA train → val Table 2 compares our method with state-of-the-art unsupervised (upper part) and test-time (lower part) domain adaptation approaches from VisDA-C train to val splits.
The proposed on-target adaptation signiﬁcantly improves the existing test-time adaptation methods.
It is worth noting that all these existing methods need to keep the same architecture when joint-training or ﬁnetuning on the source model.
On the contrary, our method could utilize a much more lightweight model, such as ResNet-18 as shown in this table.
For example, our method brings 18+ points improvement compared to source-free adapted teacher TENT-IM, while reducing over 50% parameters and runtime ﬂops, and 75% memory consumption at each feed-forward of target model.
VisDA train → test Table 3 compares our method with state-of-the-art unsupervised (upper part) and test-time (lower part) domain adaptation approaches from VisDA-C train to test splits.
Similar to table 2, our method dramatically improves the performance of all kinds of teacher models.
In the following two paragraphs, we discuss the potential application of on-target adaptation without contrastive learning .
When test-time data is not sufﬁcient enough to ﬁnish the contrastive learning, we could skip contrastive learning on target data (stage 2) of the proposed method. In other words, [Table]
[Table] [Table] we directly ﬁne-tune the target model initialized by the source model.
We believe that adaptation performance could be further improved once the contrastive learning could no longer be data-hungry or target domain data could be abundant.
ImageNet → Sketch Table 4 reports the empirical results on generalization regarding ImageNet/Sketch as source/target domain.
For on-target adaptation, we try two student models: model as same as the teacher model (ResNet-50) and small supervised model pretrained on ImageNet (ResNet-18).
Our method additionally brings ∼ 5/3% improvements compared to the teacher model with the same/shallower student models, where the teacher model, TENT-IM, already outperforms the previous state-of-the-art by ∼ 5%.
Ofﬁce Home Table 5 compares our method with state-of-the-art unsupervised (upper part) and test-time (lower part) domain adaptation approaches for the various domain shifts in Ofﬁce Home.
Our method advances the average accuracy over all domain shifts of teacher model (TENT-IM) by 3%.
4. O N - TARGET CLASS DISTRIBUTION LEARNING
In this section, we argue for calibrating classiﬁer during the test-time, without any modiﬁcation on training procedure, aiming at long-tailed recognition task.
Here we treat the long-tailed data as the source domain while regarding the class-balanced data as the target domain.
During training, instance-balanced sampling provides a generalizable representation to start with.
Then the classiﬁer is re-scaled during the test-time on the class-balanced data.
First, we train the source domain model with the instance-balanced sampling, which samples each sample with the same probability.
In this way, the learned classiﬁer has a higher prior probability on the head compared to the tail.
Then we calibrate the parameters of the classiﬁer while freezing the feature with test images and pseudo labels, following the practice of our on-target adaptation.
It is worth mentioning that we do not utilize contrastive learning to re-initialize the representation on target data or tune the feature part in teacher-student (stage 3).
The major reason for such a choice is to follow the practice of LWS (Kang et al., 2019), which points out that the domain shift only exists within class distribution so that only classiﬁer needs to be calibrated.
The empirical results indicate that our method could automatically calibrate the categorical prior and adaptive ﬁt the test data distribution without the access of training data.
Table 6 demonstrate that our test-time on-target adaptation could achieve comparable performance compared to stateof-the-art training-time methods on ImageNet-LT and iNaturalist-18 datasets.
We report results for two popular ConvNets, ResNet-50 and ResNet-101, as teacher models trained on long-tailed data.
Our method achieves comparable overall performance with the train-time method (LWS) on both datasets.
Compared to the vanilla ResNet-50 and ResNet-101, our fully test-time method signiﬁcantly improves the overall performance by a large margin.
Considering the accuracy of few-shot (less than [Table] [Table] 20 samples) categories, our method outperforms the train-time practice on three out of four cases, extending the usage scenarios of train-time long-tailed recognition methods.
4. A BLATION S TUDIES
Stage 0: network & initialization
The upper part of Table 7 presents the numbers of SHOT/TENTIM with ResNet in various depths (18, 50, 101) and initializations (from scratch, ImageNet pretrain).
We observe that our method consistently improves the teacher models with various model architectures and initializations, which indicates the usability and versatility of the proposed framework.
When adapting from synthetic to real-world domains, the ImageNet pretrained model should not be utilized to start with, due to the learned inductive bias of its parameters.
Therefore we also experiment on training from scratch for teacher models.
Larger capacity does not lead to better generalization when training from scratch.
On the contrary, We observe that a deeper ImageNet pretrained ConvNet provides a stronger inductive bias from the beginning.
Stage 1: test-time adaptation The lower part of Table 7 presents the numbers of source-only models as teacher, without any source-free adaptation (TENT-IM/SHOT).
The ﬁnal accuracy after teacher-student suffers from the poorer quality of the initial pseudo label.
ImageNet pretraining could alleviate such a phenomenon, but the numbers with test-time adapted teachers are still signiﬁcantly better than the source-only ones.
Comparing these empirical results with table 2, TENT-IM boosts the initial/ﬁnal accuracy by 15.0/28.6 points, while SHOT brings 32.4/25.9 points improvement.
In a word, test-time adaptation should be leveraged in preparation for trustworthy pseudo labels.
Stage 2: on-target feature The left part of Table 8 presents the ablation study of student architecture and initialization.
When the student model is not initialized on target data, such as source data (VisDA-C train) or the external large dataset (ImageNet), the overall accuracy drops 4+ points, which indicates the necessity of on-target feature learning.
We also ablate the same contrastive learning algorithm (MoCo v2) on a different data source, such as VisDA-C val and ImageNet.
The empirical results indicate that more external data does not bring any advantage, which echos our statement on the target-speciﬁc representation learning.
Stage 2: contrastive learning The right part of Table 8 presents the results with various contrastive learning frameworks, including SwAV, SimSiam, Barlow Twins.
We train these contrastive learned models with the same number of epochs to have a fair comparison.
Compared to the performance of the teacher model, all these learned features bring a noticeable improvement, achieving the comparable numbers with the reference performance of MoCo v2. We observe that our method is not
[Table]
VisDA train
VisDA
[Table]
VisDA test
ImageNet
Sketch
Office Home
[Table]
contrastive learning method.
In this way, our on-target adaptation could be further improved by introducing a more advanced contrastive learning approach in the future.
Stage 3: more phases Table 9 presents the detailed numbers for each phase during teacher-student.
We observe that the ﬁrst phase already signiﬁcantly outperforms the test-time adapted teacher model, which is also the state-of-the-art practice.
The following several phases gradually improve the results, taking the last generation student as the next teacher.
Considering the speed-accuracy trade-off, we choose to have three phases as our default setup, even though more phases could lead to a better result.
For example, 9-phase (3 × ) optimization brings up around 9 points improvement compared to our default 3-phase (1 × ) one with TENT-IM as the initial teacher model.
Stage 3: soft label Table 9 presents the ablation study on the design choice of loss function.
Our default setup only utilizes hard labels with cross-entropy loss.
Actually, our framework also beneﬁts from the soft label with Kullback–Leibler divergence loss, following the popular practice of knowledge distillation (Hinton et al., 2015).
We observe that the mix of both hard and soft label bring up the best performance.
we replace the cross-entropy loss (hard label) with Kullback–Leibler divergence (soft label) for the even number of phases.
We set the number of epochs as one for all these interpolated soft label phases for a more computational-friendly practice.
The known drawback is that the soft label part typically needs speciﬁc tuning on learning rate, loss weight, temperature, and so on.
Existing works (Berthelot et al., 2019b;a; Sohn et al., 2020) discuss sharpening (temperature) and thresholding (conﬁdence threshold) to improve the performance of semi-supervised learning.
Instead, we only ablate the default Kullback–Leibler divergence loss without bells and whistles like temperature and conﬁdence threshold.
Our default training objective chooses to be the most robust hard label with cross-entropy criterion for all the other experiments.
C ONCLUSION
Domain adaptation is itself adapted to many different needs: unsupervised domain adaptation jointly optimizes over labeled source and unlabeled target data, source-free adaptation adapts to target given source parameters instead of source data, and test-time adaptation even adapts while making predictions.
Across each of these varieties, the source comes ﬁrst.
The target representation is either aligned to the source representation or it is initialized from it by transfer learning.
On-target adaptation departs from this standard practice by transferring the source predictions without the source representation.
This decoupling is unconventional, but useful, because it enables learning all of the target model parameters on the target data.
Given enough target data, on-target adaptation improves accuracy by learning the model for target data on target data.