A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
Abstract
Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones.
In this work, we consider the Partial
Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain.
Most successful algorithms use model selection strategies that rely on target labels to find the best hyper-parameters and/or models along training.
However, these strategies violate the main assumption in PDA: only unlabeled target domain samples are available.
The main goal of this work is to provide a realistic evaluation of PDA methods with the different model selection strategies under a consistent evaluation protocol.
We evaluate 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies.
Our two main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points; (ii) only one method and model selection pair performs reasonably well on both datasets.
Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.
Introduction
Domain adaptation.
Deep neural networks are highly successful in image recognition for indistribution samples (He et al., 2016).
However, they tend to not generalize as well on images with different background or colors not seen during training.
Unfortunately, enriching the training set with new samples from different domains is challenging as labeling data is both an expensive and time-consuming task.
Thus, researchers have focused on unsupervised domain adaptation (UDA) where we have access to unlabelled samples from a different domain, known as the target domain.
The goal of UDA is to classify these unlabeled samples by leveraging the knowledge given by the labeled samples from the source domain (Pan & Yang, 2010).
In this work, we consider a variant called partial domain adaptation (PDA) where the target domain classes form a subset of the source domain classes (Cao et al., 2018)
The extra source classes, not present in the target domain, make the
PDA problem more difficult: simply aligning the source and target domains forces a negative transfer where target samples are matched to outlier source-only labels.
Realistic evaluations.
Most recent PDA methods report an increase of the target accuracy up topercentage points on average when compared to the baseline approach that uses only source domain samples.
However target labels are used for model selection, violating the main UDA assumption.
In their absence, the effectiveness of PDA methods remains unclear and model selection constitutes a yet to be solved problem as we show in this work.
In this work, we conduct extensive experiments to study the impact of model selection strategies on the performance of partial domain adaptation methods. We evaluate 7 different PDA methods with 7 different model selection strategies ondifferent datasets under the same experimental protocol for fair comparison.
We list below our major findings:
• The accuracy attained by models selected without target labels can decrease up to 30 percentage points compared to the one reported using target labels.
• Only 1 pair of PDA methods and target label-free model selection strategies achieve comparable accuracies to when target labels are used, while still improving over a source only baseline. • Random seed plays an important role in the selection of hyper-parameters.
Selected parameters are not stable across different seeds and the standard deviation between accuracies on the same task can be up to 8 .
4% even when relying on target labels for model selection.
• Under a more realistic scenario where some target labels are available, 100 random samples only leads to a drop of 1 percentage point in accuracy (when compared to using all target samples).
Outline.
The paper is organized as follows.
In Section 2 we describe the training procedures and evaluation protocols used to evaluate all methods fairly.
In Section 3, we discuss the results of the different benchmarked methods and the performance of the different model selection strategies.
Finally in Section 4, we give some recommendations for future work in partial domain adaptation.
Experimental Protocol
In this section, we discuss our choices regarding the training details, datasets and neural network architecture.
To perform our experiments we developed and openly released a PyTorch (Paszke et al., 2019) framework named BenchmarkPDA : https://github.com/oberman-lab/BenchmarkPDA
It is the standard in the literature when proposing a new method to report directly the results of its competitors from the original papers (Cao et al., 2018; Xu et al., 2019).
In addition, some methods also differ in the network architecture implementation (Xu et al., 2019; Gu et al., 2021) and other experimental details leading to unfair comparisons between the different methods.
Therefore we chose to implement all methods with the same commonly used neural network architecture, optimizer, learning rate schedule and evaluation protocol.
We discuss the details below. 2.1 Methods, Datasets, Training and Evaluation Details
Model Selection Strategies.
We consider 7 different strategies: S - ACC , DEV , ENT , SND , ORACLE , 1- SHOT , 100- RND , described in detail
Appendix A. We use them both for hyper-parameter tuning and selecting the best model along training.
Implementation details are discussed in Appendix C.2.
Methods.
We implemented 7 PDA methods by adapting the code from the Official GitHub repositories of each method: Source Only, PADA (Cao et al., 2018), SAFN (Xu et al., 2019), BA 3 US (Jian et al., 2020), AR (Gu et al., 2021), JUMBOT (Fatras et al., 2021b), MPOT (Nguyen et al., 2022).
We give an overview of the methods in Appendix B and provide the links to the repositories in Appendix C.1.
Datasets.
Our first dataset is OFFICE - HOME (Venkateswara et al., 2017).
It has four domains: Art (A), Clipart (C), Product (P) and Real-World (R) with 65 object categories in each of them.
The second dataset is VISDA (Peng et al., 2017) with synthetic images as source domain and real-world images as target domain.
The domains have 12 object categories.
For the partial setting, we follow
Cao et al.
and select the first, in alphabetical order, 25 and 6 categories for OFFICE - HOME and
VISDA , respectively, in each partial target domain.
Architecture.
Our network is composed of a feature extractor with a linear classification layer on top of it.
The feature extractor is a ResNet50 (He et al., 2016), pre-trained on ImageNet (Deng et al., 2009), with its last linear layer removed and replaced by a linear bottleneck layer of dimension 256.
Optimizer.
We use the SGD (Robbins & Monro, 1951) with momentum of 0.9, a weight decay of 5 e − 4 and Nesterov acceleration.
As the bottleneck and classifer layers are randomly initialized, we set their learning rates to be 10 times that of the pre-trained ResNet50 backbone.
We schedule the learning rate with a strategy similar to the one in Ganin et al.
(2016): χ p = χ 0 (1+ µi ) − ν , where i is the current iteration, χ 0 = 0 .
001 , γ = 0 .
001 , ν = 0 .
See Appendix C.4 for more details.
Evaluation Protocol.
For the hyper-parameters chosen with each model selection strategy, we run the methods for each task 3 times, each with a different seed (2020, 2021, 2022).
We tried to control for the randomness across methods by setting the random seeds at the beginning of training.
[Table]
Partial domain adaptation experiments
We start the results section by comparing the performance of the different model selection strategies.
Then, we discuss the sensitivity of methods to the random seed.
2547 models were run for this study.
3.1 Results for Model Selection Strategies
Model Selection Strategies (w/ vs w/o target labels)
All average accuracies on the OFFICE - HOME and VISDA datasets can be found in Table 1.
For all methods on OFFICE - HOME , we can see that the results for model selections strategies which do not use target labels are below the results given by ORACLE .
For some pairs, the drop of performance can be significant, leading some methods to perform on par or worse than the S .
ONLY method.
That is the case on OFFICE - HOME when DEV is paired with either BA 3 US , JUMBOT and MPOT .
Even worse is MPOT with SND as the average accuracy is more than 10 percentage points below that of S .
ONLY with any model selection strategy.
Overall on OFFICE - HOME , except for MPOT , all methods when paired with either ENT or SND give results that are at most 2 percentage points below compared to when paired with ORACLE .
A similar situation can be seen over the VISDA dataset where the accuracy without target labels can be down to 25 percentage points.
Yet again, some model selection strategies can lead to scores even worse than S .
ONLY .
That is the case for PADA , SAFN and BA 3 US .
Contrary to OFFICE - HOME , all model selection strategies without target labels lead to at least one method with results on par or worse in comparison to the S .
ONLY method.
Overall, no model selection strategy without target labels can lead to score on par to the ORACLE model selection strategy.
Finally, PADA performs worse than S .
ONLY for most model selection strategies, including the ones which use target labels.
However, when combined with SND it performs better than with ORACLE on average, although still within the standard deviation.
This is a consequence of the random seed dependence, discussed in detail below: as the hyper-parameters were chosen by performing just one run, we were “unlucky”.
In general, it confirms the assumption in the literature about the difficulty of the VISDA dataset.
Model Selection Strategies (w/ target labels)
We recall that the ORACLE model selection strategy uses all the target samples to compute the accuracy.
1- SHOT and 100-
RND use only subsets of target samples:
1- SHOT has only one sample per class for a total of 25 and 6 on OFFICE - HOME and VISDA , respetively, while 100- RND has 100 random target samples.
Our results show that using onlyrandom target labeled samples is enough to reasonably approximate the target accuracy leading to only a small accuracy drop (one percentage point in almost all cases) for both datasets.
Not surprisingly, the gap between the 1- SHOT and ORACLE model selection strategies is even bigger, leading in some instances to worse results than with a model selection strategy that uses no target labels.
This poor performance of the 1- SHOT model selection strategy also highlights that semi-supervised domain adaptation (SSDA) methods are not a straightforward alternative to the 100- RND model selection strategy.
While one could argue that the target labels could be leveraged during training like in SSDA methods, one still needs labeled target data to perform model selection.
However our results suggest that we would need at least 3 samples per class for SSDA methods.
In addition, knowing that we have a certain number of labeled samples per class provides information regarding which classes are target only, one of the main assumptions in PDA.
In that case, PDA methods could be tweaked.
This warrants further study that we leave as future work.
Finally, we have also investigated a smaller labeled target set of 50 random samples (50- RND ) instead of 100 random samples.
The accuracies of methods using 50- RND were not as good as when using 100- RND .
All results of pairs of methods and 50- RND can be found in Appendix D.
The smaller performance show that the size of the labeled target set is an important element and we suggest to use at least 100 random samples.
Model Selection Strategies (w/o target labels)
Only the ( BA 3 US , ENT ) pair achieved average task accuracies within 3 percentage points of its ORACLE counterpart ( i.e. , ( BA 3 US , ORACLE )), while still improving over S. O NLY model.
Our experiments show that there is no model selection strategy which performs well for all methods.
That is why to deploy models in a real-world scenario, we advise to test selected models on a small labeled target set (like 100- RND ) to assess the performance of the models as current model selection without target labels can perform poorly.
Our conclusion is that the model selection for PDA methods is still an open problem.
We conjecture that is also likely the case for UDA.
For future proposed methods, researchers should specify not only which model selection strategy should be used, but also which hyper-parameter search grid should be considered, in order to deploy them in a real-world scenario.
3.2 Random Seed Dependence
Ideally, PDA methods should be robust to the choice of random seed.
This is of particular importance when performing hyper-parameter tuning since typically only one run per set of hyper-parameters is done (that was the case in our work as well).
We investigate this robustness by averaging all the results presented over three different seeds (2020, 2021 and 2022) and reporting the standard deviations.
This is in contrast with previous work where only single run is reported (Fatras et al., 2021b; Gu et al., 2021).
Other works (Cao et al., 2018; Xu et al., 2019; Jian et al., 2020) that report standard deviations do not specify if the random seed is different across runs.
Results for all tasks on
VISDA dataset and OFFICE - HOME are in Appendix D due to space constraints.
Our experiments show that some methods express a non-negligible instabilities over randomness with respect to any model selection methods.
This is particularly true for BA 3 US when paired with
DEV and 1- SHOT as model selection strategies: there are several tasks where the standard deviation is above 10%.
While in this case this instability may stem from the poor performance of the model selection strategies, it is also visible when ORACLE is the model selection strategy used.
For instance, the M - POT has a standard deviation of 3.3% on the AP task of OFFICE - HOME which corresponds to a variance of 11%.
On VISDA this instability and seed dependence is even larger.
Conclusion
In this paper, we investigated how model selection strategies affect the performance of PDA methods.
We performed a quantitative study with seven PDA methods and seven model selection strategies on two real-word datasets.
Based on our findings, we provide the following recommendations:
i)
Target label samples should be used to test models before using them in real-world scenario.
While this breaks the main PDA assumption, it is impossible to confidently deploy PDA models selected without the use of target labels.
Indeed, model selection strategies without target labels lead to a significant drop in performance in most cases in comparison to using a small validation set.
We argue that the cost of labeling it outweighs the uncertainty in current model selection strategies.
ii)
The robustness of new PDA method to randomness should be tested over at least three different seeds.
We suggest to use the seeds (2020, 2021, 2022) to allow for a fair comparison with our results.
iii)
An ablation study should be considered when a novel architecture is proposed to quantify the associated increase of performance.
As our work focus on a quantitative study of model selection methods and reproducibility of state-ofthe-art partial domain adaptation methods, we do not see any potential ethical concern.
Future work will investigate new model selection strategies which can achieve similar results to validation sets. Acknowledgments
This work was partially supported by NSERC Discovery grant (RGPIN-2019-06512) and a Samsung grant.
Thanks also to CIFAR for their support through the CIFAR AI Chairs program.
Authors thank
Christos Tsirigotis and Chen Sun for early comments on the manuscript.
A Model Selection Strategies: An Overview
Model selection (choosing hyper-parameters, training checkpoints, neural network architectures) is a crucial part of training neural networks.
In the supervised learning setting, a validation set is used to estimate the model’s accuracy.
However, in UDA such approach is not possible as we have unlabeled target samples.
Several strategies have been designed to address this issue.
Below, we discuss the ones used in this work.
Source Accuracy ( S - ACC ).
Ganin & Lempitsky (2015) used the accuracy estimated on a small validation set from the source domain to perform the model selection.
While the source and target accuracies are related, there are no theoretical guarantees.
You et al.
(2019) showed that when the domain gap is large this approach fails to select competitive models.
Deep Embedded Validation ( DEV ).
Sugiyama et al.
(2007) and Long et al.
(2018) perform model selection through Importance-Weighted Cross-Validation (IWCV).
Under the assumption that the source and target domain follow a covariate shift, the target risk can be estimated from the source risk through importance weights that give increased importance to source samples that are closer to target samples.
These importance weights correspond to the ratio of the target and source densities and are estimated using Gaussian kernels.
Recently, You et al.
(2019) proposed an improved variant, Deep
Embedded Validation ( DEV ), that controls the variance of the estimator and estimates the importance weights with a discriminative model that distinguish source samples from target samples leading to a more stable and effective method.
Entropy ( ENT ).
While minimizing the entropy of the target samples has been used in domain adaptation to improve accuracy by promoting tighter clusters, Morerio et al.
(2018) showed that it can also be used for model selection.
The intuition is that a lower entropy model corresponds to a high confident model with discriminative target features and therefore reliable predictions.
Soft Neighborhood Density ( SND ).
Saito et al.
(2021) argue that a good UDA model will have a neighborhood structure where nearby target samples are in the same class.
They point out that entropy is not able to capture this property and propose the Soft Neighborhood Density ( SND ) score to address it.
Target Accuracy ( ORACLE ).
We consider as well the target accuracy on all target samples.
While we emphasize once again its use is not realistic in unsupervised domain adaptation (hence why we will refer to it as ORACLE ), it has nonetheless been used to report the best accuracy achieved by the model along training in several previous works (Cao et al., 2018; Xu et al., 2019;
Jian et al., 2020; Gu et al., 2021; Nguyen et al., 2022).
Here, we use it as an upper bound for all the other model selection strategies and to check the reproducibility of previous works.
Small Labeled Target Set (1- SHOT and 100- RND ).
For real-world applications in an industry setting, it is unlikely that a model will be deployed without the very least an estimate of its performance for which target labels are required.
Therefore, one can imagine a situation where a PDA method is used and a small set of target samples is available.
Thus, we will compute the target accuracy with 1 labeled sample per class (1- SHOT ) and 100 random labeled target samples (100- RND ) as model selection strategies.
One could argue that the 100 random samples could have been used in the training with semi-supervised domain adaptation methods.
However, note that we do not know how
[Table]
many classes we have on the target domain so it is hard to form a split when we have uncertainty of classes.
For instance, 100 random represents possibly less than 2 samples per class for one of our real-world dataset, as we do not know the number of classes, making a potential split between a train and validation target sets not possible.
Since S - ACC , DEV and SND require a source validation set, we divide the source samples into a training subset (80%) and validation subset (20%).
Regardless of the model selection strategy used, all methods are trained using the source training subset.
This is in contrast with previous work that uses all source samples, but necessary to ensure a fair comparison of the model selection strategies.
B Partial Domain Adaptation Methods
In this section, we give a brief description of the PDA methods considered in our study.
They can be grouped into two families: adversarial training and divergence minimization.
Adversarial training.
To solve the UDA problem, Ganin et al.
(2016) aligned the source and target domains with the help of a domain discriminator trained adversarially to be able to distinguish the samples from the two domains.
However, when applied to the PDA problem this strategy leads to negative transfer and the model performs worse than a model trained only on source data.
Cao et al. (2018) proposed PADA that introduces a PDA specific solution to adversarial domain adaptation: the contribution of the source-only class samples to the training of both the source classifier and the domain adversarial network is decreased.
This is achieved through class weights that are calculated by simply averaging the classifier prediction on all target samples.
As the source-only classes should not be predicted in the target domain, they should have lower weights.
More recently, Jian et al. (2020) proposed BA 3 US which augments the target mini-batch with source samples to transform the PDA problem in a vanilla DA problem.
In addition, an adaptive weighted complement entropy objective is used to encourage incorrect classes to have uniform and low prediction scores.
Divergence minimization.
Another standard direction to align the source and target distributions in the feature space of a neural network is to minimize a given divergence between distributions of domains.
Xu et al.
(2019) empirically found than target samples have low feature norm compared to source samples.
Based on this insight, they proposed SAFN which progressively adapts the feature norms of the two domains by minimizing the Maximum Mean Feature Norm Discrepancy.
Other approaches are based on optimal transport (OT) (Bhushan Damodaran et al., 2018) with mini-batches (Peyré & Cuturi, 2019; Fatras et al., 2020, 2021b).
For the PDA problem in specific, (Fatras et al., 2021a) developed JUMBOT , a mini-batch unbalanced optimal transport that learns a joint distribution of the embedded samples and labels.
The use of unbalanced OT is critical for the PDA problem as it allows to transport only a portion of the mass limiting the negative transfer between distributions.
Based on this work, (Nguyen et al., 2022) investigated the partial OT variant (Chapel et al., 2020), a particular case of unbalanced OT, proposing M - POT .
Finally, another line of work is to use the Kantorovich-Rubenstein duality of optimal transport to perform the alignment similarly to WGAN(Arjovsky et al., 2017).
This is precisely the work of Gu et al.
(2021) that proposed, AR .
In addition, source samples are reweighted in order to reduce the negative transfer from the source-only class samples.
The Kantorovich-Rubenstein duality relies on a one Lipschitz function which is approximated using adversarial training like the PDA methods described above.
We sum-up all considered methods and their experimental protocol in Table 2.
C Additional details on Experimental Protocol C.1 Implementations in BenchmarkPDA
In order to reimplement the different PDA methods, we adapted the code from the official repository associated with each of the paper.
We list them in Table 3.
[Table]
One of our main claims regarding previous work is the use of target labels to choose the best model along training.
This can be easily verified by inspecting the code.
For PADA it can be seen on line 240 of the script “train_pada.py”, for BA 3 US in line 116 for the script “run_partial.py”, for M - POT it can be seen line 164 of the file “run_mOT.py”, for SAFN it can be seen in the “eval.py” file and finally for AR in line 149 of the script “train.py”.
For JUMBOT and M - POT which are based on optimal transport, we used the optimal transport solvers from (Flamary et al., 2021).
C.2 Model Selection
DEV requires learning a discriminative model to distinguish source samples from target samples.
Its neural network architecture must be specified as well the training details.
You et al.
( DEV ) use a multilayer perceptron, while Saito et al.
(2021) ( SND ) use a Support Vector Machine in their reimplementation of DEV .
We empirically observed the latter to yield more stable weights and so that was the one we used.
In order to train the SVM discriminator, following (Saito et al., 2021), we take 3000 feature embeddings from source samples used in training and 3000 random feature embeddings from target samples, both chosen randomly.
We do a 80/20 split into training and test data.
The SVM is trained with a linear kernel for a maximum of 4000 iterations.
Of 5 different SVM models trained with decay values spaced evenly on log space between 10 − 2 and 10 4 the one that leads to the highest accuracy (in distinguishing source from target features) on the test data split is the chosen one.
As for SND , it also requires specifying a temperature for the temperature scaling component of the strategy.
We used the default value of 0.05 that is suggested in (Saito et al., 2021).
Finally, we mention that the samples used for 100-
RND were randomly selected and their list is made available together with the code.
As for the samples used for 1- SHOT , they are the same as the ones used in semi-supervised domain adaptation.
C.3 Optimizer
We use the SGD (Robbins & Monro, 1951) algorithm with momentum of 0.9, a weight decay of 5 e −and Nesterov acceleration.
As the bottleneck and classifer layers are randomly initialized, we set their learning rates to be 10 times that of the pre-trained ResNet50 backbone.
We schedule the learning rate with a strategy similar to the one in (Ganin et al., 2016): χ p = χ 0 (1+ µi ) − ν , where i is the current iteration, χ 0 = 0 .
001 , γ = 0 .
001 , ν = 0 .
While this schedule is slightly different than the one reported in previous work, it is the one implemented in the different official code implementations.
Finally, as for the mini-batch size, JUMBOT and M - POT were designed with a stratified sampling, i.e., a balanced source mini-batch with the same number of samples per class.
This allows to reduce the negative transfer between domains and is crucial to their success.
On the other hand, it was shown that for some methods (e.g. BA3US) using a larger mini-batch, than what was reported, leads to a decreased performance (Fatras et al., 2021b).
As a result, we used the default mini-batch strategies
[Table] [Table]
C.4 Hyper-Parameters
Previous works (Gulrajani & Lopez-Paz, 2021; Musgrave et al., 2021, 2022) perform random searches with the same number of runs for each method.
In contrast, we perform hyper-parameter grid searches for each method.
As a result, the hyper-parameter tuning budgets differs across the methods depending on the number of hyper-parameters and the chosen grid.
While one can argue this leads to an unfair comparison of the methods, in practice in most real-world applications one will be interested in using the best method and our approach will capture precisely that.
The hyper-parameter tuning needs to be performed for each task of each dataset, but that would require a significant computational resources without a clear added benefit.
Instead for each dataset, we perform the hyper-parameter tuning on a single task: AC for OFFICE - HOME and SR for VISDA .
This same strategy was adopted in (Fatras et al., 2021b) and the hyper-parameters were found to generalize to the remaining tasks in the dataset.
We conjecture that this may be due to the fact that information regarding the number of target only classes is implicitly hidden in the hyper-parameters.
Several runs in our hyper-parameter search for JUMBOT , M - POT and BA 3 US were unsuccessful with the optimization reaching its end without the model being trained at all.
This poses a challenge to DEV ,
SND and ENT and its one of the failures modes accounted for in (Saito et al., 2021).
Following their
[Table]
recommendations, for JUMBOT , M - POT and BA 3 US , before applying the model selection strategy, we discard models whose source domain accuracy is below a certain threshold thr , which is set with the heuristic as thr = 0 .
9 · Acc . Here Acc denotes the source domain accuracy of the Source-Only model. In our experiments, this leads to select models whom the source accuracy is at least of thr = 69 . 01% for the A2C task on OFFICE - HOME and thr = 89 .
83% for the S2R task on VISDA . We choose this heuristic because the ablation study of some methods showed that doing the adaptation decreased slightly the source accuracy (Bhushan Damodaran et al., 2018).
Table 4 shows that our heuristic leads to improved results.
Lastly, when choosing the hyper-parameters, we only consider the model at the end of training, discarding the intermediate checkpoint models in order to select hyper-parameters which do not lead to overfitting at the end of training and better generalize to the other tasks.
Following the above protocol, for each dataset we trained 468 models in total in order to find the best hyper-parameters.
Then, to obtain the results with our neural network architecture on all tasks of each dataset, we trained an additional 1224 models for OFFICE - HOME and 156 models for VISDA .
We additionally trained
[Table] [Table] 231 models with the different neural network architectures for AR and SAFN .
In total, 2547 models were trained to make this study
In Table 5, we report the values used for each hyper-parameter in our grid search.
We report in [-Table
-]Table 6 the hyper-parameters chosen by each model selection strategy for each method on both datasets.
In addition, for the reproducibility of AR with the proposed architecture in Gu et al.
(2021), a feature normalization layer is added in the bottleneck which requires specifying r , the value to which the 2-norm is set.
This hyper-parameter is therefore included in the hyper-parameter grid search with the possible values of [5 , 10 , 20] which are the different values used in the experiments in (Gu et al., 2021). [Table]
D Additional Discussion of Results D.1 Reproducibility Of Previous Results
In order to trust our results, we need to unsure we that we correctly reimplementated the different
PDA methods.
We do so by comparing our reproduced results and the previously reported results in
Table 7.
On OFFICE - HOME , both PADA and JUMBOT achieved higher average task accuracy (1.6 and 1.7 percentage points, respectively) in our reimplementation, while for BA 3 US and MPOT we recover the reported accuracy in their respective papers.
However, we saw a decrease in performance for both
SAFN and AR of roughly 8 and 5 percentage points respectively.
This is to be expected due to the differences in the neural network architectures.
While we use a linear bottleneck layer, SAFN use a nonlinear bottleneck layer.
As for AR , they make two significant changes: the linear classification head is replaced by a spherical logistic regression (SLR) layer (Gu et al., 2020) and the features are normalized (the 2-norm is set to a dataset dependent value, another hyper-parameter that requires tuning) before feeding them to the classification head.
While we account for the first change by comparing to the AR (w/ linear) results reported in (Gu et al., 2021), in our neural network architecture we do not normalize the features.
These changes, nonlinear bottleneck layer for SAFN and feature normalization for AR , significantly boost the performance of both methods.
When now comparing our reimplementation with the same neural network architectures, our SAFN reimplementation achieves a higher average task accuracy by 3 percentage points, while our AR reimplementation is now only 1 percentage points below.
The fact that AR reported results are from only one run, while ours are averaged across 3 distinct seeds, justifies the small remaining gap.
Moreover, we report higher accuracy or on par on 4 tasks of the 12 tasks.
In Table 8, we compare previously reported results with ours on VISDA .
While proposed methods reported results on OFFICE - HOME , only PADA and AR results are reported in the original papers for VISDA .
Gu et al.
(2021) AR ) also report results for BA 3 US .
Analysing the results, we see apercentage point decrease in average task accuracy for PADA , but our experiments show that there is a significant seed dependence which we discuss in detail below.
This is particularly important since Cao et al.
(2018) ( PADA ) report results from a single run.
Comparing our best seeds for PADA on the SR and RS tasks, we achieve 58.01% and 67.9% accuracy versus a reported 53.53% and 76.5%.
Moreover, we point out that the official code repository for PADA does not include the details to reproduce the VISDA experiments, so it is possible that minor tweaks (e.g learning rate) are necessary.
As for BA 3 US , our results are within the standard deviation being better on the SR task and worse on the RS task.
Finally as for AR we see a decrease in performance which, as the results on OFFICE - HOME show, can be explained by the differences in the neural network architecture.
Given all the above results, our reimplementations are trustworthy and give validity to our results.
D.2 Additional Results
In this section, we provide additional results that we could not add to the main paper due to the space constraints.
In Table 9 and Table 10, we show the accuracy per task on OFFICE - HOME and VISDA respectively, averaged over three different seeds (2020, 2021, 2022) for all pairs of methods and model selection strategies.
Finally in Table 11, we show all the average task accuracies from all pairs of methods and model selection strategies on the OFFICE - HOME and VISDA datasets including the 50- RND model selection strategy. [Table]
[Table]