T RANSFER L EARNING WITH P RE - TRAINED C ONDITIONAL G ENERATIVE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Transfer learning is crucial in training deep neural networks on new target tasks.
Current transfer learning methods always assume at least one of (i) source and target task label spaces overlap, (ii) source datasets are available, and (iii) target network architectures are consistent with source ones.
However, holding these assumptions is difﬁcult in practical settings because the target task rarely has the same labels as the source task, the source dataset access is restricted due to storage costs and privacy, and the target architecture is often specialized to each task.
To transfer source knowledge without these assumptions, we propose a transfer learning method that uses deep generative models and is composed of the following two stages: pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL).
PP trains a target architecture with an artiﬁcial dataset synthesized by using conditional source generative models.
P-SSL applies SSL algorithms to labeled target data and unlabeled pseudo samples, which are generated by cascading the source classiﬁer and generative models to condition them with target samples.
Our experimental results indicate that our method can outperform the baselines of scratch training and knowledge distillation.
I NTRODUCTION
For training deep neural networks on new tasks, transfer learning is essential, which leverages the knowledge of related (source) tasks to the new (target) tasks via the joint- or pre-training of source models.
There are many transfer learning methods for deep models under various conditions (Pan & Yang, 2010; Wang & Deng, 2018).
For instance, domain adaptation leverages source knowledge to the target task by minimizing the domain gaps (Ganin et al., 2016), and ﬁne-tuning uses the pre-trained weights on source tasks as the initial weights of the target models (Yosinski et al., 2014).
These existing powerful transfer learning methods always assume at least one of (i) source and target label spaces have overlaps, e.g., a target task composed of the same class categories as a source task, (ii) source datasets are available, and (iii) consistency of neural network architectures i.e., the architectures in the target task must be the same as that in the source task.
However, these assumptions are seldom satisﬁed in real-world settings (Chang et al., 2019; Kenthapadi et al., 2019; Tan et al., 2019).
For instance, suppose a case of developing an image classiﬁer on a totally new task for an embedded device in an automobile company.
The developers found an optimal neural architecture for the target dataset and the device by neural architecture search, but they cannot directly access the source dataset for the reason of protecting customer information.
In such a situation, the existing transfer learning methods requiring the above assumptions are unavailable, and the developers cannot obtain the best model.
To promote the practical application of deep models, we argue that we should reconsider the three assumptions on which the existing transfer learning methods depend.
For assumption (i), new target tasks do not necessarily have the label spaces overlapping with source ones because target labels are often designed on the basis of their requisites.
In the above example, if we train models on
StanfordCars (Krause et al., 2013), which is a ﬁne-grained car dataset, there is no overlap with
ImageNet (Russakovsky et al., 2015) even though ImageNet has 1000 classes.
For (ii), the accessibility of source datasets is often limited due to storage costs and privacy (Liang et al., 2020; Kundu et al., 2020; Wang et al., 2021a), e.g., ImageNet consumes over 100GB and contains person faces co-occurring with objects that potentially raise privacy concerns (Yang et al., 2022).
For (iii), the consistency of the source and target architectures is broken if the new architecture is specialized for
[Table]
[Figure]
[Table]
the new tasks like the above example.
Deep models are often specialized for tasks or computational resources by neural architecture search (Zoph & Le, 2017; Lee et al., 2021) in particular when deploying on edge devices; thus, their architectures can differ for each task and runtime environment.
Since existing transfer learning methods require one of the three assumptions, practitioners must design target tasks and architectures to ﬁt those assumptions by sacriﬁcing performance.
To maximize the potential performance of deep models, a new transfer learning paradigm is required.
In this paper, we shed light on an important but less studied problem setting of transfer learning, where (i) source and target task label spaces do not have overlaps, (ii) source datasets are not available, and (iii) target network architectures are not consistent with source ones (Tab. 1).
To transfer source knowledge while satisfying the above three conditions, our main idea is to leverage source pre-trained discriminative and generative models; their architectures differ from that of target tasks.
We focus on applying the generated samples from source class-conditional generative models for target training.
Deep conditional generative models precisely replicate complex data distributions such as ImageNet (Brock et al., 2018; Karras et al., 2020; Dhariwal & Nichol, 2021), and the pre-trained models are widely used for downstream tasks (Wang et al., 2018;
Zhao et al., 2020a; Patashnik et al., 2021; Ramesh et al., 2022).
Furthermore, deep generative models have the potential to resolve the problem of source dataset access because they can compress information of large datasets into much smaller pre-trained weights (e.g., about 100MB in the case of a BigGAN generator), and safely generate informative samples without re-generating training samples by differential privacy training techniques (Torkzadehmahani et al., 2019;
Augenstein et al., 2020; Liew et al., 2022).
By using conditional generative models, we propose a two-stage transfer learning method composed of pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL).
Figure 1 illustrates an overview of our method.
PP pre-trains the target architectures by using the artiﬁcial dataset generated from the source conditional generated samples and given labels.
This simple pre-process provides effective initial weights without accessing source datasets and architecture consistency.
To address the non-overlap of the label spaces without accessing source datasets, P-SSL trains a target model with SSL (Chapelle et al., 2006; Van Engelen & Hoos, 2020) by treating pseudo samples drawn from the conditional generative models as the unlabeled dataset.
Since SSL assumes the labeled and unlabeled datasets are drawn from the same distribution, the pseudo samples should be target-related samples, whose distribution is similar enough to the target distribution.
To generate target-related samples, we cascade a classiﬁer and conditional generative model of the source domain.
Speciﬁcally, we (a) obtain pseudo source soft labels from the source classiﬁer by applying
[Table]
D
[Table]
+//qio0Sxx8qf70LFHEiudVkHfbY9q30Dv62tHp69ZqOtaYZ1fshfxfsha7pxuYtTf9OsXTFwjRByR+Pnc3yC7GE0vxZCoZXUv6XxHEDOawQO+9jDVsYBMZOlfgBGc4DzwrEWVSme6kKgFfM4Fvocx9AiVCi+Y=</latexit> x t <latexit sha1_base64="PuPCtfefNxRGbPQ5cOPenfan2CQ=">AAACeHichVHLSgMxFD0d3/XRajeCm9Lia1NSKSqCILhx6atVaEs7M6Z16LyYSQu19Af8ARcuREFU/Aw3/oCLfoK4rCCIC2+nA6JFvSHJyck9NyeJYuuaKxhrBaS+/oHBoeGR4OjY+EQoPDmVca2qo/K0aumWc6jILtc1k6eFJnR+aDtcNhSdHyiVzc7+QY07rmaZ+6Ju87whl02tpKmyIKoQjtQLYr2REyJaLG5VDYM7sWYhHGcJ5kW0FyR9EIcf21b4BjkcwYKKKgxwmBCEdchwqWWRBINNXB4N4hxCmrfP0USQtFXK4pQhE1uhsUyrrM+atO7UdD21Sqfo1B1SRjHLntgda7NHds+e2cevtRpejY6XOs1KV8vtQuh0eu/tX5VBs8Dxl+pPzwIlrHpeNfJue0znFmpXXzs5a++t7c425tgVeyH/l6zFHugGZu1Vvd7hu+cI0gckfz53L8gsJZLLidROKr6R8r9iGDOIYYHeewUb2MI20nRuHRe4xV3gXYpK89JiN1UK+JoIvoW09AnbAJGV</latexit> y t = “ Hummer ” <latexit sha1_base64="mzVOV2VAL7dmcZ0+gmW8Znbd6BU=">AAACeHichVHLSgMxFD0dX7W
[Figure]
target data, and (b) generate conditional samples given a pseudo source soft label.
By using the target-related samples, P-SSL trains target models with off-the-shelf SSL algorithms.
In the experiments, we ﬁrst conﬁrm the effectiveness of our method through a motivating example scenario where the source and target labels do not overlap, the source dataset is unavailable, and the architectures are specialized by manual neural architecture search (Sec. 4.2).
Then, we show that our method can stably improve the baselines in transfer learning without three assumptions under various conditions: e.g., multiple target architectures (Sec. 4.3), and multiple target datasets (Sec. 4.4).
These indicate that our method succeeds to make the architecture and task designs free from the three assumptions.
Further, we conﬁrm that our method can achieve practical performance without the three assumptions: the performance was comparable to the methods that require one of the three assumptions (Sec. 4.5 and 4.7).
We also provide extensive analysis revealing the conditions for the success of our method.
For instance, we found that the target performance highly depends on the similarity of generated samples to the target data (Sec. 3.2.4 and 4.4), and a general source dataset (ImageNet) is more suitable than a speciﬁc source dataset (CompCars) when the target dataset is
StanfordCars (Sec. 4.6).
P ROBLEM S ETTING
We consider a transfer learning problem where we train a neural network model
f
A t θ on a labeled target dataset D t = { ( x it ∈
X t , y it ∈
Y t ) } N t
i =1 given a source classiﬁer C A s s and a source conditional generative model G s ;
C A s s and G s are off-the-shelf i.e., we do not access source datasets to pre-train them.
f A t θ is parameterized by θ of a target neural architecture A t .
C A s s outputs class probabilities by softmax function and G s is pre-trained on a labeled source dataset D s .
We mainly consider classiﬁcation problems and denote f A t θ as the target classiﬁer C A t t in the below.
In this setting, we assume the following conditions.
(i) no label overlap : Y s ∩ Y t = ∅ (ii) no source dataset access : D s is not available when training C A t t (iii) architecture inconsistency : A s (cid:54) =
A t
Existing methods are not available when the three conditions are satisﬁed simultaneously.
Unsupervised domain adaptaion (Ganin et al., 2016) require Y s ∩ Y t (cid:54) = ∅ , accessing D s , and A s =
A t .
Source-free domain adaptation methods (Liang et al., 2020; Kundu et al., 2020; Wang et al., 2021a) can adapt models without accessing D s , but still depend on Y s ∩ Y t (cid:54) = ∅ and A s =
A t .
Fine- tuning (Yosinski et al., 2014) can be applied to the problem with the condition (i) and (ii) if and only if A s =
A t .
However, since recent progress of neural architecture search (Zoph & Le, 2017; Wang et al., 2021b; Lee et al., 2021) enable to ﬁnd specialized
A t for each task, situations of A s (cid:54) =
A t are common when developing models for environments requiring both accuracy and model size e.g., embedded devices.
As a result, the specialized C A t t currently sacriﬁces the accuracy so that the size requirement can be satisﬁed.
Therefore, tackling this problem setting has the potential to enlarge the applicability of deep models.
P ROPOSED M ETHOD
In this section, we describe our proposed method.
An overview of our method is illustrated in
Figure 1.
PP yields initial weights for a target architecture by training it on the source task with synthesized samples from a conditional source generative model.
P-SSL takes into account the pseudo samples drawn from generative models as an unlabeled dataset in an SSL setting.
In P-SSL, we generate target-related samples by pseudo conditional sampling (PCS, Figure 2), which cascades a source classiﬁer and a source conditional generative model.
3. P SEUDO P RE - TRAINING
Without accessing source datasets and architectures consistency, we cannot directly use the existing pre-trained weights for ﬁne-tuning C
A t t .
To build useful representations under these conditions, we pre-train the weights of A t with synthesized samples from a conditional source generative model.
Every training iteration of PP is composed of two simple steps: (step 1) synthesizing a batch of generating source conditional samples { ˆ x is ∼ G s ( y is ) } B (cid:48)
s i =1 from uniformly sampled source labels y is ∈ Y s , and (step 2) optimizing θ on the source classiﬁcation task with the labeled batch of { (ˆ x is , y is ) } B (cid:48)
s i =1 by minimizing
[Equation]
where B (cid:48) s is the batch size for PP and CE is cross-entropy loss.
Since PP alternately performs the sample synthesis and training in an online manner, it efﬁciently yields pre-trained weights without consuming massive storage.
Further, we found that this online strategy is better in accuracy than the ofﬂine strategy: i.e., synthesizing ﬁxed samples in advance of training (Appendix C.4).
We use the pre-trained weights from PP as the initial weights of C A t t by replacing the ﬁnal layer of the source task to that of the target task
. 3. P SEUDO S EMI - SUPERVISED L EARNING 3.2. S EMI - SUPERVISED L EARNING
Given a labeled dataset D l = { ( x i , y i ) } N l i =1 and an unlabeled dataset D u
= { ( x i ) } N u i =1 , SSL is used to optimize the parameter θ of a deep neural network by solving the following problem.
[Equation]
where L sup is a supervised loss for a labeled sample ( x l , y l ) (e.g., cross-entropy loss), L unsup is an unsupervised loss for an unlabeled sample x u , and λ is a hyperparameter for balancing L sup and
L unsup .
In SSL, it is generally assumed that D l and D u shares the same generative distribution p ( x ) .
If there is a large gap between the labeled and unlabeled data distribution, the performance of SSL algorithms degrades (Oliver et al., 2018).
However, Xie et al.
(2020) have revealed that unlabeled samples in another dataset different from a target dataset can improve the performance of
SSL algorithms by carefully selecting target-related samples from source datasets.
This implies that
SSL algorithms can achieve high performances as long as the unlabeled samples are related to target datasets, even when they belong to different datasets.
On the basis of this implication, our P-SSL exploits pseudo samples drawn from source generative models as unlabeled data for SSL.
3.2. P SEUDO C ONDITIONAL S AMPLING
To generate informative target-related samples, our method uses PCS, which generates target-related samples by cascading C A s s and G s .
With PCS, we ﬁrst obtain a pseudo source label y s ← t from a source classiﬁer C A s s with a uniformly sampled x t from D t .
[Equation]
Intuitively, y s ← t represents the relation between source class categories and x t in the form of the probabilities.
We then generate target-related samples x s ← t with y s ← t as the conditional label by
[Equation]
Require: Target dataset D t , source classiﬁer C A s s , source generator G s , number of pseudo samples
N s ← t , output label function g
Ensure:
Pseudo unlabeled dataset D s ← t 1: Y s ← t ← ∅ 2: ˆ C A t s ← SwapFinalLayer ( C A s s ,l ) 3: for x t in D t do 4: y s ← t ← ˆ C A s s ( x t ) 5:
Add y s ← t to Y s ← t 6: end for 7: Repeat concatenating Y s ← t with itself until the length reaches to N s ← t 8: D s ← t ← ∅ 9: for y s ← t in Y s ← t do 10: x s ← t ∼ G s ( y s ← t ) 11:
Add x s ← t to D s ← t 12: end for [Table] Although G s is trained with discrete (one-hot) class labels, it can generate class-wise interpolated samples by the continuously mixed labels of multiple class categories (Miyato & Koyama, 2018;
Brock et al., 2019).
By leveraging this characteristic, we aim to generate target-related samples by y s ← t constructed with an interpolation of source classes.
For the training of the target task, we compose a pseudo dataset D s ← t by applying Algorithm 1.
In line 2, we swap the ﬁnal layer of C A s s with an output label function g , which is softmax function as the default.
Sec.
C.5.3, we empirically evaluate the effects of the choice of g .
3.2. T RAINING
By applying PCS, we obtain a target-related sample x s ← t from x t .
In the training of C A t t , one can assign the label y t of x t to x s ← t since x s ← t is generated from x t .
However, it is difﬁcult to directly use ( x s ← t , y t ) in supervised learning because x s ← t can harm the performance of C A t t due to the gap between label spaces; we empirically conﬁrm that this na ¨ ıve approach fails to boost the target performance mentioned in Sec.
C.5.1.
To extract informative knowledge from x s ← t , we apply SSL algorithms that train a target classiﬁer C A t t by using a labeled target dataset D t and an unlabeled dataset D s ← t generated by PCS.
On the basis of the implication discussed in Sec.
3.2.1, we can expect that the training by SSL improve C A t t if D s ← t contains target-related samples.
We compute the unsupervised loss function for a pseudo sample x s ← t as L unsup ( x s ← t , θ ) .
We can adopt arbitrary SSL algorithms for calculating
L unsup .
For instance, UDA (Xie et al., 2020) is deﬁned by
[Equation]
where 1 is an indicator function, CE is a cross entropy function, ˆ C A t t ( x, τ ) is the target classiﬁer replacing the ﬁnal layer with the temperature softmax function with a temperature hyperparameter τ , β is a conﬁdence threshold, and T ( · ) is an input transformation function such as RandAugment (Cubuk et al., 2020).
In the experiments, we used UDA because it achieves the best result; we compare and discuss applying the other SSL algorithms in Sec.
C.5.1.
Eventually, we optimize the parameter θ by the following objective function based on Eq (2).
[Equation]
3.2. Q UALITY OF P SEUDO S AMPLES
In this method, since we treat x s ← t ∼ G s
( C A s s ( x t ))
as the unlabeled sample of the target task, the following assumption is required:
[Equation]
where p D ( x ) is a data distribution of a dataset D .
That is, if pseudo samples satisfy Assumption 3.1,
P-SSL should boost the target task performance by P-SSL. [Table]
[Table] To conﬁrm that pseudo samples can satisfy Assumption 3. [Table] we assess the difference between the target distribution and pseudo distribution.
Since it is difﬁcult to directly compute the likelihood of the pseudo samples [Table] we leverage Fr´echet Inception Distance (FID, Heusel et al.
(2017)), which measures a distribution gap between two datasets by 2-Wasserstein distance in the closed-form (lower
FID means higher similarity).
We evaluate the quality of D s ← t by comparing FID ( D s ← t , D t ) to FID (
D s , D t ) and FID ( F ( D s ) , D t ) , where F ( D s ) is a subset of D s constructed by conﬁdence-based ﬁltering similar to a previous study (Xie et al., 2020) (see Appendix B.4 for more detailed protocol).
That is, if D s ← t achieves lower FID than D s or D (cid:48)
s , then D s ← t can approximate D t well.
Table 2 shows the FID scores when D s is ImageNet.
The experimental settings are shared with
Sec. 4.4.
Except for DTD and StanfordDogs, D s ← t outperformed D s and F ( D s ) in terms of the similarity to
D t .
This indicates that PCS can produce more target-related samples than the natural source dataset.
On the other hand, in the case of DTD (texture classes), D s ← t relatively has low similarity to D t .
This implies that PCS does not approximate p D t ( x ) well when the source and target datasets are not so relevant.
Note that StanfordDogs is a subset of ImageNet, and thus
FID ( F ( D s ) , D t ) was much smaller than other datasets.
Nevertheless, the FID ( D s ← t , D t ) is com- parable to FID ( F (
D s ) , D t ) .
From this result, we can say that our method approximates the target distribution almost as well as the actual target samples.
We further discuss the relationships between the pseudo sample quality and target performance in Sec. 4.4.
E XPERIMENTS
We evaluate our method with multiple target architectures and datasets, and compare it with baselines including scratch training and knowledge distillation that can be applied to our problem setting with a simple modiﬁcation.
We further conduct detailed analyses of our pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL) in terms of (a) the practicality of our method by comparing to the transfer learning methods that require the assumptions of source dataset access and architecture consistency, (b) the effect of source dataset choices, (c) the applicability toward another target task (object detection) other than classiﬁcation.
We further provide more detailed experiments including the effect of varying target dataset size (Appendix C.2), the performance difference when varying source generative model (Appendix C.3), the detailed analysis of PP (Appendix C.4) and PSSL (Appendix C.5), and the qualitative evaluations of pseudo samples (Appendix D).
We provide the detailed settings for training in Appendix B.3.
4. S ETTING
Baselines Basically, there are no existing transfer learning methods that are available on the problem setting deﬁned in Sec. 2.
Thus, we evaluated our method by comparing it with the scratch training ( Scratch ), which trains a model with only a target dataset, and na¨ıve applications of knowledge distillation methods: Logit Matching (Ba & Caruana, 2014) and Soft Target (Hinton et al., 2015).
Logit Matching and Soft Target can be used for transfer learning under architecture inconsistency since their loss functions use only ﬁnal logit outputs of models regardless of the intermediate layers.
To transfer knowledge in C A s s to C A t t , we ﬁrst yield C A s t by ﬁne-tuning the parameters of
C A s s on the target task, and then train C
A t t with knowledge distillation penalties by treating C A s t as the teacher model.
We provide more details in Appendix B.2.
Datasets We used ImageNet (Russakovsky et al., 2015) as the default source datasets.
In Sec. 4.6, we report the case of applying CompCars (Yang et al., 2015) as the source dataset.
For the target dataset, we mainly used StanfordCars (Krause et al., 2013), which is for ﬁne-grained classiﬁcation of car types, as the target dataset.
In Sec. 4.4, we used the nine classiﬁcation datasets listed in
[Table]
[Table] [Table]
Network Architecture As a source architecture A s , we used the ResNet-50 architecture (He et al., 2016) with the pre-trained weight distributed by torchvision ofﬁcial repository.
For a target architecture A t , we used ﬁve architectures publicly available on torchvision : WRN-502 (Zagoruyko & Komodakis, 2016), MNASNet1.0 (Tan et al., 2019), MobileNetV3-L (Howard et al., 2019), and EfﬁcientNet-B0/B5 (Tan & Le, 2019).
Note that, to ensure reproducibility, we assume them as entirely new architectures for the target task in our problem setting, while they are actually existing architectures.
For a source conditional generative model G s , we used BigGAN (Brock et al., 2018) generating 256 × 256 resolution images as the default architecture.
We also tested the other generative models such as ADM-G (Dhariwal & Nichol, 2021) in Sec.
C.3.
We implemented
BigGAN on the basis of open source repositories including pytorch-pretrained-BigGAN 2 ; we used the pre-trained weights distributed by the repositories.
4. M OTIVATING E XAMPLE : M ANUAL A RCHITECTURE S EARCH
First of all, we conﬁrm the effectiveness of our setting and method through a practical scenario described in Sec.
Here, we consider a case of manually optimizing the number of layers of ResNet-18 (RN18) for the target task to improve the performance while keeping the model size.
We evaluated our method on the above scenario by assuming StanfordCars as the target dataset and
ImageNet as the source dataset.
We searched the custom architecture by grid search of the layers in four blocks of RN18 from (2 , 2 , 2 , 2) to (2 , 2 , 2 , 10) by varying layers in { 2 , 4 , 6 , 8 , 10 } for each block of ResNet while keeping the sum of layers less than or equal to 18 to maintain architecture size. We found that the best architecture is one with (2 , 10 , 2 , 2) .
The test accuracies on StanfordCars are shown in Table 4.
We can see that ﬁnding an optimal custom RN18 architecture for the target task brings test accuracy improvements, and our method contributes to further improvements under this difﬁcult situation.
This result indicates that our method can widen the applicability of neural architecture search techniques, which have been difﬁcult in practice in terms of accuracy.
4. T ARGET A RCHITECTURES
We discuss the performance evaluations by varying the neural network architectures of the target classiﬁers to evaluate our method on the condition of architecture inconsistency.
Table 5 lists the results on StanfordCars with multiple target architectures.
Note that we evaluated our method by ﬁxing the source architecture to ResNet-50.
Our method outperformed the baselines on all target architectures without architecture consistency.
Remarkably, our method stably performs arbitrary
[Table] [Table]
relationships between target and source architectures including from a smaller architecture (ResNet50) to larger ones (WRN-50-2 and EfﬁcientNet-B5), and from a larger one (ResNet-50) to smaller ones (MNASNet1.0, MobileNetV3, and EfﬁcientNet-B0).
This ﬂexibility can lead to the effectiveness of the neural architecture search in Sec. 4.2.
4. T ARGET D ATASETS
[Figure]
We show the efﬁcacy of our method on multiple target datasets other than StanfordCars.
We used WRN-50-2 as the architecture of the target classiﬁers.
Table 6 lists the top-1 accuracy of each classiﬁcation task.
Our method stably improved the baselines across datasets.
We also print the ablation results of our method (Ours w/o PP and Ours w/o P-UDA) for assessing the dependencies of PP and P-SSL on datasets.
PP outperformed the baselines on all target datasets.
This suggests that building basic representation by pre-training is effective for various target tasks even if the source dataset is drawn from generative models.
For P-SSL, we observed that it boosted the scratch models in all target datasets except for DTD and
OxfordFlower.
As the reasons for the degradation on DTD and OxfordFlower, we consider that the pseudo samples do not satisfy Assumption 3.1 as discussed in Sec. 3.2.4.
In fact, we observe that FID ( D s ← t , D t ) is correlated to the accuracy gain from Scratch models ( − 0 .
80 of correlation coefﬁcient, − 0 .
97 of Spearman rank correlation) as shown in Fig.
These experimental results suggest that our method is effective on the setting without no label overlap as long as D s ← t approximates D t well.
4. C OMPARISON TO M ETHODS R EQUIRING P ARTIAL C ONDITIONS
To conﬁrm the practicality of our method, we compared it with methods requiring A s =
A t and accessing
D s .
We tested Fine-tuning (FT) and R-SSL , which use a source pre-trained model and a real source dataset for SSL as reference.
For FT and R-SSL, we used ImageNet and a subset of ImageNet (Filtered ImageNet), which was collected by conﬁdence-based ﬁltering similar to Xie et al.
(2020) (see Appendix B.4).
This manual ﬁltering process corresponds to PCS in P-SSL.
In
Table 7, PP and P-SSL outperformed FT and R-SSL, respectively.
This suggests that the samples from G s not only preserve essential information of D s but are also more useful than D s , i.e., accessing D s may not be necessary for transfer learning.
In summary, PP and P-SSL are practical enough compared to existing methods that require assumptions.
4. S OURCE D ATASETS
We investigate the preferable characteristics of source datasets for PP and P-SSL by testing another source dataset, which was CompCars (Yang et al., 2015), a ﬁne-grained vehicle dataset containing 136K images (see Appendix B.5 for details).
This is more similar to the target dataset (StanfordCars) than ImageNet.
All training settings were the same as mentioned in Sec. 4.1.
Table 8 lists the scores for each model of our methods.
The models using ImageNet were superior to those using CompCars.
To seek the difference, we measured FID( D s ← t , D t ) when using CompCars as with the protocol in Sec. 4.5, and the score was 22.12, which is inferior to 19.92 when using ImageNet.
This suggests that the ﬁdelity of pseudo samples toward target samples is important to boost the target performance and is not simply determined by the similarity between the source and target datasets.
We consider that the diversity of source classes provides the usefulness of pseudo samples; thus ImageNet is superior to CompCars as the source dataset.
4. A PPLICATION TO O BJECT D ETECTION T ASK
[Table]
Although we have mainly discussed the cases where the target task is classiﬁcation through the paper, our method can be applied to any task for which the SSL method exists.
Here, we evaluate the applicability of PP+P-SSL toward another target task other than classiﬁcation.
To this end, we applied our method to object detection task on PASCAL-VOC 2007 (Everingham et al., 2015) by using FPN
(Lin et al., 2017) models with ResNet-50 backbone.
As the SSL method, we used Unbiased Teacher (UT, (Liu et al., 2021)), which is a method for object detection based on self distillation and pseudo labeling, and implemented PP+P-SSL on the code base provided by Liu et al.
We generated samples for PP and P-SSL in the same way as the classiﬁcation experiments; we used ImageNet as the source dataset.
Table 9 shows the results of average precision scores calculated by following detectron2 (Wu et al., 2019).
Note that the baseline of Table 9 is FT instead of Scratch because the Scratch setting of the object detection task is hard to train and too slow to converge.
We conﬁrm that PP achieved competitive results to FT and P-SSL signiﬁcantly boosted the PP model.
This can be caused by the high similarity between D s ← t and D t (19.0 in FID).
This result indicates that our method has the ﬂexibility to be applied to other tasks as well as classiﬁcation and is expected to improve baselines.
R ELATED W ORK
We brieﬂy discuss related works by focusing on training techniques applying generative models.
We also provide discussions on existing transfer learning and semi-supervised learning in Appendix A.
Similar to our study, several studies have applied the expressive power of conditional generative models to boost the performance of discriminative models; Zhu et al.
(2018) and Yamaguchi et al.
(2020) have exploited the generated images from conditional GANs for data augmentation in classiﬁcation, and Sankaranarayanan et al.
(2018) have introduced conditional GANs into the system of domain adaptation for learning joint-feature spaces of source and target domains.
Moreover, Li et al.
(2020b) have implemented an unsupervised domain adaptation technique with conditional GANs in a setting of no accessing source datasets.
These studies require label overlapping between source and target tasks or training of generative models on target datasets, which causes problems of overﬁtting and mode collapse when the target datasets are small (Zhang et al., 2020;
Zhao et al., 2020b;
Karras et al., 2020).
Our method, however, requires no additional training in generative models because it simply extracts samples from ﬁxed pre-trained conditional generative models.
C ONCLUSION AND L IMITATION
We explored a new transfer learning setting where (i) source and target task label spaces do not have overlaps, (ii) source datasets are not available, and (iii) target network architectures are not consistent with source ones.
In this setting, we cannot use existing transfer learning such as domain adaptation and ﬁne-tuning.
To transfer knowledge, we proposed a simple method leveraging pre-trained conditional source generative models, which is composed of PP and P-SSL.
PP yields effective initial weights of a target architecture by generated samples and P-SSL applies an SSL algorithm by taking into account the pseudo samples from the generative models as the unlabeled dataset for the target task.
Our experiments showed that our method can practically transfer knowledge from the source task to the target tasks without the assumptions of existing transfer learning settings.
One of the limitations of our method is the difﬁculty to improve target models when the gap between source and target is too large.
A future step is to modify the pseudo sampling process by optimizing generative models toward the target dataset, which was avoided in this work to keep the simplicity and stability of the method.
A PPENDIX
The following manuscript provides the supplementary materials of the main paper: Transfer Learning with Pre-trained Conditional Generative Models.
We describe (A) additional related works of domain adaptation and ﬁne-tuning, (B) details of experimental settings used in the main paper, (C) additional experiments including comparison of our method and ﬁne-tuning, and detailed analysis of PP and P-SSL, and (D) qualitative evaluations of pseudo samples by PCS.
A E XTENDED R ELATED W ORK A. D OMAIN A DAPTATION
Domain adaptation leverages source knowledge to the target task by minimizing domain gaps between source and target domains through joint-training (Ganin et al., 2016).
It is generally assumed that the source and target task label spaces overlap (Pan & Yang, 2010; Wang & Deng, 2018) and labeled source datasets are available when training target models.
Several studies have attempted to solve the transfer learning problems called source-free adaptation (Chidlovskii et al., 2016; Liang et al., 2020; Kundu et al., 2020; Wang et al., 2021a), where the model must adapt to the target domain without target labels and the source dataset.
However, they still require architecture consistency and overlaps between the source and target tasks, i.e., it is not applicable to our problem setting.
A. F INETUNING
Our method is categorized as an inductive transfer learning approach (Pan & Yang, 2010), where thelabeled target datasets are available and the source and target task label spaces do not overlap.
In deep learning, ﬁne-tuning (Yosinski et al., 2014; Agrawal et al., 2014; Girshick et al., 2014), which leverages source pre-trained weights as initial parameters of the target modesl, is one of the most common approaches of inductive transfer learning because of its simplicity.
Previous studies have attempted to improve ﬁne-tuning by adding a penalty of the gaps between source and target models such as adding L 2 penalty term (Li et al., 2018) or penalty using channel-wise importance of feature maps (Li et al., 2019).
You et al.
(2020) have introduced category relationships between source and target tasks into target-task training and penalized the target models to predict pseudo source labels that are the outputs of the source models by applying target data.
Shu et al.
(2021) have presented an approach leveraging multiple source models pre-trained on different datasets and tasks by mixing the outputs via adaptive aggregation modules.
Although these methods outperform the na ¨ ıve ﬁne-tuning baselines, they require architecture consistency between source and target tasks.
In contrast to the ﬁne-tuning methods, our method can be used without architecture consistency and source dataset access since it transfers source knowledge via pseudo samples drawn from source pre-trained generative models.
A. S EMI - SUPERVISED L EARNING
SSL is a paradigm that trains a supervised model with labeled and unlabeled samples by minimizing supervised and unsupervised loss simultaneously.
Historically, various SSL algorithms have been used or proposed for deep learning such as entropy minimization (Grandvalet & Bengio, 2005), pseudo-label
(Lee et al., 2013), virtual adversarial training (Miyato et al., 2017), and consistency regularization (Bachman et al., 2014; Sajjadi et al., 2016; Laine & Aila, 2016).
UDA (Xie et al., 2020) and FixMatch (Sohn et al., 2020), which combine ideas of pseudo-label and consistency regularization, have achieved remarkable performance.
An assumption with these SSL algorithms is that the unlabeled data are sampled from the same distribution as the labeled data.
If there is a large gap between the labeled and unlabeled data distribution, the performance of SSL algorithms degrades (Oliver et al., 2018).
However, Xie et al.
(2020) have revealed that unlabeled samples in another dataset different from a target dataset can improve the performance of SSL algorithms by carefully selecting target-related samples from source datasets.
This indicates that SSL algorithms can achieve high performances as long as the unlabeled samples are related to target datasets, even when they belong to different datasets.
On the basis of this implication, our P-SSL exploits pseudo samples drawn from source generative models as unlabeled data for SSL.
We tested SSL algorithms for P-SSL and compared the resulting P-SSL models with SSL models using the ﬁltered real source dataset constructed by the protocol of Xie et al.
(2020) in Appendix C.5.1 and Sec. 4.5.
B D ETAILS OF E XPERIMENTS B. D ATASET D ETAILS
ImageNet (Russakovsky et al., 2015): We downloaded ImageNet from the ofﬁcial site https: //www.image-net.org/ .
ImageNet is released under license that allows it to be used for non-commercial research/educational purposes (see https://image-net.org/download. php ).
Caltech-256 (Grifﬁn et al., 2007):
We downloaded Caltech-256 from the ofﬁcial site https:// data.caltech.edu/records/20087 .
Caltech-256 is released under CC-BY license.
CUB-200-2011 (Welinder et al., 2010): We downloaded CUB-200-2011 from the ofﬁcial site http://www.vision.caltech.edu/datasets/cub_200_2011/ .
CUB-200-2011 is released under license that allows it to be used for non-commercial purposes (see https:// authors.library.caltech.edu/27452/ ).
DTD (Cimpoi et al., 2014):
We downloaded DTD from the ofﬁcial site https://www.robots.
ox.ac.uk/˜vgg/data/dtd/ .
DTD is released under license that allows it to be used for non-commercial research purposes (see https://www.robots.ox.ac.uk/˜vgg/data/ dtd/ ).
FGVC-Aircraft (Maji et al., 2013):
We downloaded FGVC-Aircraft from the ofﬁcial site https: //www.robots.ox.ac.uk/˜vgg/data/fgvc-aircraft/ .
FGVC-Aircraft is released under license that allows it to be used for non-commercial research purposes (see https://www.
robots.ox.ac.uk/˜vgg/data/fgvc-aircraft/ ).
Indoor67 (Quattoni & Torralba, 2009):
We downloaded Indoor67 from the ofﬁcial site https: //web.mit.edu
/torralba/www/indoor.html .
Indoor67 is released under license that allows it to be used for non-commercial research purposes (see https://web.mit.edu/ torralba/www/indoor.html ).
OxfordFlower (Nilsback & Zisserman, 2008):
We downloaded OxfordFlower from the ofﬁcial site https://www.robots.ox.ac.uk/˜vgg/data/flowers/102/ .
OxfordFlower is released under unknown license.
OxfordPets
(Parkhi et al., 2012):
We downloaded OxfordPets from the ofﬁcial site https:// www.robots.ox.ac.uk/˜vgg/data/pets/ .
OxfordPets is released under Creative Commons Attribution-ShareAlike 4.0 International License.
StanfordCars (Krause et al., 2013):
We downloaded StanfordCars from the ofﬁcial site https: //ai.stanford.edu/˜jkrause/cars/car_dataset.html .
StanfordCars is released under license that allows it to be used for non-commercial research purposes (see https://ai.
stanford.edu/˜jkrause/cars/car_dataset.html ).
StanfordDogs (Khosla et al., 2011):
We downloaded StanfordDogs from the ofﬁcial site http: //vision.stanford.edu/aditya86/ImageNetDogs/ .
StanfordDogs is released under license that allows it to be used for non-commercial research/educational purposes (see https: //image-net.org/download.php ).
CompCars (Yang et al., 2015): We downloaded CompCars from the ofﬁcial site http://mmlab.
ie.cuhk.edu.hk/datasets/comp_cars/ .
CompCars is released under license that allows it to be used for non-commercial research purposes (see http://mmlab.ie.cuhk.edu.hk/ datasets/comp_cars/ ).
PASCAL-VOC (Everingham et al., 2015):
We downloaded PASCAL-VOC from the ofﬁcial site http://host.robots.ox.ac.uk/pascal/VOC/ .
PASCAL-VOC is released under license of ﬂickr (see https://www.flickr.com/help/terms ).
B. K NOWLEDGE D ISTILLATION B ASELINES
As stated in the main paper, we evaluated our method by comparing it with na¨ıve knowledge distillation methods Logit Matching (Ba & Caruana, 2014) and Soft Target (Hinton et al., 2015).
We exploited Logit Matching and Soft Target for transfer learning under the architecture inconsistency since their loss functions use only ﬁnal logit outputs of models regardless of the intermediate layers.
To tranfer knowledge in C A s s to C A t t , we ﬁrst ﬁne-tune the parameters of C A s s on the target task then train C t with knowledge distillation penalties by treating
the trained C A s s as the teacher model.
We optimized the knowledge distillation models by the following objective function:
[Equation]
where λ d is a hyperparameter, L KD is a loss function of a knowledge distillation method, φ is the parameter of a teacher model, and l θ ( · ) and l φ ( · ) are the output of the logit function on θ and φ .
In Logit Matching, L KD is a simple mean squared loss between l θ and l φ .
Soft Target computes a
Kullback-Leibler divergence between the softmax output of l θ and the temperature softmax output of l φ as L KD .
We set the temperature parameter T to 4 by searching in { 2 , 4 , 6 } .
B. T RAINING S ETTINGS
We selected the training conﬁgurations on the basis of the previous works (Li et al., 2020a; Xie et al., 2020).
In PP, we trained a source classiﬁer by Neterov momentum SGD for 1M iterations with a mini-batch size of 128, weight decay of 0.0001, momentum of 0.9, and initial learning rate of 0.1; we decayed the learning rate by 0.1 at 30, 60, 90 epochs.
We trained a target classiﬁer C A t t by
Neterov momentum SGD for 300 epochs with a mini-batch size of 16, weight decay of 0.0001, and momentum of 0.9.
We set the initial learning rate to 0.05 for the scratch models and 0.005 for the models with PP.
We dropped the learning rate by 0.1 at 150 and 250 epochs.
For each target dataset, we split the training set into 9 : 1 , and used the former in the training and the later in validating.
The input samples were resized into 224 × 224 resolution.
For the SSL algorithms, we set the minibatch size for L unsup to 112, and ﬁxed λ in Eq.
(2) to 1.0.
We ﬁxed the hyperparameters of UDA as the conﬁdence threshold β = 0 .
5 and the temperature parameter τ = 0 .
4 following Xie et al.
For P-SSL, we generated 50,000 samples by PCS.
We ran the target trainings three times with different seeds and selected the best models in terms of the validation accuracy for each epoch.
We report the average top-1 test accuracies and standard deviations.
B. F ILTERING OF R EAL D ATASET BY R ELATION TO T ARGET
We provide the details of the protocol of dataset ﬁltering discussed in Sec.
4.5 of the main paper and list the correspondences between the target classes and selected source classes.
For ﬁltering datasets, we ﬁrst calculated the conﬁdence (the maximum probability of a class in the prediction) of the target samples by using the pre-trained source classiﬁers, then averaged the conﬁdence scores for each target class, and ﬁnally selected the source classes with a conﬁdence higher than 0 .
001 as the unlabeled dataset similar to (Xie et al., 2020).
We list the ﬁltered ImageNet classes for each target dataset in Table 24 and 25.
B. S OURCE D ATASET : C OMP C ARS
CompCars (Yang et al., 2015) is a ﬁne-grained vehicle image dataset for classifying the vehicle manufacturers or models.
It contains 163 manufacturer classes, 1,716 model classes, and 136, images of entire vehicles collected from the web.
As the source dataset for PP and P-SSL, we used 163 manufacturer classes for training classiﬁers and conditional generative models since the manufacturer classes do not overlap with the classes of the target dataset, i.e., StanfordCars.
[Table]
[Table]
[Table]
C A DDITIONAL E XPERIMENTS C. C OMPARISON OF O UR M ETHOD WITH F INE - TUNING M ETHODS
For assessing the practicality of our method, we additionally compare our method with the ﬁnetuning methods that require architecture consistency, i.e., Fine-tuning : na ¨ ıvely training target classiﬁers by using the source pre-trained weights as the initial weights.
L2-SP (Li et al., 2018): ﬁnetuning with the L 2 penalty term between the current training weights and the pre-trained source weights.
DELTA (Li et al., 2019): ﬁne-tuning with a penalty minimizing the gaps of channel-wise outputs of feature maps between source and target models.
BSS (Chen et al., 2019): ﬁne-tuning with the penalty term enlarging eigenvalues of training features to avoid negative transfer.
CoTuning (You et al., 2020): ﬁne-tuning on source and target task simultaneously by translating the target labels to the source labels.
We implemented these methods on the basis of the open source repositories provided by the authors.
All of hyperparameters used in L2-SP, DELTA, BSS, and
Co-Tuning are those in the respective papers: β for L2-SP and DELTA was 0.01, η for BSS was 0.001, and λ for Co-Tuning was 2.3.
We also tested the models by combining our methods and the ﬁne-tuning methods.
Table 12 and 13 list the extended results of the experiments on multiple architectures and target datasets in Secs.
4.3 and 4.4, respectively.
Table 10 summarizes the results using the ﬁne-tuning variants.
The +P-SSL column indicates the results of the combination models of a ﬁne-tuning method and P-SSL.
We conﬁrm that our method (pseudo pre-training + P-SSL) achieved competitive or superior results to the na¨ıve ﬁne-tuning.
This means that our method can improve target models as well as ﬁne-tuning without architectures consistency.
We also observed that P-SSL can outperform ﬁne-tuning baselines by being combined with them.
These results indicate that our
P-SSL can be applied even if the source and target architectures are consistent.
C.1. C OMPARSION TO F INE - TUNING D ISCRIMINATOR
We compare PP with a transfer learning method applying encoders of generative models as the pre-trained encoder for target tasks. Here, we assume the discriminator of ImageNet pre-trained BigGAN as the pre-trained encoder, and ﬁne-tune it on the target classiﬁcation task of StanfordCars.
Table 11 shows the results.
The BigGAN discriminator easily overﬁtted the target dataset and catastrophically degraded the test performance.
It seems difﬁcult to achieve high accuracy by na¨ıvely applying the discriminator as a simple pre-trained encoder.
This suggests that the representation of
[Table]
[Table]
the generative model is quite different from one of the classiﬁcation tasks and that our method is well suited to absorb such a difference between tasks for knowledge transfer
[Table]
C.1. C OMPARSION OF RN18 AND C USTOM A RCHITECTURE
We provide further comparison of RN-18 architecture and the custom architecture found by the architecture search in Sec. 4.2.
Here, we tested RN18 with our methods (PP, P-SSL, and PP+P-SSL) on the same setting as Sec.
4.2 to conﬁrm the superiority of the custom architecture.
We summarize the results in Table 16.
This shows that the custom architecture is superior to the original RN (4 , 4 , 4 , 4) in all settings, and also indicates that this is a fair comparison among methods.
C. T ARGET D ATASET S IZE
We evaluate the performance of our method on a limited target data setting.
We randomly sampled the subsets of the training dataset (StanfordCars) for each sampling rate of 25, 50, and 75%, and used them to train target models with our method.
Note that the pseudo datasets were created by using the reduced datasets.
Table 14 lists the results.
We observed that our method can outperform the baselines in all sampling rate settings.
C. S OURCE G ENERATIVE M ODELS
Table 17 lists the results with the architectures of multiple generative models.
We tested our method by using pseudo samples generated from the generative models for multiple resolutions including
SNGAN (Miyato et al., 2018), SAGAN (Zhang et al., 2019), and ADM-G (Dhariwal & Nichol, 2021).
We implemented these generative models on the basis of open source repositories including pytorch-pretrained-BigGAN 3 , Pytorch-StudioGAN 4 by Minguk et al.
(2021), and guided-diffusion 5 by Dhariwal & Nichol (2021); we used the pre-trained weights distributed by the repositories.
We measured the top-1 test accuracy on the target task (StanfordCars) and
Fr´echet Inception Distance (FID, Heusel et al.
In Table 17, the generative models with better FID scores tended to achieve a higher top-1 accuracy score with PP and P-SSL. Regarding
[Table] [Table] [Table] the resolution, the models of 256 × 256, the generated samples of which are the nearest to the input size of C A t t (224 × 224), were the best.
From these results, we recommend using generative models synthesizing high-ﬁdelity samples at a resolution close to the target models when applying our method.
C. A NALYSIS OF P SEUDO P RE - TRAINING
We analyze the characteristics of PP by varying the synthesized samples.
C.4. S YNTHESIZING S TRATEGY
We compare the four strategies using the source generative models for PP.
We tested Uniform : synthesizing samples for all source classes (default), Filtered : synthesizing samples for target-related source classes identiﬁed by the same protocol as in Sec. 4.5, PCS : synthesizing samples by PCS,
Ofﬂine : synthesizing ﬁxed samples in advance of training and training
C
A t s with them instead of sampling from G s .
In PCS, we optimized the pre-training models with pseudo source soft labels generated in the process of PCS.
Table 18 shows the target task performances.
We found that the
Uniform model achieved the best performance.
We can infer that, in PP, pre-training with various classes and samples is more important than that with only target-related classes or ﬁxed samples.
C. A NALYSIS OF P SEUDO S EMI - SUPERVISED L EARNING C.5. S EMI - SUPERVISED L EARNING A LGORITHM
We compare SSL algorithms in P-SSL.
We used six SSL algorithms: EntMin (Grandvalet & Bengio, 2005), Pseudo Label (Lee et al., 2013), Soft Pseudo Label, Consistency Regularization, FixMatch (Sohn et al., 2020), and UDA (Xie et al., 2020).
Soft Pseudo Label is a variant of Pseudo
Label, which uses the sharpen soft pseudo labels instead of the one-hot (hard) pseudo labels.
Consistency Regularization computes the unsupervised loss of UDA without the conﬁdence thresholding.
Table 19 shows the results on StanfordCars, where Pseudo Supervised is a model using a pairs of ( x s ← t , y t ) in pseudo conditional sampling for supervised training.
UDA achieved the best result.
[Table]
[Table]
[Figure] [Table] [Figure] More importantly, the methods using hard labels (Pseudo Supervised, Pseudo Label, and FixMatch) failed to outperform the scratch models, whereas the soft label based methods improved the performance.
This indicates that translating the label of pseudo samples as the interpolation of the target labels can improve the performance as mentioned in Sec.
C.5. S AMPLE S IZE
We evaluate the effect of the sizes of pseudo datasets for P-SSL on the target test accuracy.
We varied the pseudo dataset sizes in { 10K, 50K, 100K, 500K } and tested the target performance of
P-SSL on the StanfordCars dataset, as shown in Figure 4 (right).
We found that the middle range of the dataset size (50K and 100K images) achieved better results.
This suggests that P-SSL does not require generating extremely large pseudo datasets for boosting the target models.
C.5. O UTPUT L ABEL F UNCTION
We discuss the performance comparison of output label functions in PCS.
The output label function is crucial for synthesizing the target-related samples from source generative models since it directly determines the attributes on the pseudo samples.
We tested six labeling strategies, i.e., Random
Label : attaching uniformly sampled source labels, Softmax : using softmax outputs of C A s s (default), Temperature Softmax : applying temperature scaling to output logits of C A s s and using the softmax output, Argmax : using one-hot labels generated by selecting the class with the maximum probability in the softmax output of C A s s , Sparsemax (Martins & Astudillo, 2016): computing the
Euclidean projections of the logit of C t representing sparse distributions in the source label space, and Classwise Mean : computing the mean of softmax outputs of C A s s for each target class and using it as representative pseudo source labels of the target class to generate pseudo samples.
Tableshows the comparison of the labeling strategies.
Among the strategies, Softmax is the best choice for
PCS in terms of the target performance (top-1 accuracy) and the relatedness toward target datasets (FID).
This means that the pseudo source label y s ← t by Softmax succeeds in representing the characteristics of a target sample x t and its form of the soft label is important to extract target-related information via a source generative model G s . [Table] [Table]
S EARCHING T EMPERATURE H YPERPARAMETER FOR UDA
Here, we investigate the effect of hyperparameter τ for UDA, which is mainly used in the main paper.
By the deﬁnition of temperature softmax function ( exp( y i /τ ) / (cid:80)
Kj exp( y j /τ ) ), the temperature parameter τ controls the sharpness of the predicted class-conditional distribution; lower temperature outputs sharper distribution.
In this regard, in our P-SSL, ˆ C t ( x s ← t , τ ; θ )
[Table]
which determines how to represent the pseudo sample x s ← t with the soft target class labels, can be changed by τ .
Thus
[Table]
the choice of τ can be an important factor for training in P-SSL.
To evaluate the effect, we tested
{+P-SSL by varying τ on StanfordCars as shown in Table 
+}
The results show that the moderately higher τ achieved better target performances.
This suggests that representing x s ← t by softer target class labels can bring a positive effect to target classiﬁers, which is consistent with the discussions in Sec.
3.2.1 and 3.2.3.
C
[Equation]
[Table]
[Table]
We provide further analysis to conﬁrm the performance of the unlabeled pseudo samples x s ← t by comparing with a SSL method using real target samples in the unsupervised loss function.
We call this method as T-SSL, and T-SSL can be another simple baseline for
P-SSL because it can simply discard pseudo sample synthesis in
P-SSL.
The results are shown in Table 22.
We can see that applying supervised loss and unsupervised loss to the same sample has a negative effect: the use of pseudo-labels to target labeled data might promote overﬁtting.
This result indicates that x s ← t in P-SSL certainly performs as useful unlabeled data in UDA than directly using target data.
D Q UALITATIVE E VALUATION OF P SEUDO S AMPLES
We discuss qualitative studies of the pseudo samples generated by PCS.
To conﬁrm the correspondences between the target and pseudo samples, we used StanfordCars as the target dataset and generated samples from BigGAN with the same setting as in Sec. 4.
Figure 5 shows the visualizations of the source dataset (ImageNet), target dataset (StanfordCars), and pseudo samples generated by PCS.
The samples were randomly selected from each dataset.
We can see that PCS succeeded in generating target-related samples from the target samples.
To assess the validity of using pseudo source soft labels in PCS, we analyzed the pseudo samples corresponding to each target label.
Figureshows the pseudo samples generated by the target samples of Hummer and Aston Martin V
Convertible classes in StanfordCars.
We conﬁrm that the pseudo samples by PCS can capture the features of target classes.
This also can conﬁrm the ranking of the conﬁdence scores for source classes listed in Table 23; the pseudo source soft labels seem to represent the target samples by the interpolation of source classes.
[Figure]
StanfordCars
PCS
StanfordCars
PCS
T a r g e t L a b e l:
Hummer
T a r g e t L a b e l:
Aston
Martin
V
Convertible
[Table] [Table]
[Table]