O NE -S HOT G ENERATIVE D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
[Figure]
A BSTRACT
This work aims at transferring a Generative Adversarial Network (GAN) pretrained on one image domain to a new domain referring to as few as just one target image .
The main challenge is that, under limited supervision, it is extremely difﬁcult to synthesize photo-realistic and highly diverse images, while acquiring representative characters of the target.
Different from existing approaches that adopt the vanilla ﬁne-tuning strategy, we import two lightweight modules to the generator and the discriminator respectively.
Concretely, we introduce an attribute adaptor into the generator yet freeze its original parameters, through which it can reuse the prior knowledge to the most extent and hence maintain the synthesis quality and diversity.
We then equip the well-learned discriminator backbone with an attribute classiﬁer to ensure that the generator captures the appropriate characters from the reference.
Furthermore, considering the poor diversity of the training data ( i.e. , as few as only one image), we propose to also constrain the diversity of the generative domain in the training process, alleviating the optimization difﬁculty.
Our approach brings appealing results under various settings, substantially surpassing state-of-the-art alternatives, especially in terms of synthesis diversity.
Noticeably, our method works well even with large domain gaps, and robustly converges within a few minutes for each experiment.
I NTRODUCTION
Generative Adversarial Network (GAN) (Goodfellow et al., 2014), consisting of a generator and a discriminator, has signiﬁcantly advanced image synthesis yet relies on a large number of training samples (Karras et al., 2018; 2019; 2020b; Brock et al., 2018).
Many attempts have been made to train GANs from scratch with limited data (Zhang & Khoreva, 2019;
Zhao et al., 2020d;b; Karras et al., 2020a; Yang et al., 2021a), but it still requires hundreds or thousands of images to get a satisfying performance.
Sometimes, however, we may have as few as only one single image as the reference, like the masterpiece Mona Lisa from Leonardo da Vinci.
Under such a case, learning a generative model with both good quality and high diversity becomes extremely challenging.
Domain adaptation is a commonly used technique that can apply an algorithm developed on one data domain to another (Csurka, 2017).
Prior works (Wang et al., 2018; Noguchi & Harada, 2019; Wang et al., 2020; Mo et al., 2020; Zhao et al., 2020a;
Li et al., 2020;
Robb et al., 2020) have introduced this technique to GAN training to alleviate the tough requirement on the data scale.
Typically, they ﬁrst train a large-scale model in the source domain with adequate data, and then transfer it to the target domain with only a few samples.
A common practice is to ﬁne-tune both the generator and the discriminator on the target dataset until the generator produces samples conforming to the target domain.
To stabilize the ﬁne-tuning process and improve the generation quality and diversity, existing approaches propose to tune partial parameters (Noguchi & Harada, 2019; Mo et al., 2020;
Robb et al., 2020) and introduce some regularizers (Li et al., 2020; Ojha et al., 2021), but the overall adaptation strategy remains.
When there is only one image from the target domain, these methods would fell short of synthesis diversity, producing very similar images.
Recall that the pre-trained model can already produce highly diverse images from the source domain.
It makes us wonder what indeed causes the diversity drop in the adaptation process.
We argue that directly tuning the model weights will result in the loss of the prior knowledge gained from the large-scale data.
On the other hand, however, when adapting the model to the target domain, most variation factors ( e.g. , gender and pose of human faces) could be reused.
These observations help raise a question: is it possible to simply focus on the most representative characters of the reference image while inheriting all the other knowledge from the source model?
In this work, we develop a method, called GenDA , for one-shot Generative Domain Adaptation.
In particular, we design a lightweight module connecting the latent space and the synthesis network.
We call this module an attribute adaptor since it helps adapt the generator with the attributes of the target image.
Unlike the conventional ﬁne-tuning strategy, we freeze the parameters of the original generator and merely optimize the attribute adaptor during training.
Thereby, we manage to reuse the prior knowledge learned by the source model and hence inherit the synthesis quality and, more importantly, diversity.
Meanwhile, we employ the discriminator to compete with the generator via a domain-speciﬁc attribute classiﬁcation.
In this way, the generator is forced to capture the most representative attributes from the reference, or otherwise, the discriminator would spot the discrepancy.
However, instead of directly tuning the original discriminator, we freeze its entire backbone and introduce a lightweight attribute classiﬁer on top of that.
Similar to the generator, the discriminator has also learned rich knowledge in its pre-training.
Since the synthesized images before and after adaptation share most visual concepts ( e.g. , a face model would still produce faces after domain transfer), the discriminator can be reused as a well-learned feature extractor.
Therefore, we simply train the attribute classiﬁer to help guide the generator.
Furthermore, since there is only one training sample (which means no diversity in the target domain), we propose to also constrain the diversity of the generative domain by truncating the latent distribution during training.
Intuitively, learning a one-to-one mapping would be easier than learning a many-to-one mapping.
Such a design mitigates the optimization difﬁculty and further improves the synthesis quality.
We evaluate our approach through extensive experiments on human faces and outdoor churches.
Given only one training image, GenDA can convincingly adapt the source model to the target domain with sufﬁciently high quality and diversity.
Such an adaptation includes both attribute-level and style-level, as shown in Fig.
Our method outperforms the state-of-the-art competitors by a large margin both qualitatively and quantitatively.
We also show that, when the number of the samples available in the target domain increases, GenDA can ﬁlter out the individual attributes and only preserve their common characters (see Fig. 4).
Noticeably, GenDA works well for the extreme cases where there is a large domain gap, like transferring the characters of Mona Lisa to churches (see
Fig. 5).
Besides, thanks to the lightweight design of both the attribute adaptor and the attribute classiﬁer, GenDA can ﬁnish each adaptation experiment within a few minutes .
M ETHODOLOGY
The primary goal of this work is to transfer a pre-trained GAN to synthesize images conforming to a new domain with as few as only one reference image.
Due to the limited supervision, it is challenging to ensure both high quality and large diversity of the synthesis.
Intuitively, according to the rationale of GANs ( i.e. , adversarial training between the generator and the discriminator), the discriminator can easily memorize that only the reference image is real while all others are fake.
In this way, to fool the discriminator, the generator may have to learn to produce images highly alike the reference, resulting in a poor synthesis diversity.
To mitigate this problem, we propose a new adaptation algorithm, different from the previous ﬁne-tuning scheme.
Concretely, we (i) interpose an attribute adaptor between the latent space and the generator to help acquire the most representative characters from the target image; (ii) augment the discriminator backbone with an attribute classiﬁer to guide the generator to make appropriate adjustments; and (iii) propose a diversity-constraint training strategy.
Before going into technical details, we ﬁrst give some preliminaries of GANs.
2. P RELIMINARIES
Generative Adversarial Network (GAN) (Goodfellow et al., 2014) is formulated as a two-player game between a generator and a discriminator.
Given a collection of observed data { x i } Ni =1 with
N samples, the generator G ( · ) aims at reproducing the real distribution X via randomly sampling latent codes z subject to a pre-deﬁned latent distribution Z .
As for the discriminator D ( · ) , it targets at differentiating the real data x and the synthesized data G ( z ) as a bi-classiﬁcation task.
These two models are jointly optimized by competing with each other, as
[Equation]
After the training converges, the generator is expected to produce images as realistic as the training set, so that the discriminator cannot distinguish them anymore.
In this work, we start with a GAN model that is well trained on a source domain X src , and work on adapting it to a target domain X dst = { x dst } that has only one image.
In fact, it is ambiguous to deﬁne a “domain” using one image.
We hence expect the model to acquire the most representative characters from the reference image.
Taking face synthesis as an example, the character can include facial attributes ( e.g. , age or wearing sunglasses) and artistic styles, as shown in Fig.
2. O NE -S HOT G ENERATIVE D OMAIN A DAPTATION
A common practice to transfer GANs is to simultaneously tune the generator and the discriminator on the target dataset (Wang et al., 2018).
However, as discussed above, the transferring difﬁculty increases drastically when given only one training sample.
Existing methods attempt to address this issue by reducing the number of learnable parameters (Mo et al., 2020;
Robb et al., 2020) and introducing training regularizers (Ojha et al., 2021).
Even so, the overall ﬁne-tuning scheme ( i.e. , directly tuning G ( · ) and D ( · ) ) remains and the diversity is low.
Differently, we propose a new adaptation pipeline to preserve the synthesis diversity, which includes an attribute adaptor, an attribute classiﬁer, and a diversity-constraint training strategy.
Details are introduced as follows.
Attribute Adaptor.
Prior works have found that, a well-learned generator is able to encode rich semantics to produce diverse images (Shen et al., 2020; Yang et al., 2021b).
For instance, a face synthesis model could capture the variation factors like gender, age, wearing glasses, lighting, etc .
Ideally, this knowledge should be sufﬁciently reused as much as possible in the target domain, and then the synthesis diversity can be preserved accordingly.
In this way, the generator focuses on transferring the most distinguishable characters of the only reference, instead of learning the common variation factors repeatedly and leading to overﬁtting due to lack of samples.
This helps improve the data efﬁciency signiﬁcantly, which is vital to the one-shot setting.
According to Xu et al.
(2021), the latent code z can be viewed as the generative feature of G ( z ) that determines the multi-level attributes of the output image.
Motivated by this, we propose to adapt such features regarding the reference image, yet keep the convolutional kernels untouched.
Concretely, before feeding the latent code z to the generator G ( · ) .
we propose to ﬁrst transform it through a lightweight attribute adaptor A ( · ) , as
[Equation]
where (cid:12) stands for the element-wise multiplication, while a and b are the learnable weight and bias respectively.
With such a design, the transformed latent code z (cid:48) is assumed to carry the sufﬁcient information of the reference, and therefore G ( z (cid:48) ) would conform to the target domain X dst .
Attribute Classiﬁer.
Only having the attribute adaptor cannot guarantee the generator to acquire the representative characters from the training sample.
Following the formulation of GANs, we incorporate the discriminator D ( · ) , which is also pre-trained on the source domain, to compete with the generator.
In particular, we reuse the backbone d ( · ) but remove the last real/fake classiﬁcation head, and then equip it with a lightweight attribute classiﬁer φ ( · ) .
Given an image x , either the reference image x dst or a transferred synthesis G ( A ( z )) , the classiﬁer outputs a probability of how likely it possesses the target attribute, as
[Equation]
However, due to the limited supervision provided by one image, the discriminator can easily memorize the real data, which leads to the overﬁtting of discriminator as well as the collapse of the generator (Karras et al., 2020a).
As discussed above, the generated images before and after domain adaptation are expected to share most variation factors ( i.e. , a face model remains to produce faces after adaptation).
From this viewpoint, the knowledge learned by the discriminator in its pretraining could be also reused.
Therefore, unlike existing approaches that ﬁne-tune all or partial parameters of D ( · ) (Mo et al., 2020; Wang et al., 2020; Li et al., 2020; Ojha et al., 2021), we freeze all parameters of d ( · ) in the entire training process and merely optimizes φ ( · ) to guide A ( · ) with adequate adjustments.
Regarding the one-shot target domain, the mechanism behind the classiﬁer is very similar to Exemplar SVM (Malisiewicz et al., 2011), which also suggests that it is sufﬁcient to obtain a good decision boundary with one positive and many negative samples.
Differently, our attribute classiﬁer is learned in an adversarial manner through competing with the attribute adaptor.
Diversity-constraint Strategy.
Recall that this work targets at generative domain adaptation with only one reference image, which means no diversity of real data.
On the contrary, however, the latent code can be sampled randomly and the pre-trained generator can produce highly diverse images from the source domain.
From this perspective, it might be challenging to match these two distributions with such a huge diversity gap.
To alleviate the optimization difﬁculty, we propose a diversityconstraint strategy, which retains the diversity of the generator during training.
Speciﬁcally, we truncate the latent distribution with a strength factor β , as
[Equation]
) where ¯ z indicates the mean code.
Note that, truncation is a common trick used in the inference of state-of-the-art GANs, like StyleGAN (Karras et al., 2019) and
BigGAN (Brock et al., 2018), to improve synthesis quality.
Nevertheless, to our best knowledge, this is the ﬁrst time that truncation is introduced in the training process to preserve the synthesis diversity.
Full Objective Function.
In summary, the adaptor A ( · ) and the classiﬁer φ ( · ) are trained with
[Equation]
E XPERIMENTS
We evaluate the proposed method on multiple datasets and settings.
In Sec. 3.1, we focus on one-shot generative domain adaptation.
Quantitative and qualitative results indicate that the proposed GenDA can produce much more diverse and photo-realistic images than previous alternatives.
Interestingly, the shared representative attributes of multiple shots could be also captured and transferred from the source to the target domain.
Sec.
3.2 presents the quantitative comparison to prior approaches, demonstrating the effectiveness of our GenDA under the general few-shot setting.
Additionally, when there exists a large domain gap, GenDA can still synthesize reasonable outputs in Sec. 3.3.
Noticeably, the comprehensive ablation studies of each component are available in Appendix .
Implementation.
We choose the state-of-the-art StyleGAN2 (Karras et al., 2020b) as our base
GAN model, following Ojha et al.
StyleGAN2 proposes a more disentangled W space in addition to the native latent space Z .
Here, our attribute adaptor is deployed on the W space.
¯ z in Eq.
(5) becomes w avg , which is a statistical average in the training of the source generator.
To alleviate the overﬁtting problem, we apply differentiable augmentation on both the reference image and the adapted synthesis (Zhao et al., 2020b; Karras et al., 2020a).
In particular, we adopt the augmentation pipeline from StyleGAN2-ADA (Karras et al., 2020a) and linearly increase the
[Table]
augmentation strength from 0 to 0.6.
We use Adam optimizer (Kingma & Ba, 2014) for training and the learning rates are set as 1 . 25 e − 4 and 2 .
5 e − 4 for A ( · ) and φ ( · ) , respectively.
All experiments are stopped when the discriminator has seen 200 K real samples.
The last “ToRGB” layer (with 1 ×convolution kernels) of the generator is tuned together with the adaptor A ( · ) to make sure aligning the synthesis color space to the reference image.
The hyper-parameter β in Eq.
(5) is set as 0.7.
Datasets.
Our source models are pre-trained on large-scale datasets ( e.g. , FFHQ (Karras et al., 2019) and LSUN Church (Yu et al., 2015)) with resolution 256 × 256 .
We collect target images from
FFHQ-babies, FFHQ-sunglasses, face sketches (Ojha et al.
(2021)), and samples from masterpieces ( i.e. , Mona Lisa and Van Gogh’s houses) and Artistic-Faces dataset.
Evaluation Metrics.
Fr´echet Inception Distance (FID) (Heusel et al., 2017) serves as the main metric.
Akin to Ojha et al.
(2021), we always calculate the FID between 5,000 generated images and all the validation sets.
Additionally, we report the precision and recall metric (Kynk¨a¨anniemi et al., 2019) to measure the quality and diversity respectively.
3. O NE - SHOT D OMAIN A DAPTATION
Comparison with Existing Alternatives.
For one-shot generative domain adaptation, we compare against FreezeD (Mo et al., 2020) and Cross-Domain (Ojha et al., 2021).
Noticeably, the implemented FreezeD is further improved by the data augmentations, which is already a challenging baseline.
Considering the variance brought by different single shots, we calculate all metrics overtraining shots.
As shown in Tab. 1, our GenDA remains to surpass multiple approaches by a clear gap from the perspective of FID.
Besides, to further compare different methods from the views of image quality and diversity, we also report the precision and recall.
Intuitively, a higher precision means
[Figure]
higher image quality (more closer to the real sample), while a higher recall indicates higher diversity.
Although FreezeD (Mo et al., 2020) and Cross-Domain (Ojha et al., 2021) plausibly achieves better synthesis quality on sunglasses and sketches, their low recalls strongly imply overﬁtting.
Fig.conﬁrms that there is insufﬁcient diversity for such two methods.
As a contrast, our GenDA results in the competitive quality and standing-out diversity from the quantitative and qualitative perspectives.
In particular, GenDA manages to synthesize a man with a cap which is only available in prior knowledge.
Fig.
3 suggests that our GenDA works on transferring both attributes ( sunglasses, gender, vegetation and material ) and artistic styles for face and church model respectively.
Common Attributes from Multiple References.
There might be a number of representative variation factors in a given face image ( e.g. , age, gender, smile and sunglasses).
Therefore, when transferring a pre-trained generative model on one image, multiple factors could be adapted together.
Fig.
4 provides an example of our GenDA on sunglasses.
Speciﬁcally, we train GenDA on multiple target domains which might contain a single individual with different identities or a pair of images.
Compared with the source output (the ﬁrst row), representative attributes besides sunglasses are also transferred.
For instance, the second and third rows suggest that although all individuals wear sunglasses, the target’s gender is also adapted to the new synthesis.
The third shot could even affect the gender and hairstyles.
This reveals that our GenDA could capture multiple representative attributes of the target domain.
When the target domain contains more than one shot like the last two rows of Fig. 4, the representative attributes become the common attributes of all individuals, leading to the corresponding results ( i.e. , sunglasses and gender).
Namely, our GenDA is able to capture and adapt the representative attributes no matter how many images the target domain has.
[Figure]
Before
[Figure]
3. 10- SHOT D OMAIN A DAPTATION
As our GenDA could capture the representative attributes precisely with an increasing number of target shots, we also compare again prior approaches under the general few-shot adaptation setting.
Following the setting of Ojha et al.
(2021), we choose 10-shot target domain for comparison.
Quantitative Comparison.
Multiple baselines are introduced to conduct the quantitative comparisons, including Transferring GANs (TGAN)
(Wang et al., 2018), Batch Statistics Adaptation (BSA) (Noguchi & Harada, 2019), FreezeD
(Mo et al., 2020), MineGAN (Wang et al., 2020),
EWC (Li et al., 2020) and Cross-Domain (Ojha et al., 2021).
All these methods adapt a pre-trained source model to a target domain ( e.g. , babies, sunglasses and sketches).
Akin to Cross-domain (Ojha et al., 2021), FID serves as the metric for evaluation.
Tab. 2 presents the 10-shot quantitative comparison on several target domains.
Although CrossDomain (Ojha et al., 2021) has already obtained the competitive results, our method could still signiﬁcantly improve the performance by a clear margin, resulting in the new state-of-the-art synthesis quality on few-shot adaptation.
Speciﬁcally, for domains that differ in semantic attributes ( i.e. , babies and sunglasses), huge gains are obtained by GenDA.
It is also worth noting that when the domain gap increases ( i.e. , from realistic facial images to sketches), the performances of Crossdomain (Ojha et al., 2021) and FreezeD (Mo et al., 2020) become almost identical (45.67 v.s 46.54) while substantial improvements of synthesis could remain observed by our method (31.97).
It demonstrates the effectiveness of our method on the general few-shot settings.
[Table]
[Figure]
3. C ROSS -D OMAIN A DAPTATION
In this part
[Table]
we study the adaptation on unrelated source and target domains.
Speciﬁcally, Van
Gogh’s houses, Superman and Mona Lisa serve as the target for a face and church source model respectively.
Considering the motivation that we aim at reusing the prior knowledge ( i.e. , variation factors) by freezing the parameters, the synthesis after adaptation is supposed to share similar visual concepts.
That is, a face model would still produce faces no matter what the target image is.
Fig.suggests that the source models remain to produce the corresponding content.
More importantly, the color scheme and painting styles are also transferred.
For example, the red roof at the ﬁrst row renders the red glasses, the yellow sky at the second row draws the front head in yellow.
The blue hair, green background and the shadows of Superman are well adapted to church.
The painting style of Mona Lisa is also transferred.
Obviously, the shared attributes between face and church are quite rare.
Therefore, GenDA pours more attention to the variation factors like color scheme, texture and painting styles which could be directly transferred across unrelated domains.
From this perspective, our GenDA might enable a new alternative for neural style transfer task that aims at transferring the styles of a given image.
But more generally, our GenDA is able to transfer more high-level attributes like gender, age, and sunglasses while the technique of style transfer might fail, which might be of beneﬁt to art creation.
R ELATED W ORK AND L IMITATIONS
Training Generative Models with Limited Data.
Many attempts have been taken to train a generative model on limited data.
For one thing, some of the prior approaches proposed to leverage the data augmentation to prevent the discriminator from overﬁtting.
Speciﬁcally, Zhang & Khoreva (2019) introduced a type of progressive augmentations.
Zhao et al.
(2020d) investigated the effects of various augmentations during the training.
Theoretical analysis was conducted by Tran et al.
(2021) for several data augmentations.
Zhao et al.
(2020b) proposed to apply the augmentations to both the real and synthesized images in a differentiable manner.
Karras et al.
(2020a) designed an adaptive discriminator augmentation that does not leak to stabilize the training process.
For another, multiple regularizers were also introduced to provide extra supervision.
For instance, Zhao et al.
(2020c) involved the consistency regularization for GANs which shows competitive performances with limited data.
Yang et al.
(2021a) incorporated contrastive learning as an extra task to improve the data efﬁciency.
Karras et al.
(2020a) also pointed out that decreasing the number of parameters of generators and introducing the dropout (Srivastava et al., 2014) could alleviate the overﬁtting problems.
Shaham et al.
and Sushko et al.
(2021) designed different frameworks which learn from a single natural image or video.
However, when the number of available images is less than 10, they usually lead to unsatisfying diversity.
Different from these works which learn from scratch, we focus on the generative domain adaptation, a practical alternative that ﬁrst pre-trains a source model on the large-scale dataset and then transfers it on a target domain with the one-shot image.
Few shot Generative
Domain Adaptation
Generative domain adaptation has attracted a considerable number of interests due to its practical importance.
Wang et al.
(2018) proposed to use the same objective for adaptation.
Noguchi & Harada (2019) ﬁne-tuned the batch statistics merely for the few-shot adaption.
Wang et al.
(2020) transformed the original latent space and tuned the entire parameters for the target domain.
Mo et al.
(2020) froze the lower-level representations of the discriminator to prevent overﬁtting.
Zhao et al.
(2020a) revealed that low-level ﬁlters of both the generator and discriminator can be transferred via a new adaptive ﬁlter modulation.
Li et al.
penalized certain weights identiﬁed by Fisher information.
Robb et al.
(2020) learned to adapt the singular values of the pre-trained weights while freezing the corresponding singular vectors.
Ojha et al.
(2021) proposed the cross-domain consistency as a regularization to maintain the diversity.
Different from prior work, we aim at reusing the learned knowledge on the source domain and thus identify and adapt the representative attributes, which enables the one-shot domain adaptation and outperforms other alternatives by a clear margin under the general few-shot settings.
Limitations.
Despite the state-of-the-art performances on both one-shot and few-shot generative domain adaptation, our proposed GenDA still has some limitations.
For example, the rationale behind GenDA is to reuse the prior knowledge learned by the source GAN model, which hinders it from transferring a model to a completely different domain.
As suggested in Fig.
5, when we adapt a church model regarding a face image, the outputs are still churches but not faces.
This implies that our method would fail when the inter-subclass variations are huge.
Such a property is a silver lining, depending on the practical application.
A second limitation is that our current design treats all characters of the reference image as a whole.
Taking the ﬁrst row of Fig.
3 as an example, the sunglasses, skin color, and background are transferred simultaneously.
It is hard to accurately transfer some particular attributes.
However, it is indeed possible to use some auxiliary samples to help deﬁne a common attribute, as shown in Fig. 4.
Besides, our GenDA also relies on the layer-wise stochasticity involved in the generator structure.
Concretely, in our base model,
StyleGAN2 (Karras et al., 2020b), the latent code is fed into all convolutional layers instead of the ﬁrst layer only.
Without the layer-wise design, the supervision will be hard to back-propagate to the attribute adaptor given a deep synthesis network.
Fortunately, however, such a design is commonly adopted by the state-of-the-art GANs (Karras et al., 2019; 2020b; Brock et al., 2018).
C ONCLUSION
In this work, we propose GenDA for one-shot generative domain adaptation.
We introduce two lightweight modules, i.e. , an attribute adaptor and an attribute classiﬁer , to the ﬁxed generator and discriminator respectively.
By efﬁciently learning these two modules, we manage to reuse the prior knowledge and hence enable one-shot transfer with impressively high diversity.
Our method demonstrates substantial improvements over existing baselines in a wide range of settings.
E THICS S TATEMENT
Training a generative model with limited data is a fundamental and practical problem.
This work signiﬁcantly advances this ﬁeld by using as few as only one image for domain adaptation.
Such an improvement is of great importance to the community.
Firstly, it pushes the data efﬁciency in GAN training to the limit and demonstrates an early success.
We believe more and more methods will be proposed to keep improving the training efﬁciency along this direction.
Secondly, our approach is based on reusing the prior knowledge of a pre-trained GAN model.
It provides insights on how to better utilize the pre-trained generative model for various downstream tasks (like semantic manipulation and super resolution), which is a widely studied research area.
Thirdly, this work also enables many potential real-world applications, like data augmentation (to solve the long-tail problem), entertainment, creative design, artistic imitation, etc .
However, every coin has two sides.
Our method considerably reduces the cost of producing fake data, in terms of using less data, less time, and fewer computing resources.
But we believe that its academic contribution and its positive impact on the industrial algorithm are more valuable, and that the negative impact will be lowered along with the development of deep fake detector.
R EPRODUCIBILITY S TATEMENT
The base model used in this work is StyleGAN2 (Karras et al., 2020b), with ofﬁcial implementation stylegan2-ada-pytorch.
The large-scale source datasets we use are FFHQ and LSUN
Church.
For one-shot domain adaptation, the reference image can be easily customized by the users, either from online or from other datasets, since our approach only requires one training sample.
The implementation and training details are described at the beginning of Sec. 3, which are shared by all experiments.
Besides, all experiments are conducted with 8 GPUs.
The evaluation metrics, including FID (Heusel et al., 2017), precision and recall (Kynk ¨ a ¨ anniemi et al., 2019), also follow their ofﬁcial implementation, FID and improved-precision-and-recallmetric.
To ensure the reproducibility, the code and models will be made publicly available.
A PPENDIX
This appendix is organized as follows.
Sec.
A conducts ablation study on the proposed approach, including the attribute adaptor , the attribute classiﬁer , the diversity-constraint strategy and the hyper-parameter β in Eq.
Sec.
B analyzes the transformation ( i.e. , the learnable vector a in
Eq.
(3)) for attribute adaptation, which provides insights on how our algorithm works.
Sec.
C shows how the latent representations are transformed after adaptation.
Sec.
D presents the additional comparison with the style transfer method and investigates the visual relation between and after adaptation.
Sec.
E performs latent interpolation with the models adapted on the target domain, which suggests that our GenDA does not harm the learned latent structure.
[Table]
A A BLATION S TUDY
Component Ablation.
In order to further investigate the role of each proposed component ( i
[Table]
e. , attribute adaptor, attribute classiﬁer and truncation strategy), we conduct comprehensive ablation studies on 10-shot adaptation.
In particular, we choose FreezeD
(Mo et al., 2020) as our baseline model which freezes the lower features of the discriminator while ﬁne-tunes the rest of the discriminator and the entire generator.
Moreover, we also apply adaptive discriminator augmentation strategy (ADA) on it since data augmentations usually help alleviate the overﬁtting problem.
As is shown in Tab. A1, the baseline alternative could achieve the satisfying synthesis quality but worse diversity, suggesting that it might overﬁt the training shot signiﬁcantly.
By involving the attribute adaptor, the baseline is substantially improved, which already outperforms prior approaches.
Meanwhile, the boosted diversity also demonstrates our motivation that we could reuse the variation factors of the source model and hence maintain the synthesis diversity to some extent. [Table] [Table]
However, the quality ( i.e. , precision) on babies and sketches becomes worse.
After being equipped with the attribute classiﬁer, the overall measurement is further enhanced.
Together with the diversityconstraint strategy, our GenDA could facilitate the quality to the baseline level but achieve better diversity, leading to the new state-of-the-art few-shot adaptation performances. Architectural Ablation.
To further study the effect of the lightweight design of the attribute adaptor and the attribute classiﬁer, we conduct experiments of replacing the lightweight module with twolayer multi0layer perceptron (MLP), which has a higher learning capacity. Tab. A2 presents the results. Obviously, with either a heavier attribute adaptor or a heavier attribute classiﬁer, the synthesis performance drops signiﬁcantly.
This might be caused by the overﬁtting problem since we only have one training image under our one-shot setting.
It veriﬁes the effectiveness and necessity of our lightweight design for the challenging one-shot generative domain adaptation task.
Hyper-parameter Ablation. We also conduct the ablation of hyper-parameter β in Eq.
Speciﬁcally, we train our model with 10-shot sunglasses images and maintain the evaluation metrics used in Sec. 3.2.
As is shown in Tab. A3, when increasing the value of β , the synthesis quality becomes worse while the diversity is enhanced.
Noticeably, when the β is less than 0.5, the training diverges.
Therefore, we choose 0.7 in all experiments for its overall performance.
B A NALYSIS ON T RANSFORMATION V ECTOR
We also visualize the learned attributes via Principal components analysis (PCA).
Concretely, we ﬁrst choose 15 shots from 3 domains ( i.e. , babies, sunglasses and sketches).
The corresponding models are trained on them and 15 attribute vectors in Eq.
(3) are collected to perform PCA.
Fig.
A presents the PCA results.
Obviously, the attribute vectors of different shots but the same domain assemble closely, while there are obvious decision boundaries across domains.
Such interpretation happens to match our motivation that the attribute adaptor could capture the representative attributes and make the corresponding adjustment for a source model to multiple target domains.
C A NALYSIS ON T RANSFORMED L ATENT R EPRESENTATION
We also visualize how the latent representations change before and after adaptation, with the help of
PCA.
Speciﬁcally, we sample 2 K latent codes from the latent space after adaptation for each model used in Fig.
4 and Fig.
Fig.
A2 suggests that the distributions after adapting the source model to three different target domains are clearly separable.
Also, when using more than one reference image, which have common attributes, from the target domain, the transformed representations tend to locate at the overlapped region between the representations learned from each reference image independently.
Fig.
A3 shows the latent representation change after cross-domain adaptation, where the latent distribution of the source domain is successfully shifted to the target one.
[Figure]
D A DDITIONAL R ESULTS
Comparison with the style transfer task.
As discussed in Sec.
the proposed approach might be an alternative for style transfer.
Here, we qualitatively compare our approach with the state-ofthe-art method, SWAG (Wang et al., 2021) regarding the style transfer task.
The comparison results are included in Fig.
A4.
From the perspective of the color transfer, SWAG (Wang et al., 2021) does better since it could successfully transfer the representative color from the target image to the synthesis.
For example, the green of the vegetation and the pink of the skin appears in the transferred samples.
However, it also reveal its weakness that the transfer is inconsistent.
Speciﬁcally, the wall or the sky in church synthesis is in pink while our method pictures them in the same color.
This
SourceDomain
TargetDomain
Synthesis
After Adaptation
Representation
Change implies that the style transfer based methods could have no semantic concepts regarding to the content, leading to such inconsistency and obvious artifacts.
Therefore, it naturally cannot transfer higher-level attributes like sunglasses while ours could.
The visual relation before and after adaptation.
A byproduct of our proposed GenDA is that we manage to inherit some characteristics from the source model in the process of domain adaptation.
Here, we take the images synthesized by the source generator as the references, and feed the adapted generator with the same latent codes.
In this way, we can check how the synthesis varies before and after adaptation.
Fig. A5 and Fig.
A6 correspond to the settings of Fig.
4 and Fig.
5, respectively.
We can tell that many attributes are maintained in the adapting process, verifying our motivation, which is that the most of variation factors could be reused in the task of generative domain adaptation.
E I NTERPOLATION OF L ATENT V ECTORS
Here, we perform the semantic interpolation to evaluate the tuned generator.
Typically, the interpolation at the latent space (Radford et al., 2015) could interpolate the semantics of two images, which is widely used to measure the diversity of generative models.
Fig.
A7 and Fig.
A8 present the semantic interpolation of face and church model respectively.
In particular, all models are ﬁne-tuned with one shot image.
Obviously, satisfying interpolation i.e. , smoothing change is clearly observed, even if the model is ﬁne-tuned with one shot image. [Figure] [Figure]
[Figure]