HyperDomainNet: Universal Domain Adaptation for Generative Adversarial Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
Domain adaptation framework of GANs has achieved great progress in recent yearsas a main successful approach of training contemporary GANs in the case of verylimited training data.
In this work, we significantly improve this framework byproposing an extremely compact parameter space for fine-tuning the generator.
We introduce a novel domain-modulation technique that allows to optimize only6 thousand-dimensional vector instead of 30 million weights of StyleGAN2 toadapt to a target domain.
We apply this parameterization to the state-of-art domainadaptation methods and show that it has almost the same expressiveness as thefull parameter space.
Additionally, we propose a new regularization loss thatconsiderably enhances the diversity of the fine-tuned generator.
Inspired by thereduction in the size of the optimizing parameter space we consider the problem ofmulti-domain adaptation of GANs, i.e. setting when the same model can adapt toseveral domains depending on the input query.
We propose the HyperDomainNetthat is a hypernetwork that predicts our parameterization given the target domain.
We empirically confirm that it can successfully learn a number of domains at onceand may even generalize to unseen domains.
Introduction
Contemporary generative adversarial networks (GANs) [7, 13, 14, 12, 2] show remarkable performance in modeling image distributions and have applications in a wide range of computer visiontasks (image enhancement [16, 39], editing [8, 28], image-to-image translation [11, 43, 44], etc.).
However, the training of modern GANs requires thousands of samples that limits its applicabilityonly to domains that are represented by a large set of images.
The mainstream approach to sidestepthis limitation is transfer learning (TL), i e. fine-tuning the generative model to a domain with fewsamples starting with a pretrained source model.
The standard approach of GAN TL methods is to fine-tune almost all weights of the pretrained model[17, 19, 35, 34, 12, 41, 21, 5, 45].
It can be reasonable in the case when the target domain is veryfar from the source one, e.g. when we adapt the generator pretrained on human faces to the domainof animals or buildings.
However, there is a wide range of cases when the distance between datadomains is not so far.
In particular, the majority of target domains used in works [17, 34, 21, 5, 45]are similar to the source one and differ mainly in texture, style, geometry while keep the same contentlike faces or outdoor scenes.
For such cases it seems redundant to fine-tune all weights of the sourcegenerator.
It was shown in the paper [37] that after transfer learning of the StyleGAN2 [14] to similardomains some parts of the network almost do not change.
This observation motivates us to find amore efficient and compact parameter space for domain adaptation of GANs.
In this paper, we propose a novel domain-modulation operation that reduces the parameter spacefor fine-tuning the StyleGAN2.
The idea is to optimize for each target domain only a single vector d .
We incorporate this vector into the StyleGAN2 architecture through the modulation operation ateach convolution layer.
The dimension of the vector d equals 6 thousand that is 5 thousand timesless than the original weights space of the StyleGAN2.
We apply this parameterization for thestate-of-the-art domain adaptation methods StyleGAN-NADA [5] and MindTheGAP [45].
We showthat it has almost the same expressiveness as the full parameterization while being more lightweight.
To further advance the domain adaptation framework of GANs we propose a new regularization lossthat improves the diversity of the fine-tuned generator.
Such considerable reduction in the size of the proposed parameterization motivates us to considerthe problem of multi-domain adaptation of GANs, i.e. when the same model can adapt to multipledomains depending on the input query.
Typically, this problem is tackled by previous methods just byfine-tuning separate generators for each target domain independently.
In contrast, we propose to traina hyper-network that predicts the vector d for the StyleGAN2 depending on the target domain.
Wecall this network as HyperDomainNet.
Such hyper-network would be impossible to train if we neededto predict all weights of StyleGAN2.
The immediate benefits of multi-domain framework consist ofreducing the training time and the number of trainable parameters because instead of fine-tuning nseparate generators we train one HyperDomainNet to adapt to n domains simultaneously.
Anotheradvantage of this method is that it can generalize to unseen domains if n is sufficiently large and weempirically observe this effect.
We provide extensive experiments to empirically confirm the effectiveness of the proposed parameterization and the regularization loss on a wide range of domains.
We illustrate that our parameterizationcan achieve quality comparable with the full parameterization (i.e. when we optimize all weights).
The proposed regularization loss significantly improves the diversity of the fine-tuned generator that isvalidated qualitatively and quantitatively.
Further, we conduct experiments with the HyperDomainNetand show that it can be successfully trained on a number of target domains simultaneously.
Also weshow that it can generalize to a number of diverse unseen domains.
To sum up, our main contributions are
• We reduce the number of trainable parameters for domain adaptation of StyleGAN2 [14]generator by proposing the domain-modulation technique.
Instead of fune-tuning all 30millions weights of StyleGAN2 for each new domain now we can train only 6 thousanddimensional vector.• We introduce a novel regularization loss that considerably improves the diversity of theadapted generator.•
We propose a HyperDomainNet that predicts the parameterization vector for the inputdomain and allows multi-domain adaptation of GANs.
It shows inspiring generalizationresults on unseen domains.
Related Work
Domain Adaptation of GANs The aim of few-shot domain adaptation of GANs is to learn accurateand diverse distribution of the data represented by only a few images.
The standard approach is toutilize a generator pretrained on source domain and fine-tune it to a target domain.
There are generallytwo different regimes of this task.
The first one is when we adapt the generator to completely newdata (e.g. faces → landscapes, churches, etc.), and the second regime is when the target domainrelates to the source one (e.g. faces → sketches, artistic portraits, etc.).
Methods that tackle the first regime typically require several hundreds or thousands samples to adaptsuccessfully.
Such setting assumes that the weights of the generator should be changed significantlybecause the target domain can be very far from the source.
The paper [12] shows that for distantdomains training from scratch gives comparable results to transfer learning.
It also confirms that forsuch regime there is no point to reduce the parameter space.
Typcally such approaches utilize dataaugmentations [12, 30, 41, 42], or use auxiliary tasks for the discriminator to more accurately fit theavailable data [18, 38], or freeze lower layers of the discriminator to avoid overfitting [19].
In the second regime the transfer learning is especially crucial because the pretrained generatoralready contains many information about the target domain.
In this setting the required number ofavailable data can be significantly smaller and range from hundreds to several images.
The mainchallenges in the case of such limited data are to avoid over-fitting of the generator and leverage its diversity learned from the source domain.
To tackle these challenges existing methods introducerestrictions on the parameter space [26, 20], mix the weights of the adapted and the source generators[23], utilize a small network to force sampling in special regions of the latent space [34], proposenew regularization terms [17, 31], or apply constrastive learning techniques to enhance cross-domainconsisitency [21].
The state-of-the-art methods [5, 45] leverage supervision from vision-language
CLIP model [24].
StyleGAN-NADA [5] applies it for text-based domain adaptation when we haveno access to images but only to the textual description.
MindTheGap [45] employs CLIP model tofurther significantly improve the quality of one-shot domain adaptation.
Constraining Parameter Space for GAN’s Adaptation In the second regime of GAN’s adaptationit is especially important for the generator to leverage the information from the source domain duringadapting to the target one.
The common approach is to introduce some restrictions on the trainableweights to regularize them during fine-tuning.
For example, the work [26] proposes to optimize onlythe singular values of the pretrained weights and apply it for few shot domain adaptation, however thereported results show the limited expressiveness of such parameterization [26].
Another method [20]constrains the parameter space for models with batch normalization (BN) layers such as BigGAN[2] by optimizing only BN statistics during fine-tuning.
While it allows to decrease the numberof trainable parameters, it also considerably reduces the expressiveness of the generator [26, 21].
Other approach is to adaptively choose a subset of layers during optimization at each step as in
StyleGAN-NADA [5].
It helps to stabilize the training, however it does not reduce the parameterspace because each layer can potentially be fine-tuned.
In contrast, the size of our parameterization isless by orders of magnitude than the size of the full parameter space while having the comparableexpressiveness.
Preliminaries
In this work, we focus on StyleGAN generators in the context of domain adaptation.
We consider StyleGAN2 [14] as a base model.
As the state-of-the-art domain adaptation methods we use
StyleGAN-NADA [5] and MindTheGAP [45].
StyleGAN2 The StyleGAN2 generation process consists of several components.
The first part is amapping network M ( z ) that takes as an input random vectors
z ∈ Z from the initial latent space, Zthat is typically normally distributed.
It transforms these vectors z into the intermediate latent space
W .
Each vector w ∈ W is further fed into different affine transformations A ( w ) for each layer ofthe generator.
The output of this part forms StyleSpace S [36] that consists of channel-wise styleparameters s =
A ( w ) .
The next part of the generation process is the synthesis network G sys thattakes as an input the constant tensor c and style parameters s at the corresponding layers and producesthe final feature maps at different resolutions F = G sys ( c, s ) .
These feature maps move on to thelast part which consists of toRGB layers G tRGB that generate the output image I =
G tRGB ( F ) .
Problem Formulation of Domain Adaptation
The problem of domain adaptation of StyleGAN2can be formulated as follows.
We are given a trained generator G A for the source domain A , andthe target domain B that is represented by the one image I B (one-shot adaptation) or by the textdescription t B (text-guided adaptation).
The aim is to fine-tune the weights θ of a new generator G Bθfor the domain B starting from the weights of G A .
The optimization process in the general form is
[Equation]
where L is some loss function, n is a batch size, w 1 , . . .
, w n are random latent codes, { G Bθ ( w i ) } ni =
1and { G A ( w i ) } ni =1 are batches of images sampled by G Bθ and
G A generators, respectively, and B, Aare domains that are represented by images or text descriptions.
CLIP model CLIP [24] is a vision-language model that is composed of text and image encoders
E T , E I , respectively, that maps their inputs into a joint, multi-modal space of vectors with a unitnorm (this space is often called as CLIP space).
In this space the cosine distance between embeddingsreflects the semantic similarity of the corresponding objects.
StyleGAN-NADA StyleGAN-NADA [5] is a pioneering work that utilizes the CLIP model [24] fortext-guided domain adaptation of StyleGAN.
The proposed loss function is
[Equation]
The idea is to align the CLIP-space direction between the source and target images∆
I ( G Bθ ( w ) , G A ( w )) with the direction between a pair of source and target text descriptions∆ T ( B, A ) .
So, the overall optimization process has the form
[Equation]
In StyleGAN-NADA method the L B ( θ ) loss is optimized only with respect to the weights θ of thesynthesis network G Bsys which has 24 million weights.
MindTheGap The MindTheGap method [45] is proposed for a one-shot domain adaptation of
StyleGAN, i.e. the domain B is represented by the single image I B .
In principle StyleGAN-NADAmethod can solve this problem just by replacing the text direction ∆ T ( B, A ) from Equation (2) to animage one
[Equation]
where
[Equation]
[ E I ( I A )] is the mean embedding of the images from domain A .
However, as statedin [45] this leads to an undesirable effect that transferred images lose the initial diversity of domain
A and become too close to the I B image.
So, the key idea of the MindTheGap is to replace the meanembedding from Equation (4) by the embedding of projection I ∗ A of I B image to A domain obtainedby the GAN inversion method II2S [46]:
[Equation]
So, the MindTheGap uses the modified L ′ direction loss that is renamed to L clip _ accross
[Equation]
In addition to this idea several new regularizers are introduced that force the generator G Bθ toreconstruct the I B image from its projection I ∗ A .
It further stabilizes and improves the quality ofdomain adaption.
Overall, the MindTheGAP loss function L MTG has four terms to optimize G B θ .
For more details about each loss please refer to the original paper [45].
Approach
4.1 Domain-Modulation Technique for Domain Adaptation
Our primary goal is to improve the domain adaptation of StyleGAN by exploring an effective andcompact parameter space to use it for fine-tuning G Bθ .
As we described in Section 3 StyleGAN hasfour components: the mapping network M ( · ) , affine transformations A ( · ) , the synthesis network
G sys ( · , · ) , and toRGB layers G tRGB ( · ) .
It is observed in the paper [37] that the main part of
StyleGAN that is mostly changed during fine-tuning to a target domain is the synthesis network
G sys ( · , · ) .
It is also confirmed by StyleGAN-NADA [5] and MindTheGap [45] methods as they adaptonly the weights of G sys ( · , · ) for the target domain.
So, we aim to find an effective way to fine-tune the weights of feature convolutions of G sys ( · , · ) .
In
StyleGAN2 [14] these convolutions utilize modulation/demodulation operations to process the input
[Figure]
tensor and the corresponding style parameters s .
Let us revisit the mechanism of these operations:
[Equation]
where w, w ′ and w ′′ are the original, modulated and demodulated weights, respectively, s i is thecomponent of the style parameters s , i and j enumerate input and output channels, respectively.
Theidea behind modulation/demodulation is to replace the standard adaptive instance normalization(AdaIN) [32, 4] to a normalization that is based on the expected statistics of the input feature mapsrather than forcing them explicitly [14].
So, the modulation part is basically an adaptive scalingoperation as in AdaIN that is controled by the style parameters s .
This observation inspires us to usethis technique for the domain adaptation.
The problem of fine-tuning GANs to a new domain is very related to the task of style transfer wherethe goal is also to translate images from the source domain to a new domain with the specified style.
The contemporary approach to solve this task is to train an image-to-image network which takes thetarget style as an input condition.
The essential ingredient of such methods is the AdaIN that providesan efficient conditioning mechanism.
In particular, it allows to train arbitrary style transfer models[10].
So, it motivates us to apply the AdaIN technique for adapting GANs to new domains.
We introduce a new domain-modulation operation that reduces the parameter space for fine-tuning
StyleGAN2.
The idea is to optimize only a vector d with the same dimension as the style parameterss .
We incorporate this vector into StyleGAN architecture by the additional modulation operationafter the standard one from Equation (7):
[Equation]
where d i is the component of the introduced domain parameters d (see Figure 1a).
So, instead ofoptimizing all weights θ of the G sys part we train only the vector d .
We apply these new parameterization to StyleGAN-NADA and MindTheGAP methods, i.e. insteadof optimizing its loss functions
wrt θ we optimize it wrt d vector
(see Figure 1b)
The dimensionof the vector d equals 6 thousand that is 4 thousand times less than the original weights space θ of
G sys ( · , · ) part.
While the proposed parameter space is radically more constrained we observe that ithas the expressiveness comparable with the whole weight space.
4.2 Improving Diversity of CLIP-Guided Domain Adaptation
The CLIP-based domain adaptation methods StyleGAN-NADA and MindTheGap use L direction (or
L clip _ accross ) loss (see Equations (2) and (6)) that was initially introduced to deal with the mode
[Figure]
T r a i n
D e s c
r
i p t i o n s
U n s e e n
D e s c
r
i p t i o n s
ModConvs
DomainModulation
Per Domainoffsets collapsing problem of the fine-tuned generator [5].
However, we empirically observe that it solvesthe issue only partially.
In particular, it preserves the diversity only at the begining of the fine-tuningprocess and starts collapsing after several hundred iterations.
It is a significant problem because forsome domains we need much more iterations to obtain the acceptable quality.
The main cause of such undesirable behaviour of the L direction (the same for L clip _ accross ) loss isthat it calculates the CLIP cosine distance between embeddings that do not lie in the CLIP space.
Indeed, the cosine distance is a natural distance for objects that lie on a CLIP sphere but becomesless evident for vectors ∆ T, ∆ I that represent the difference between clip embeddings that no longerlie on a unit sphere.
Therefore, the idea behind the L direction loss may be misleading and in practicewe can observe that it still suffers from mode collapse.
We introduce a new regularizer for improving diversity that calculates the CLIP cosine distance onlybetween clip embeddings.
We called it indomain angle consistency loss and we define it as follows
[Equation]
The idea of L indomain − angle loss is to preserve the CLIP pairwise cosine distances between imagesbefore and after domain adaptation.
We observe that this loss significantly improves the diversity ofthe generator G Bd compared to the original L direction or L clip _ accross losses.
4.3 Designing the HyperDomainNet for Universal Domain Adaptation
The proposed domain-modulation technique allows us to reduce the number of trainable parameterswhich motivates us to tackle the problem of multi-domain adaption of StyleGAN2.
Our aim is totrain the HyperDomainNet that predicts the domain parameters given the input target domain.
Thisproblem can be formulated as follows.
We are given a trained generator G A for a source domain Aand a number of target domains B 1 , . . .
, B m that can be represented by the single image or the textdescription.
The aim is to learn the HyperDomainNet D φ ( · ) that can predict the domain parametersd B i
=
D φ ( B i ) which will be used to obtain the fine-tuned generator G B i
d Bi by the domain-modulationoperation (see Section 4.1).
In this work, we focus on the setting when the target domains B 1 , . . .
, B m are represented by textdescriptions t B 1 , . . .
, t B m .
The HyperDomainNet D φ ( · ) takes as an input the embedding of the textobtained by the CLIP encoder E T ( · ) and outputs the domain parameters
d B i
=
D φ ( E T ( t B i )) .
Thetraining process is described in the Figure 2.
To train the HyperDomainNet D φ ( · ) we use the sum of L direction losses for each target domains.
Inaddition, we introduce L tt − direction loss ("tt" stands for target-target) that is the same as L direction ,but we compute it between two target domains instead of target and source.
The idea is to keep awaythe images from different target domains in the CLIP space.
We observe that without L tt − directionloss the HyperDomainNet tends to learn the mixture of domains.
[Figure]
In multi-domain adaptation setting, the regularizer L indomain − angle becomes inefficient becauseduring training batch consists of samples from different domains and the number of images from onedomain can be very small.
Therefore, we introduce an alternative regularization L domain − norm forthe HyperDomainNet that constrains the norm of the predicted domain parameters.
To be exact itequals to ∥ D φ ( E T ( t B i ))
− 1 ∥ 2 .
So, the objective function of the HyperDomainNet consists of L direction , L tt − direction and
L domain − norm losses.
For more detailed description of these losses the overall optimization process,please refer to Appendix A.2.
Experiments
In this section, we provide qualitative and quantitative results of the proposed approaches.
At first, weconsider the text-based domain adaptation and show that our parameterization has comparable qualitywith the full one.
Next, we tackle one-shot domain adaptation and confirm the same quantitatively andalso show the importance of the L indomain − angle loss.
Finally, we solve the multi-domain adaptationproblem by the proposed HyperDomainNet, show its generalization ability on unseen domains.
Forthe detailed information about setup of the experiments please refer to Appendix A.1.
Text-Based Domain Adaptation We compare the StyleGAN-NADA [5] method with the proposedparameterization and the original version on a number of diverse domains.
In Figure 3, we see that theexpressiveness of our parameterization is on par with the original StyleGAN-NADA.
We observe thatthe domain-modulation technique allows to adapt the generator to various style and texture changes.
For results on more domains please refer to Appendix A.3.
One-Shot Domain Adaptation In this part, we examine our parameterization and the indomainangle consistency loss by applying them to the MindTheGap [45] method.
We show qualitativeand quantitative results and compare them with other few-shot domain adaptation methods such as
Ours
StyleGAN
NADA
StyleGAN-NADA, TargetCLIP [3] and Cross-correspondence [21] method.
To assess the domainadaptation quality we use the standard metrics FID [9], precision and recall [15].
As a target domainwe take the common benchmark dataset of face sketches [33] that has approximately 300 samples.
We consider the one-shot adaptation setting.
We provide the results in Table 1.
At fisrt, we seethat the MindTheGap with our parameterization shows comparable results with the original versionwhile having less trainable parameters by three orders of magnitude.
Secondly, we examine theeffectiveness of the indomain angle consistency.
We show that it considerably improves FID andprecision metrics for both the original MindTheGap and the one with our parameterization.
The qualitative results are provided in Figure 4 for MindTheGap+indomain, MindTheGap+indomainwith our parameterization ("Ours") and StyleGAN-NADA.
For other methods please see Appendix A.4.
We observe that MindTheGap+indomain and our version shows comparable visualquality and outperform StyleGAN-NADA in terms of diversity and maintaining the similarity to thesource image.
Overall, we demonstrate that our parameterization is applicable to the state-of-the-art methods
StyleGAN-NADA and MindTheGap
and it can be further improved by the indomain angle consistencyloss.
Multi-Domain Adaptation Now we consider the multi-domain adaptation problem.
We apply the
HyperDomainNet in two different scenarios: (i) training on fixed number of domains, (ii) training onpotentially arbitrary number of domains.
The first scenario is simple, we train the HyperDomainNeton 20 different domains such as "Anime Painting", "Pixar", etc.
(for the full list of domains please
[Table]
[Figure]
refer to Appendix).
The second scheme is more complicated.
We fix large number of domains(several hundreds) and calculate its CLIP embeddings.
During training we sample new embeddingsfrom the convex hull of the initial ones and use them in the optimization process (see Figure 2).
Thistechnique allows us to generalize to unseen domains.
For more details about both scenarios pleaserefer to Appendix A.2.
The results of the HyperDomainNet for both scenarios are provided in Figure 5.
The left part is resultsfor the first setting, the right one is results for the unseen domains in the second scheme.
For moredomains and generated images please refer to Appendix A.2.
We see that in the first scenario the HyperDomainNet shows results comparable to the case when we train separate models for each domain(see Figure 3).
It shows that the proposed optimization process for the HyperDomainNet is effective.
The results for the second scenario looks promising.
We can observe that the HyperDomainNet haslearnt very diverse domains and shows sensible adaptation results for unseen ones.
Conclusion
We propose a novel domain-modulation technique that allows us to considerably reduce the numberof trainable parameters during domain adaptation of StyleGAN2.
We successfully apply it to the stateof-the-art methods.
In addition, we introduce a new indomain angle consistency loss that improvesthe diversity of the fine-tuned generator.
Finally, equipped with the efficient parameterization wepropose the HyperDomainNet for multi-domain adaptation of StyleGAN2.
We empirically show thatit can be trained to several domains successfully.
And when the number of training domains is largeit can generalize to unseen domains.
The main limitation of our approach is that it is not applicable for the cases when target domains arevery far from the source one.
In such setting, we cannot limit the parameter space, so we should usethe full parameterization.
The potential negative societal impacts of domain adaptation of GANs and generally training of
GANs include different forms of disinformation, e.g. deepfakes of celebrities or senior officials, fakeavatars in social platforms.
However, it is the issue of the whole field and this work does not amplifythis impact.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Appendix A.
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] The reported results of the baselines we compare withhave no error bars
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 5(b)
Did you mention the license of the assets?
[Yes] See Appendix A.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A AppendixA.1 Setup of the ExperimentsA.1.1 Implementation Details
We implement our experiments using PyTorch 1 deep learning framework.
For StyleGAN2 [14]architecture we use the popular PyTorch implementation 2 .
We attach all source code that reproducesour experiments as a part of the supplementary material.
We also provide configuration files to runeach experiment.
A.1.2 Datasets
We use source StyleGAN2 models pretrained on the following datasets: (i) Flickr-Faces-HQ (FFHQ)[13], (ii) LSUN Church, (iii) LSUN Cars, and (iv) LSUN Cats [40].
As target domains we mainlyuse the text descriptions from [5] and style images from [45].
For quantitative comparison with othermethods we use face sketches [33] as the standard dataset for domain adaptation.
A.1.3 Licenses and Data Privacy
Tables 2, 3 provide sources and licenses of the models and datasets we used in our work.
[Table]
[Table]
A.1.4 Total Amount of Compute Resources
We run our experiments on Tesla A100 GPUs.
We used approximately 12000 GPU hours to obtainthe reported results and for intermediate experiments.1 https://
A.2 Training of the HyperDomainNet (HDN)A.2.1 Training Losses
As we describe in Section 4.3 we train HDN D φ ( · ) using three losses L direction , L tt − direction , and
L domain − norm .
Each loss is defined as follows:
[Equation]
Then the overall training loss for the HDN D φ ( · ) is
[Equation]
A.2.2 Architecture of the HDN
We use the standard ResNet-like architecture for the HDN.
It has the backbone part which has10 ResBlocks and the part that consists of 17 heads.
The number of heads equals the number of
StyleGAN2 layers in the synthesis network G sys .
Each head has 5 ResBlocks and outputs the domainvector for the corresponding StyleGAN2 layer.
We illustrate the overal architecture of the HDN in
Figure 6.
It has 43M parameters.
We use the same architecture for all experiments.
A.2.3 Inference Time
The inference time of the HDN network on 1 Tesla A100 GPU is almost the same as the one forwardpass through StyleGAN2 generator which works in 0.02 seconds.
A.2.4 Training on Fixed Number of Domains
For training the HDN on fixed number of domains we use the loss function from Equation (16).
As training target domains we take the following 20 domains (we provide in the format "the targetdomain - the corresponding source domain"):
Anime Painting - Photo2. Impressionism Painting - Photo
Mona Lisa Painting - Photo4.
3D Render in the Style of Pixar - Photo5.
Painting in the Style of Edvard Munch - Photo6.
Cubism Painting - Photo7.
Sketch - Photo
Dali Painting - Photo
Fernando Botero Painting - Photo
A painting in Ukiyo-e style - Photo
[Figure]
Backbone
[Figure]
Tolkien Elf - Human
[Figure]
Neanderthal - Human
[Figure]
The Shrek - Human
[Figure]
Zombie - Human
[Figure]
The Hulk - Human
[Figure]
The Thanos - Human
[Figure]
Werewolf - Human
[Figure]
Nicolas Cage - Human
[Figure]
The Joker - Human
[Figure]
Mark Zuckerberg - Human
[Figure]
Hyperparameters For 20 descriptions in traning list following setup is considered.
The HDNtrained for 1000 number of iterations.
Batch size 24 is used.
Weights of the terms from Equation (16)as follows:
λ direction = 1 . 0 , λ tt − direction = 0 .
4 , λ domain − norm = 0 . 8 .
We use two VisionTransformer based CLIP models, "ViT-B/32" and "ViT-B/16".
To optimize HDN we use an ADAM
Optimizer with betas = (0 . 9 , 0 . 999) , learning rate = 5 e − 5 , weight decay = 0 .
A.2.5 Training Time
The training time of the HDN on 20 domains for 1000 iterations on single Tesla A100 GPUs takesabout 2 hours.
A.2.6 Ablation Study on the Loss Terms
We perform both the quantitative and qualitative ablation study on the domain-norm and tt-directionloss terms that are defined in Appendix A.2.1.
For the qualitative analysis we consider three domains (Anime Painting, Mona Lisa Painting, Apainting in Ukiyo-e style) for the HyperDomainNet that was trained on 20 different domains (seethe full list in Appendix A.2.4).
We provide the visual comparison for these domains with respectto the using loss terms in the training loss of the HyperDomainNet (see Figure 7).
We can see thatwithout additional loss terms the model considerably collapses within each domain.
After addingdomain-norm it solves the problem of collapsing within domains but it starts mix domains with eachother, so we obtain the same style for different text descriptions.
And after using tt-direction losseventially allows us to train the HyperDomainNet efficiently on these domains without collapsing.
For the quantitative results we use the metrics Quality and Diversity that were introduced in Appendix A.3.3.
The results are provided in Table 4.
We see that the initial model without loss termsobtains good Quality but very low Diversity.
The domain-norm significantly improves the diversity inthe cost of degrading the Quality.
The tt-direction provides a good balance between these two metricswhich we also we qualitatively in Figure 7.
[Figure]
Additional Samples We show results for the first 10 domains in Figure 5.
The next 10 domains weprovide in Figure 8.
A.2.7 Training on Potentially Arbitrary Number of Domains
Improving the generalization ability of the HDN is challenging because it tends to overfit on trainingdomains and for unseen ones it predicts domains that are very close to train ones.
One way totackle this problem is to considerably extend the training set.
For this purpose, we use threetechniques: (i) generating many training domains by taking combinations of different ones; (ii)sampling CLIP embeddings from convex hull of the initial training embeddings;
(iii) resample initial
CLIP embeddings given cosine similarity.
We discuss each technique further.
Generating Training Domains by Taking Combinations We can describe domains by indicatingdifferent properties of the image such as image style (e.g. Impressionism, Pop Art), image type(e.g.
Photo, Painting), artist style (e.g. Modigliani).
Also we can construct new domains by takingcombinations of these properties (e.g. Modigliani Painting, Impressionism Photo).
So, we take 32 image styles, 13 image types and 7 artists and by taking all combinations of these properties we comeup with 1040 domains.
We provide the full list of properties we use in our training:
[Table] • image styles: ’Pop Art’, ’Impressionism’, ’Renaissance’, ’Abstract’, ’Vintage’, ’Antiquity’,
[Table]
’Cubism’, ’Disney’, ’Chinese’, ’Japanese’, ’Spanish’, ’Italian’, ’Dutch’, ’German’, ’Surreal’,
[Table]
’WaltDisney’, ’DreamWorks’, ’Modern’, ’Realism’, ’Starry Night’, ’Old-timey’, ’Pencil’,
[Table]
’Gouache’, ’Acrylic’, ’Watercolor’, ’Oil’, ’Black’, ’Blue’, ’Charcoal’, ’Manga’, ’Kodomo’;
[Table]
• image types: ’Portrait’, ’Image’, ’Photo’, ’Painting’, ’Graffiti’, ’Photograph’, ’Cartoon’,
[Table]
’Stereo View’, ’Drawing’, ’Graphics’, ’Mosaic’, ’Caricature’, ’Animation’;
[Table]
• artists: ’Raphael’, ’Salvaror Dali’, ’Edvard Munch’, ’Modigliani’, ’Van Gogh’, ’Claude
[Table]
Monet’, ’Leonardo Da Vinci’
[Table]
The algorithm of generating combinations is
[Table] [Table]
[Figure]
Generating CLIP Embeddings from Convex Hull
In the usual training of the HDN as in Appendix A.2.4 we use the CLIP embeddings of the target domains t B 1 = E T ( B 1 ) , . . .
, t B m =
E T ( B m ) .
To cover more CLIP space we propose to use new embeddings t ′ B 1 , . .
t ′ B m from the convex hull of the initial ones:
[Equation]
17) where α 1 , . . .
, α m ∼ Dir ( β ) (Dirichlet distribution) , m
[Equation] [Equation]
m.
[Equation] [Equation]
We use β = batch size .
Resampling Initial CLIP Embeddings Given Cosine Similarity To further extend the CLIPspace we cover during training of the HDN we resample initial CLIP embeddings of the targetdomains t B 1 , . . .
, t B m constrained to the cosine similarity.
So, before generating from convex hullwe replace the initial embeddings by new ones ˆ t B 1 , . .
, ˆ t B m such that cos( t B 1 , ˆ t B 1 ) = γ .
To obtainthese embeddings we use the following operation:
[Equation]
19) where resample ( t B i ) =
t B i · cos γ + norm ( v − proj t Bi v ) · sin γ
[Equation]
(20) v ∼ N ( v | 0 , I ) , norm ( u ) =
u
|| v ||(21)
It allows us to cover the part of the CLIP space outside of the initial convex hull.
We observe that itimproves the generalization ability of the HDN.
Hyperparameters We train the HDN for 10000 number of iterations.
We use batch size of 96.
We set weights of the terms from Equation (16) as follows: λ direction = 1 . 0 , λ tt − direction = 0 .
4 ,λ domain − norm = 0 . 8 .
We use two Vision-Transformer based CLIP models, "ViT-B/32" and "ViTB/16".
To optimize HDN we use an ADAM Optimizer with betas =
(0 . 9 , 0 . 999) , learning rate = 5 e − 5 ,weight decay = 0 .
Training Time
The training time of the HDN for 10000 iterations on 4 Tesla A100 GPUs takesabout 50 hours.
Additional Samples Additional samples of unseen domains for the HDN is demonstrated in
Figure 10.
[Figure]
A.3 Results for Text-Based Domain Adaptation
A.3.1 Hyperparameters
StyleGAN-NADA with our parameterization trained for 600 iterations with batch size of 4.
Stylemixing probability is set to 0 .
9 , the weight of the L direction is 1 .
and L indomain − angle is 0 .
5 and
ADAM optimizer with betas = (0 ., 0 . 999) , learning rate = 0 .
002 , weight decay = 0 .
For the original StyleGAN-NADA [5] number of iterations is decreased to 200 because for moreiterations it starts to collapse.
"ViT-B/32" and "ViT-B/16" CLIP Vision-Transformed models used in all setups.
A.3.2 Training and Inference Time
The training of the one target domain for 600 iterations on a single Tesla A100 GPU takes about 15minutes on batch size 4.
The inference time consists of two parts.
The first one is the embedding process of the real imagewhich takes 0.23 seconds using ReStyle [1].
The second part is the forward pass through adapted
GAN generator which works in 0.02 seconds.
A.3.3 Quantitative Results
We provide the quantitative comparison for the text-based domain adaptation by evaluating the"Quality" and "Diversity" metrics in a straightforward way.
As the “Quality” metric we estimate how close the adapted images to the text description of the targetdomain.
That is we compute the mean cosine similarity between image CLIP embeddings and theembedding of the text description:
Quality
[Equation]
( target_text
[Equation]
I i ) ⟩ , where (22) n - number of the generated adapted images (we use 1000) , [Equation] encoder ,
[Equation]
I
[Equation]
, I n - generated adapted images .
As E I encoder we use only ViT-L/14 image encoder that is not applied during training (in the trainingwe use ViT-B/16, ViT-B/32 image encoders).
As the “Diversity” metric we estimate the mean pairwise cosine distance between all adapted images:
Diversity
[Equation]
, where
[Equation] [Equation]
n - number of the generated adapted images (we use 1000) ,
E I - image CLIP encoder ,
I 1 , . .
, I n - generated adapted images .
We compute these two metrics for the ten text domains: Anime Painting, Mona Lisa Painting, 3D
Render Pixar, Sketch, Ukiyo-e Painting, Fernando Botero Painting, Werewolf, Zombie, The Joker,
Neanderthal.
We separately report metrics for two domains Anime Painting and Zombie to betterreflect the metrics behaviour.
Also we report the overall metrics across all nine domains.
The resultsare provided in Table 5.
From these results we see that our model performs comparably with the StyleGAN-NADA withrespect to Quality while having better Diversity.
Also we can observe that the indomain angleloss significantly improves the Diversity for both models StyleGAN-NADA and Ours while lightlydecreases the Quality.
For the multi-domain adaptation model we see that it has lower diversity than StyleGAN-NADA and
Ours and comparable Quality while being adapted to all these domains simultaneously.
Also we report samples for the StyleGAN-NADA and our model with and without indomain angleloss in Figures 11 and 12.
We see that qualitatively indomain angle loss also significantly improvesthe diversity of the domain adaptation methods.
A.3.4 Additional Samples
We show additional domains for FFHQ dataset in Figure 13.
Also we demonstrate how our methodworks on another datasets such as LSUN Church in Figure 14, LSUN Cats in Figure 15, and LSUN
Cars in Figure 16.
[Table]
StyleGAN-NADA
Ours
[Figure] [Figure]
[Figure] [Figure]
StyleGAN-NADA
Ours
Capybara
Fox
Hedgehog
Koala
Lemur
Lion
Meerkat
Panda
Raccoon
[Figure]
A.4 Results for One-Shot Domain Adaptation
[Figure]
A.4.1 Hyperparameters
[Figure]
For each target style image we adapt the generator for 600 iterations as in [45].
We use batch size of4, fine-tune all layers of the StyleGAN2, set the mixing probability to 0.9.
We use all loss terms as in[45] with the same weights and add the L indomain − angle term with weight 2.
For all experiments,we use an ADAM Optimizer with a learning rate of 0.002.
A.4.2 Training and Inference Time
[Figure]
The training of the one target style image for 600 iterations on a single Tesla A100 GPU takes about20 minutes.
The same as for the text-based adaptation the inference time consists of two parts:embedding process and the forward pass through the generator.
The embedding process takes 0.36seconds for e4e [29] and two minutes for II2S [46].
The second part is the forward pass throughadapted GAN generator which works in 0.02 seconds.
A.4.3 Additional Samples
We provide additional samples in Figures 17 and 18.
Also we provide results for other baselinemethods in Figure 19.
[Figure] [Figure]