Generalized One-shot Domain Adaption of Generative Adversarial Networks
[Figure]
Abstract
The adaption of Generative Adversarial Network (GAN) aims to transfer a pretrained GAN to a given domain with limited training data.
In this paper, wefocus on the one-shot case, which is more challenging and rarely explored inprevious works.
We consider that the adaptation from source domain to targetdomain can be decoupled into two parts: the transfer of global style like textureand color, and the emergence of new entities that do not belong to the sourcedomain.
While previous works mainly focus on the style transfer, we propose anovel and concise framework 1 to address the generalized one-shot adaption taskfor both style and entity transfer, in which a reference image and its binary entitymask are provided.
Our core objective is to constrain the gap between the internaldistributions of the reference and syntheses by sliced Wasserstein distance.
Tobetter achieve it, style fixation is used at first to roughly obtain the exemplary style,and an auxiliary network is introduced to the original generator to disentangleentity and style transfer.
Besides, to realize cross-domain correspondence, wepropose the variational Laplacian regularization to constrain the smoothness of theadapted generator.
Both quantitative and qualitative experiments demonstrate theeffectiveness of our method in various scenarios.
Introduction
Benefiting from numerous excellent pre-trained GANs ( e.g., styleGAN [15] and BigGAN [6]), GANadaption has become an important research topic, which leverages the knowledge of GANs pretrained on large-scale datasets to mitigate the lack of data and speed up the training on a new domain
It can be divided into few-shot, one-shot and zero-shot [12] cases.
The few-shot (usually ≥ 10 ) casehas been extensively studied in the past years [39, 38, 24, 22].
The latest works [27, 40, 43] seek thecross-domain correspondence that keeps the shapes or contents of syntheses before and after adaptioninvariant.
In recent times, the more challenging one-shot case has been explored by several works[44, 50, 19], which mainly focus on adapting GANs to the target domain of similar style as the givenimage.
By utilizing GAN inversion [42], the adapted models can be applied to image style transferand manipulation tasks.
However, there exist limitations in previous one-shot task settings, since an exemplar contains richinformation more than general color and texture style.
As shown in Fig.
1, where the source domainrefers to natural faces, and the exemplars of target domains are artistic portraits with some entities( e.g., glasses or accessories), previous works are only able to transfer the artistic style while theentities are neglected.
This may lead to two problems: 1)
In some cases, the entity ( e.g., hat),considered as an important part of the style, should be transferred with the color style simultaneously.2)
For the entities of large size, it easily causes artifacts by previous works due to the influence ofentities on the whole color space.
To tackle the above problems, we describe the target domain withthe aid of an extra binary mask, which labels the entities we are interested in and helps to define thetarget domain more exactly and flexibly.
We name the new task definition as generalized one-shot GAN adaption .
The previous scenarios can be taken as the special case when the binary mask is fullof zeros.
We believe the exploration of the novel task is quite meaningful, which not only allowscustomization for more complex scenarios, but also facilitates further discovery and utilization of theprior knowledge in pre-trained models.
In specific, we propose a concise manifold regularized GAN adaption framework to deal with thenovel task.
Following previous works [44, 50, 19], StyleGAN2 [16] is selected as our base model.
We modify the architecture of the original generator to decouple the adaption of style and entity,where the new generator comprises an additional auxiliary network to facilitate the generation ofentity, and the original generator is dedicated to generating stylized images.
To train efficiently, wefirst compute the latent code of the reference by GAN inversion, and mix it with the random latentcodes for rough style transfer. Then we employ the sliced Wasserstein distance [5] to minimize thedivergence of the internal distributions between the exemplar and syntheses for learning the domainknowledge More importantly, to prevent the content distortion during training, inspired by the classicmanifold learning [13, 2] we propose the variant Laplacian regularization to smooth the networkchanges before and after adaptation and preserve the geometric structure of the source generativemanifold.
We conduct extensive experiments on various references with and without entities.
Theresults show that our framework can fully exploit the cross-domain correspondence and achieve hightransfer quality.
Moreover, the adapted model can handily perform various image manipulationsbeyond previous works.
Related works
Few-shot GAN adaption [39] is the first to study and evaluate different transferring strategies of
GANs, it shows that transferring both pre-trained generator and discriminator can efficiently improvethe convergence time and the quality of the generated images.
[26] finds that merely tuning theparameters for batch statistics of the source generator is also valid.
[38] discovers that fixing thesource generator and only transforming the noise space into a new space works for most cases.
[24]tunes GANs by freezing the low layers of the discriminator. [49] demonstrates that transferringlow-level filters of pre-trained GANs can facilitate generation.
[22] proposes an Elastic Weight Consolidation loss to regularize the change of GANs.
[30] tries to adapt the singular values whilefreezing the singular vectors of the pre-trained weights.
As an important breakthrough, [27] proposesthe cross-domain consistency loss to maintain the diversity of the source generator, which makes thesamples from the adapted generator and source generator have a corresponding relation like styletransfer.
Some works [20, 40, 43, 17] are based on and improve [27] to get better performance. Zero-shot GAN adaption Recently many works [28, 18, 10] have explored the zero-shot domaintransfer, they define the target domain by the text prompts, and align the semantic features of imageand text predicted by the pre-trained image-text model CLIP [29]. StyleGAN-nada [12] proposes the
CLIP directional loss to represent the domain gap by the difference of textual embeddings from CLIP.
MTG [50] is the first to tackle the one-shot GAN adaption task, itfollows [12] to preserve the CLIP direction extracted by CLIP image encoder for adaption. OSGD[44] introduces extra modules to the original architecture to better reuse the prior knowledge.
[19]proposes a contrastive regularization for patchwise relationships in the CLIP space. However, thesemodels perform unstably in practice, and they mainly consider the style transfer without entity.
PreliminariesStyleGAN
A canonical StyleGAN [16] can be summarized in two parts: Firstly, given a noisedistribution P ( z ) , a mapping network transforms the noise from space Z = { z | P ( z ) ̸ = 0 } ⊆ 82 R 1 ×into the replicated latent space W = { [ F ( z ); . . .
; F ( z )] | z ∈ Z} ⊆ R 18 × 512 , F is a fullyconnected network.
Secondly, a synthesis network G composed of convolutional blocks transfers
W space into image space M = { G ( w ) |
w ∈ W} ⊆
R H
× W × 3 .
Another important concept is theextended latent space W + =
{ [ F ( z 1 ); . .
F ( z 18 )]
| z i ∈ Z} .
For ∀ w 1 , w 2 ∈ W , the latent codemanipulation merges their information by the linear combination diag ( α 1 ) w 1 + diag ( α 2 ) w 2 ∈ W + ,α 1 , α 2 ∈ R 18 .
Empirically, W + space is inferior to W in generation fidelity and editability, butsuperior in GAN inversion that finds a code w ref to reconstruct a reference
y ref ∈
R
H
× W × 3 .
Inthis paper we only focus on tuning G , and fix F for keeping the distribution of source latent spaces.
Manifold learning Considering the sample matrix
X =
[ x T 1 ; . . .
; x Tn ] from source manifold M s ,
W = [ w i,j ] ni,j =1 is the weight matrix, where w i,j is the weight of x i and x j usually defined ase −∥ x
i − x j ∥ 2 /σ .
Given a task specific function f and y = f ( x ) , manifold learning requires the transformed samples Y =
[ y T 1 ; . . .
; y Tn ] on the new manifold M t should preserve the geometric structureof M s , which means the data nearby on M s are also nearby on M t , and vice versa.
The requirementis mostly realized by minimizing the manifold regularization term [4] (cid:82) M s ∥∇ M s y
∥ 2 dP ( x ) , ∇ M sis the gradient along manifold, and P ( x ) is the probability measure.
If M t is a submanifold of R , theformer is equalized with (cid:82) M s y ∆ M s y dP ( x ) by Stokes theorem.
∆ called the Laplace–Beltrami operator, is a key geometric object in the Riemannian manifold, thus the integral is also called Laplacianregularization.
In practice, it is estimated by the discrete form 2 tr ( Y T LY ) =
(cid:80) i,j w i,j (cid:13)(cid:13)
y i − y j
(cid:13)(cid:13) 2 ,
L = D − W is known as the graph Laplacian operator, D is the degree matrix where the diagonalelement d i,i = (cid:80) nj
=1 w i,j and other elements are zero.
The convergence between L and ∆ can befound in [3].
Intuitively, the regularization encourages the local smoothness of f .
If data distributedensely in an area, the function should be smooth to avoid local oscillations.
If x i and x j are close,in the discrete form w i,j will impose a strong penalty to the distance between y i and y j .
Task definition The generalized one-shot adaption can be formally described as: Given a referencey ref , a mask m ref ∈ { 0 , 1 } H × W is provided to label the location of entity contained in y ref .
Thetarget domain T is defined to contain images of similar style and entities as y ref .
With the knowledgestored in a generative model G s pre-trained on source domain S , a generative model G t is learnedto generate diverse images belonging to domain T .
Note that we can set m ref = 0 when thereis no interested entities existed in y ref .
Moreover, the adaptation should satisfy the cross-domaincorrespondence.
i.e. , ∀ w ∈ W , G s ( w ) and G t ( w ) should have similar shape or content in visual.
With the correspondence, adapted models can be applied for not only synthesis, but also the transferand manipulation of source images.
Methodology
4.1 Overall
Our framework for tackling the generalized one-shot GAN adaption task is illustrated in Fig.
Forthe sake of description, in the following part, we will take the StyleGAN pre-trained on FFHQ as an 𝒚 𝒓𝒆𝒄 𝒚(cid:3549) 𝒓𝒆𝒄 𝒘 ⋕ 𝒚(cid:3549)
A u x
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k ⋯ ⋯ ` ⋯ 𝑢𝑛𝑒𝑡 𝒚 𝒓𝒆𝒇 𝒙 𝒓𝒆𝒄 𝓛 𝒓𝒆𝒄 𝓛 𝑽𝑳𝒂𝒑𝑹 𝒚
𝓛 𝒔𝒕𝒚𝒍𝒆 (a) Reconstruction (c) Loss (d) Auxiliary net (b) Random synthesis
Legend
Conv Block Conv Block
Style injector
StyleGANblock
𝒘 𝒓𝒆𝒇 𝓛 𝒆𝒏𝒕 𝒙 𝑮 𝒔 (cid:4666)𝒘(cid:4667) ⋯
A u x
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k
Style fixation
Flow of 𝐺 (cid:3047)
Flow of 𝐺(cid:3552) (cid:3047)
Conv Block example to describe each component of the framework in detail.
We will demonstrate our frameworkcan be applied to other source domains in the experiments.
Network architecture
Different from previous works [50, 19] that directly adapt the sourcegenerator G s , our target generator G t (as Fig.
2 shows) consists of a generator ˆ G t inherited from
G s , and an auxiliary network ( aux ) trained from scratch.
The design is inspired by the fact that G scan approximate arbitrary real faces but cannot handle most entities since G s captures the prevalentelements (clear natural faces) of FFHQ rather than the tails (faces with entities like hats or ornaments)of the distribution [46, 25].
We make aux serve to entities, and ˆ G t just focus on stylizing clear faceto benefit from the prior knowledge stored in G s .
As shown in Fig. 2(d), for each w , its feature mapgenerated by the convolutional block of StyleGAN is fed into aux , where a UNet [31] predicts thefeature map f ent of entity and the mask m that exactly labels the position of the entity, then they aremerged with f in by Eq.
(1) to get the feature map of face covered with entity.
[Equation]
⊙ denotes the Hadamard product.
We hope the shape of the entity is only influenced by the contentof synthesis, hence we place aux after the fourth block of StyleGAN, where the 32 × 32 resolutionf in contains the most content information and the least style information of synthesis (see Sec.4.2).
Because the global style and local entity are usually independent, the disentangled design alsoprevents the stylization of the face from being contaminated by the style of entity (see Sec. 4.3).
Training process Our training process can be divided into three main parts: 1 ⃝ Style fixation andexemplar reconstruction.
In Fig. 2(a), we get the latent code w ref =
[ w cref ; w sref ] of y ref by the
GAN inversion technique [36], and train G t to reconstruct y ref with reconstruction loss L rec .
In
Fig. 2(b), each w will be transformed into the style-fixed code w ♯ =
[ w c ; w sref ] to roughly obtainthe exemplary style.
2 ⃝ Internal distribution learning.
We minimize the divergence of internal patchdistributions between syntheses and exemplar for style and entity transfer.
Instead of using GANloss that is hard to optimize, we take sliced Wasserstein distance (SWD) [5] as style loss L style andentity loss L ent to learning internal distribution efficiently.
3 ⃝ Manifold Regularization.
To suppressthe content distortion during training, we propose the variational Laplacian regularization L V lapR tosmooth the change from G s to G t by preserving the geometric structure of the source manifold.
Asshown in Fig. 2(c), the overall loss is
[Equation]
We will illustrate and discuss the training process in detail in the following sections.
For convenienceof description and self-consistency, we may use the abbreviations:
y =
G t ( w ♯ ) , ˆ y
=
ˆ G t ( w ♯ ) ,y rec =
G t ( w ref ) , ˆ y rec =
ˆ G t ( w ref ) , x = G s ( w ♯ ) , x rec =
G s ( w ref ) .
4.2 Style fixation and exemplar reconstruction
It is well known that StyleGAN is equipped with the disentangled latent spaces W and W + [1],where the latter vectors of code mainly determine the style of synthesis whereas the earlier vectorsdetermine the coarse-structure or content of the synthesis.
Taking advantage of this trait, we firstroughly transfer the exemplary style to other syntheses by fixing the style code.
The exemplar y ref isfed into the pre-trained GAN inversion encoder e4e [36] to get the latent code w ref =
[ w cref ; w sref ] ,where w cref ∈ R l × 512 and w sref ∈ R (18 − l ) × 512 respectively encode content and style information ofexemplar.
For each w ∈ W , it will be transformed to w ♯ before fed into networks,
[Equation]
The content part of w is reserved and style part is replaced by w sref , and we denote the new spaceas W ♯ .
Because style fixation should not change the content of the original synthesis, by using thepre-trained Arcface model [11] to compare the identity similarity before and after style fixation, wefind setting l = 8 is acceptable which gets 65% average similarity and obtains the adequate exemplarystyle in visual.
In MTG [50], style mixing assists as a post-processing step to improve the stylequality, but the experiments (Fig. 3) show that the style of syntheses is underfitting and not consistent.
In our framework, it is used as a pre-processing step to facilitate the following learning.
Because x rec is just the projection of y ref on the source domain, there often exists obvious distinctionbetween them in visual as shown in Fig.
We use the reconstruction loss L rec to narrow the gap,
[Equation]
dssim denotes the negative structural similarity metric [41] and lpips is the perceptual loss [47],m rec is the upsampled reconstructed mask from aux .
It is worth noting L rec acts on G t rather thanthe vanilla architecture ˆ G t .
If m ref = 0 , aux does not work and G t be equivalent to ˆ G t .
4.3 Internal distribution learning
Recent works about internal learning [32, 34, 48, 21] prove that the internal patch distribution of 171 a single image contains rich meaning.
Minimizing the divergence of internal distributions worksout many tasks like image generation, style transfer, and super-resolution.
It is usually achieved 173 by the adversarial patch discriminator [14].
However, it is well known that the training of GAN istime-consuming, unstable, and requires large GPU memory.
Previous few-shot GAN adaption works 175 [22, 27] using GAN loss almost have the serious model collapse phenomenon.
We find that slice
Wasserstein distance ( SWD ) [5] can lead to the same destination but with greater efficiency.
For twotensor A , B ∈ R H × W × d , the SWD of their empirical internal distributions is defined as
[Equation]
where S d = { θ ∈ R d : ∥ θ ∥ = 1 } , proj : R H × W × d → R H × W that projects each pixel from dchannels to a scalar by θ , sort denotes the sort operator ordering all values.
SWD does not introduceextra parameter, and can be readily realized by a 1 × 1 convolutional kernel with quicksort algorithm.
To realize style and entity transfer, we use SWD to minimize the divergence of the empirical internal 182 distribution of syntheses and reference.
The style loss is defined as
[Equation]
Φ style is a set of convolutional layers from pre-trained lpips network to extract spatial features.
Note 184 that the generator learns the style of ˆ y rec rather than that of y ref to prevent the style of the entityfrom leaking into the other area.
The entity loss is defined as
[Equation]
Φ ent is also a set of convolutional layers from pre-trained lpips net, m ↑ is upsampled by m to getthe same resolution as the final synthesis.
Taking the reconstructed entity m ↑ rec ⊙ y rec
as the targetcan be viewed as an implicit augmentation technique to avoid overfitting.
[Figure]
4.4 Manifold regularization
The training losses, particularly the style loss, inevitably distort the contents of source syntheses, 191 and change their relative relationship.
For achieving the cross-domain correspondence, we use thevariant Laplacian regularization L V LapR to alleviate the distortion.
Since we do not tune the mapping 193 network of StyleGAN, the W space coupled with source generator G s is not changed, we define
[Equation]
where ϕ is a pre-trained semantic extractor, here we use the pre-trained CLIP image encoder [29] 195 that is sensitive to both style and content variation.
L V LapR can be interpreted from two perspectives.
The first one is the smoothness of function: Because the integrand also represents the smoothnessof the residual ϕ ( G t ( w ♯ ))
− ϕ ( G s ( w ♯ )) , minimizing L V lapR expects syntheses from G s and G t 198 across W have a smooth semantic difference.
In the idea case, the integrand is nearly zero, thus forvarious w in a local area, G t ( w ♯ ) and G s ( w ♯ ) have similar difference in terms of style and entity.
The second perspective is the geometric structure preservation:
According to the theory of Laplacian 201 regularization [3], the above integral form in Eq.
(8) can be estimated by a discrete form,
[Equation]
(9a)
=
tr ( R T LR ) , (9b) where R =
[ ϕ ( y 1 ) T − ϕ ( x 1 ) T ; . . .
; ϕ ( y n ) T − ϕ ( x n ) T ] , w i,j = e − ∥ w ♯i − w ♯j ∥ /σ , L is the Laplacianmatrix (see Sec. 3).
In Eq.
(9a), if w i and w j are close in latent space, w i,j will be a large scalarencouraging (cid:13)(cid:13) ϕ
( y i ) − ϕ
( y j ) (cid:13)(cid:13) and ∥ ϕ
( x i ) − ϕ ( x j ) ∥ to be same.
It means the relative distance ofadjacent syntheses before and after adaption shall be isometric in the feature space.
In practice, theloss can be efficiently computed by Eq.
(9b), the derivation is provided in supplementary materials.
Relation between L V lapR and L CDC
Recently the influential work FSGA [27] introduces the 208 cross-domain distance consistency loss L CDC to achieve cross-domain correspondence.
For arbitraryw i and w j , without loss of generality FSGA defines the conditional probability p
j | i and q j | i in Eq.(10) to form the similarity distributions, then minimizes their Kullback-Leibler divergence Eq.
(11) 211 to encourage syntheses of G t have the identical similarity property as that of G s .
[Equation]
[Equation]
In the above equation, ϕ denotes the feature map from the corresponding generator.
From the perspective of manifold learning, p ·| i and q ·| i depict the neighborhood structure of x i and y i respectively,
[Table]
L CDC tries to place x i in a new space so as to optimally preserve neighborhood identity.
Actually,we find that this idea is equivalent to the Stochastic Neighbor Embedding [13, 37], which is also apopular method to preserve the manifold structure.
L CDC has the following disadvantages.
Firstly,it needs a large batch that is nearly unacceptable for high-resolution GAN to estimate p ·| i and q ·| i .
L V LapR directly constrains the distance and performs well even when the batch is 2 .
Secondly,
L CDC is non-convex and hard to optimize, in practice the loss value is usually around 1e-5, implying
L CDC does not provide an effective penalty to the differences.
Finally, because the softmax formof p j | i and q j | i is scale-invariant about the inputs, L CDC cannot guarantee the isometric relationship of source and adapted syntheses like L V LapR , thus it is hard to avoid mode collapse.
In the
Supplementary, we show replacing L CDC with L V LapR can hugely remedy the collapse of FSGA.
Experiments
Our implementation is based on the official code of StyleGAN 2 .
If m ref ̸ = 0 , the total epoch is2000 , we adopt the Adam optimizer with learning rate 1 e − 3 , β 1 = 0 , β 2 = 0 .
The cosineannealing strategy of learning rate is adopted to reduce the learning rate to 1 e − 4 gradually.
In Eq.(2), λ 1 = 10 , λ 2 = 0 . 2 , λ 3 = 2 , λ 4 = 1 , and in Eq.
(4) λ 5 = 100 .
We use the pre-trained lpipsnetwork with vgg architecture [35] containing five convolutional blocks.
In Eq.
(6) and Eq. (7),Φ style = { vgg 1:5 } , Φ ent = { vgg 1:1 , vgg 1:2 , vgg 1:3 , vgg 1:4 } , m = 1 .
In Eq.
(9b), n = 2 includingthe samples from reconstruction and internal distribution learning, σ for calculating the graph weightsis 128 .
We use the Monte Carlo simulation that randomly sampling 256 vectors on the unit sphere tocompute the integral in Eq.
We only augment the reference by the horizontal flip.
If m ref = 0 ,we change λ 2 to 2 , λ 4 to 0 .
5 , Φ style to { vgg 1:4 , vgg 1:5 } , and epoch to 1000 for speeding up training.
5.1 Comparison with SOTA methods
We take the few-shot adaption method FSGA [27], one-shot adaption methods MTG [50] and OSCLIP[19], one-shot face style transfer method JoJoGAN [9] to compare on the commonly studied facedomain.
Since these works can not deal with the entity yet, for a fair and comprehensive comparison,as shown in Fig.
3 we select three portraits without entity (Sketch, Disney, Arcane) from previousworks and three portraits with entities (hat, Zelda elf decorations, and mask) from AAHQ [23] .
Qualitative results Fig.
3 shows the qualitative results.
The top row images are the source naturalfaces, they are inverted by e4e [36] to obtain the latent codes.
From the results, we can draw thefollowing conclusions.
Firstly, our syntheses are of competitive style as the given references.
Thoughshown in thumbnails, they have more obvious high-frequency details to look sharp and realistic,the results of JoJoGAN are most fuzzy.
Secondly, our method achieves the strongest cross-domaincorrespondence that preserves the content and shape of the source image, which is a great advancecompared with other methods without a doubt.
It also implies our method can keep the diversity ofthe source model without content collapse.
Thirdly, our method can generate high-quality entities,which are adaptive to the various face shapes and look harmonious with the rest area of the syntheses.
Experiments prove that our method has sufficient visual advantages over the competition.
[Figure]
Quantitative results
Previous works mainly conduct user studies to compare the visual quality.
Following them, we survey user preferences for different methods.
We first randomly sample 50 latentcodes in W space and feed them into various models.
We provide the references, source syntheses, 255 and adapted syntheses to volunteers, they will consider both style and content and subjectively vote fortheir favorite adapted syntheses.
Moreover, to objectively evaluate the cross-domain correspondence, 257 we use the face alignment network [7] to extract facial landmarks and calculate the Normalized meanerror (NME) of landmarks between source syntheses and adapted syntheses.
[Equation]
lmk denotes the 68 points two-dimension landmarks extractor, n is set to 1000.
NME reflectsthe face shape difference before and after adaption.
We further report the identity (ID) similaritypredicted by the Arcface [11] to metric the preservation of identity information.
The model of thelowest NME and highest ID has the best cross-domain corresponding property.
The quantitativeresults have been shown in Table 1.
Our method undoubtedly obtains the best scores in terms of NMEand ID.
It performs stably and gets similar NME in different domains.
FSGA and OSCLIP obtainrelatively worse scores because of the overfitting as shown in Fig.
3, their common characteristicis that the GAN discriminator is used for training.
MTG performs very unsteadily especially whenthe target domains have strong semantic style like Disney and Zelda.
Compared with our approach
JoJoGAN does similar well on NME, but poorly on ID.
For the user study, we finally collect validvotes from 53 volunteers, and a user averagely spends 4 seconds on one question.
It shows ourmethod obtains about 60 % of the votes on each adapted model, whereas other methods evenly acquirethe rest of the votes, proving that our method is much more popular with users.
Training and inference time Our method costs about 12 minutes for m ref ̸ = 0 and 4 minutes form ref = 0 on NVIDIA RTX 3090.
FSGA, MTG, JoJoGAN, OSCLIP cost about 48, 9, 2, 24 minutesrespectively.
All of them take about 30ms to generate an image.
5.2 Ablation study
We start from the basic reconstruction loss L rec and continuously improve the framework to observethe qualitative changes of syntheses.
As illustrated in Fig.
4, the target exemplar is a hard casedepicting a man with a beard and cap, and the style fixation can bring a rough but remarkableimprovement to the style of synthesis.
Although L style can enhance the style of the face, it deterioratesthe hair and introduces the noise like the beard to the female face.
L ent plays a critical rolein the generation of entity, without it the cap can not be synthesized.
Notably, L V LapR has asignificant suppression of the noise in previous syntheses.
Finally, we remove L style and fixationstyle respectively and the syntheses are no longer of the target style.
It means that both of them playvital roles in style transfer.
We also provide the quantitative ablation in Table 1 to study the impactsof L style and L V LapR .
It shows that removing L style has a positive effect on NME and ID whileremoving L V LapR has an opposite effect.
The quantitative results confirm that L V LapR is necessaryfor preventing content from being distorted by style loss.
[Figure] [Figure]
5.3 More resultsAdaption to other domains
In addition to the success of the face domain, our framework canbe applied to other source domains.
We take StyleGANs pre-trained on 512 × 512 AFHQ dog,cat dataset [8] and 256 × 256 LSUN church dataset [45] as exemplars.
As shown in Fig.
5, foreach source domain, we provide the images with entities and without entity, the source images arerandomly sampled from the source space.
For the images without entity, the adapted syntheses arerealistic and have a strong corresponding relationship in visual.
For the other case, our approach alsoachieves satisfactory results, for example, the cartoon eyes of the dogs are in the right place and verysimilar to that of the reference.
It proves that our framework is valid and has a wide universality.
Image manipulation
Because our adapted model retains the geometric structure of the sourcegenerative space, it can efficiently perform semantic editing on image using the semantic directionfound in the source space.
As shown in Fig.
6, since our model is able to accurately reconstructthe target exemplar, we can edit both the exemplar and synthesis.
We select three representativesemantic directions [33] to change smile, age, and pose rotation, the results show that the adaptedgenerator achieves exact control not only on the face but also on the entity.
In addition, our modelcan also change the style by altering the style code, which is hard for previous works.
Moreover, ourframework can be applied to remove the entity with ˆ G t , and the result can be manipulated similarly.
We consider these functions will be helpful for artistic creation.
Conclusion and limitations
In this paper, we construct a new GAN adaption framework for the generalized one-shot GANadaption task.
Because our framework relies heavily on the learning of internal distribution, it hassome limitations inevitably.
We think the most important one is that it cannot precisely controlthe position of the entity, which may lead to failures when the pose changes too much.
Anotherlimitation is that the entity cannot be too complex, otherwise it is difficult to learn by patch distribution.
However, these extreme cases require additional consideration.
In general, our framework is effectivein various scenarios.
We believe the exploration of the novel task and the introduction of manifoldregularization are meaningful for the later work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No](d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] The images we use are widely used in previous works or from
open dataset.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The images we use are widely used in previous
works or from open dataset.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[No]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]