Generalized One-shot Domain Adaption of Generative Adversarial Networks
[Figure]
Abstract
The adaption of Generative Adversarial Network (GAN) aims to transfer a pretrained GAN to a given domain with limited training data.
In this paper, wefocus on the one-shot case, which is more challenging and rarely explored inprevious works.
We consider that the adaptation from source domain to targetdomain can be decoupled into two parts: the transfer of global style like textureand color, and the emergence of new entities that do not belong to the sourcedomain.
While previous works mainly focus on the style transfer, we propose anovel and concise framework 1 to address the generalized one-shot adaption taskfor both style and entity transfer, in which a reference image and its binary entitymask are provided.
Our core objective is to constrain the gap between the internaldistributions of the reference and syntheses by sliced Wasserstein distance.
Tobetter achieve it, style fixation is used at first to roughly obtain the exemplary style,and an auxiliary network is introduced to the original generator to disentangleentity and style transfer.
Besides, to realize cross-domain correspondence, wepropose the variational Laplacian regularization to constrain the smoothness of theadapted generator.
Both quantitative and qualitative experiments demonstrate theeffectiveness of our method in various scenarios.
Introduction
Benefiting from numerous excellent pre-trained GANs ( e.g., StyleGAN [17] and BigGAN [6]), GANadaption has become an important research topic, which leverages the knowledge of GANs pretrained on large-scale datasets to mitigate the lack of data and speed up the training on a new domain.
It can be divided into few-shot, one-shot and zero-shot [11] cases.
The few-shot (usually ≥ 10 ) casehas been extensively studied in the past years [43, 42, 27, 24].
The latest works [30, 44, 47] seek thecross-domain correspondence that keeps the shapes or contents of syntheses before and after adaptioninvariant.
In recent time, the more challenging one-shot case has been explored by several works[48, 54, 21], which mainly focus on adapting GANs to the target domain of similar style as the givenimage.
By utilizing GAN inversion [46], the adapted models can be applied to image style transferand other manipulation tasks.
However, there exist limitations in previous one-shot task settings, since an exemplar contains richinformation more than general color and texture style.
As shown in Fig.
1, the source domain refersto natural faces, and the exemplars of target domains are artistic portraits with some entities ( e.g.,glasses or accessories).
Previous works are only able to transfer the artistic style while the entities areneglected.
This may lead to two problems: 1)
In some cases, the entity ( e.g., hat), considered as animportant part of style, should be transferred with the color style simultaneously.
For the entitiesof large size, it easily causes artifacts by previous works due to the influence of entities on the colorspace.
To tackle the above problems, we describe the target domain with the aid of an extra binarymask, which labels the entities we are interested in and helps to define the target domain more exactlyand flexibly.
We name the new task generalized one-shot GAN adaption .
The previous scenarios canbe taken as the special case when the binary mask is full of zeros.
We believe the exploration of thenovel task is quite meaningful, which not only allows customization for more complex scenarios, butalso facilitates further discovery and utilization of the prior knowledge in pre-trained models.
To tackle the new adaption problem, we propose a concise and effective adaption framework for
StyleGAN [18], which is a frequent base model in previous works [48, 54, 21].
Firstly, we modifythe architecture of the original generator to decouple the adaption of style and entity , where thenew generator comprises an additional auxiliary network to facilitate the generation of entity, andthe original generator is dedicated to generating stylized images.
Secondly, unlike previous worksusing CLIP similarity [54] or GAN loss [48] to learn the domain knowledge, our framework directlyminimizes the divergence of the internal distributions of the exemplar and syntheses by sliced Wasserstein distance [5] for both style and entity adaption. Combined with the style fixation that attachesall syntheses with exemplary style by the transformation in latent space, our framework is efficientand potent enough to obtain compelling results.
Thirdly, inspired by the cross-domain consistency[30] and classic manifold learning [12, 2], we propose the variant Laplacian regularization thatsmooths the network changes before and after adaptation to preserve the geometric structure of thesource generative manifold, and prevent the content distortion during training.
We conduct extensiveexperiments on various references with and without entities.
The results show that our frameworkcan fully exploit the cross-domain correspondence and achieve high transfer quality.
Moreover, theadapted model can handily perform various image manipulations beyond previous works.
Related works
GAN adaption leverages the knowledge stored in pre-trained GANs to mitigate the lack of data andspeed up the training on a new domain.
[43] is the first to study and evaluate different adaptingstrategies of GANs.
It shows that transferring both pre-trained generator and discriminator canimprove the convergence time and the quality of the generated images.
Since then, more and moreadapting strategies [29, 42, 33, 53] are proposed for various GAN architectures [14, 6, 26].
As
StyleGAN
[17] and its variants [16, 15] continue to make breakthrough progress in the generation ofvarious classes of data, designing the adapted strategies of StyleGAN has gained a lot and sustainedattention.
Specifically, [31, 20] have explored the zero-shot GAN adaption, which transfers StyleGANinto the target domain defined by the text prompts with the help of CLIP [32].
[24, 30, 22, 44, 47, 19]study to transfer the StyleGAN to the domain defined by the given few-shot images.
Among the above works, FSGA [30] has a close relation to our work.
It proposes the cross-domainconsistency (CDC) loss to maintain the diversity of the source generator and makes the transferredand source samples have a corresponding relation like image-to-image translation [13].
Although
CDC loss is ineffective for FSGA when the training image is one-shot, its mechanism inspires us toconsider a more effective regularization to maintain the relation, which will be discussed in Sec. 4.4.
Recently, [54, 21, 9, 48] have also considered the one-shot adaption task.
[54, 21] align the generatedsamples and target image in the CLIP semantic space.
[9] learns a style mapper by constructing asubstantial paired dataset.
[48] introduces the extra latent mapper and classifier to the original GAN.
These methods almost suffer the content distortion.
In contrast, we focus on both style and entityadaption that has never been studied before.
we show that under the decoupling of style and entity,aligning the internal distributions with the proper regularization is effective for domain adaption.
PreliminariesStyleGAN
A canonical StyleGAN [18] can be summarized in two parts: Firstly, given a noisedistribution P ( z ) , a mapping network transforms the noise from space Z = { z | P ( z ) ̸ = 0 } ⊆ R 1 × 512into the replicated latent space W = { [ F ( z ); . . .
; F ( z )] | z ∈ Z} ⊆ R 18 × 512 , where F is a fullyconnected network.
Secondly, a synthesis network G composed of convolutional blocks transfers
W space into image space M = { G ( w ) |
w ∈ W} ⊆
R H
× W × 3 .
Another important concept is theextended latent space W + =
{ [ F ( z 1 ); . .
F ( z 18 )]
| z i ∈ Z} .
For ∀ w 1 , w 2 ∈ W , the latent codemanipulation merges their information by the linear combination diag ( α 1 ) w 1 + diag ( α 2 ) w 2 ∈ W + ,α 1 , α 2 ∈ R 18 .
Empirically, W + space is inferior to W in generation fidelity and editability, butsuperior in GAN inversion that finds a code w ref to reconstruct a reference
y ref ∈
R
H
× W × 3 .
Inthis paper we only focus on tuning G , and fix F for keeping the distribution of source latent spaces.
Manifold learning Considering the sample matrix
X =
[ x T 1 ; . . .
; x Tn ] from source manifold M s ,
W = [ w i,j ] ni,j =1 is the weight matrix, where w i,j is the weight of x i and x j usually defined ase −∥ x
i − x j ∥ 2 /σ .
Given a task specific function f and y = f ( x ) , manifold learning requires the transformed samples Y =
[ y T 1 ; . . .
; y Tn ] on the new manifold M t should preserve the geometric structureof M s , which means the data nearby on M s are also nearby on M t , and vice versa.
The requirementis mostly realized by minimizing the manifold regularization term [4] (cid:82) M s ∥∇ M s y
∥ 2 dP ( x ) , ∇ M sis the gradient along manifold, and P ( x ) is the probability measure.
If M t is a submanifold of R , theformer is equalized with (cid:82) M s y ∆ M s y dP ( x ) by Stokes theorem.
∆ called the Laplace–Beltrami operator, is a key geometric object in the Riemannian manifold, thus the integral is also called Laplacianregularization.
In practice, it is estimated by the discrete form 2 tr ( Y T LY ) =
(cid:80) i,j w i,j (cid:13)(cid:13)
y i − y j
(cid:13)(cid:13) 2 ,
L = D − W is known as the graph Laplacian operator, D is the degree matrix where the diagonalelement d i,i = (cid:80) nj
=1 w i,j and other elements are zero.
The convergence between L and ∆ can befound in [3].
Intuitively, the regularization encourages the local smoothness of f .
If data distributedensely in an area, the function should be smooth to avoid local oscillations.
If x i and x j are close,in the discrete form w i,j will impose a strong penalty to the distance between y i and y j .
Task
definition The generalized one-shot adaption can be formally described as: Given a referencey ref , a mask m ref ∈ { 0 , 1 } H × W is provided to label the location of entity contained in y ref .
The target domain T is defined to contain images of similar style and entities as y ref .
With theknowledge stored in a generative model G s pre-trained on source domain S , a generative model G tis learned from y ref and m ref to generate diverse images belonging to domain T .
Note that we canset m ref = 0 when there is no interested entities existed in y ref .
Moreover, the adaptation shouldsatisfy the cross-domain correspondence.
i.e. , ∀ w ∈ W , G s ( w ) and G t ( w ) should have similarshape or content in visual.
With the correspondence, adapted models can be applied for not onlysynthesis, but also the transfer and manipulation of source images.
Methodology
4.1 Overall
Our framework for tackling the generalized one-shot GAN adaption task is illustrated in Fig.
Forthe sake of description, in the following part, we will take the StyleGAN pre-trained on FFHQ as an 𝒚 𝒓𝒆𝒄 𝒚(cid:3549) 𝒓𝒆𝒄 𝒘 ⋕ 𝒚(cid:3549)
A u x
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k ⋯ ⋯ ` ⋯ 𝑢𝑛𝑒𝑡 𝒚 𝒓𝒆𝒇 𝒙 𝒓𝒆𝒄 𝓛 𝒓𝒆𝒄 𝓛 𝑽𝑳𝒂𝒑𝑹 𝒚
𝓛 𝒔𝒕𝒚𝒍𝒆 (a) Reconstruction (c) Loss (d) Auxiliary net (b) Random synthesis
Legend
Conv Block Conv Block
Style injector
StyleGANblock
𝒘 𝒓𝒆𝒇 𝓛 𝒆𝒏𝒕 𝒙 𝑮 𝒔 (cid:4666)𝒘(cid:4667) ⋯
A u x
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k
Style fixation
Flow of 𝐺 (cid:3047)
Flow of 𝐺(cid:3552) (cid:3047)
Conv Block example to describe each component of the framework in detail.
We will demonstrate our frameworkcan be applied to other source domains in the experiments.
Network architecture
Different from previous works [54, 21] that directly adapt the sourcegenerator G s , our target generator G t (as Fig.
2 shows) consists of a generator ˆ G t inherited from
G s , and an auxiliary network ( aux ) trained from scratch.
The design is inspired by the fact that G scan approximate arbitrary real faces but cannot handle most entities since G s captures the prevalentelements (clear natural faces) of FFHQ rather than the tails (faces with entities like hats or ornaments)of the distribution [50, 28].
We make aux serve to entities, and ˆ G t just focus on stylizing clear faceto benefit from the prior knowledge stored in G s .
As shown in Fig. 2(d), for each w , its feature mapgenerated by the convolutional block of StyleGAN is fed into aux , where a UNet [34] predicts thefeature map f ent of entity and the mask m that exactly labels the position of the entity, then they aremerged with f in by Eq.
(1) to get the feature map of face covered with entity.
[Equation]
⊙ denotes the Hadamard product.
We hope the shape of the entity is only influenced by the contentof synthesis, hence we place aux after the fourth block of StyleGAN, where the 32 × 32 resolutionf in contains the most content information and the least style information of synthesis (see Sec.4.2).
Because the global style and local entity are usually independent, the disentangled design alsoprevents the style of faces from being contaminated by that of entities (see Sec. 4.3).
Training process Our training process can be divided into three main parts: 1 ⃝ Style fixation andexemplar reconstruction.
In Fig. 2(a), we get the latent code w ref =
[ w cref ; w sref ] of y ref by the
GAN inversion technique [40], and train G t to reconstruct y ref with reconstruction loss L rec .
In
Fig. 2(b), each w will be transformed into the style-fixed code w ♯ =
[ w c ; w sref ] to roughly obtainthe exemplary style.
2 ⃝ Internal distribution learning.
We minimize the divergence of internal patchdistributions between syntheses and exemplar for style and entity transfer.
Instead of using GANloss that is hard to optimize, we take sliced Wasserstein distance (SWD) [5] as style loss L style andentity loss L ent to learn internal distribution efficiently.
3 ⃝ Manifold Regularization.
To suppressthe content distortion during training, we propose the variational Laplacian regularization L V lapR tosmooth the change from G s to G t by preserving the geometric structure of the source manifold.
Asshown in Fig. 2(c), the overall loss is
[Equation]
We will illustrate and discuss the training process in detail in the following sections.
For convenienceof description and self-consistency, we may use the abbreviations:
y =
G t ( w ♯ ) , ˆ y
=
ˆ G t ( w ♯ ) ,y rec =
G t ( w ref ) , ˆ y rec =
ˆ G t ( w ref ) , x = G s ( w ♯ ) , x rec =
G s ( w ref ) .
4.2 Style fixation and exemplar reconstruction
It is well known that StyleGAN is equipped with the disentangled latent spaces W and W + [1],where the latter vectors of code mainly determine the style of synthesis whereas the earlier vectorsdetermine the coarse-structure or content of the synthesis.
Taking advantage of this trait, we firstroughly transfer the exemplary style to other syntheses by fixing the style code.
The exemplar y ref isfed into the pre-trained GAN inversion encoder e4e [40] to get the latent code w ref =
[ w cref ; w sref ] ,where w cref ∈ R l × 512 and w sref ∈ R (18 − l ) × 512 respectively encode content and style information ofexemplar.
For each w ∈ W , it will be transformed to w ♯ before fed into networks,
[Equation]
The content part of w is reserved and style part is replaced by w sref , and we denote the new spaceas W ♯ .
Because style fixation should not change the content of the original synthesis, by using thepre-trained Arcface model [10] to compare the identity similarity before and after style fixation, wefind setting l = 8 is acceptable which gets 65% average similarity and obtains the adequate exemplarystyle in visual.
In MTG [54], style mixing assists as a post-processing step to improve the stylequality, but the experiments (Fig. 3) show that the style of syntheses is underfitting and not consistent.
In our framework, it is used as a pre-processing step to facilitate the following learning.
Because x rec is just the projection of y ref on the source domain, there often exists obvious distinctionbetween them in visual as shown in Fig.
We use the reconstruction loss L rec to narrow the gap,
[Equation]
dssim denotes the negative structural similarity metric [45] and lpips is the perceptual loss [51],m ↑ rec is the upsampled reconstructed mask from aux .
It is worth noting L rec acts on G t rather thanthe vanilla architecture ˆ G t .
If m ref = 0 , aux does not work and G t be equivalent to ˆ G t .
4.3 Internal distribution learning
Recent works about internal learning [36, 38, 52, 23] prove that the internal patch distribution of 171 a single image contains rich meaning.
Minimizing the divergence of internal distributions worksout many tasks like image generation, style transfer, and super-resolution.
It is usually achieved 173 by the adversarial patch discriminator [13].
However, it is well known that the training of GAN istime-consuming, unstable, and requires large GPU memory.
Previous few-shot GAN adaption works 175 [24, 30] using GAN loss almost have the serious model collapse phenomenon.
We find that slice
Wasserstein distance ( SWD ) [5] can lead to the same destination but with greater efficiency.
For twotensor A , B ∈ R H × W × d , the SWD of their empirical internal distributions is defined as
[Equation]
where S d = { θ ∈ R d : ∥ θ ∥ = 1 } , proj : R H × W × d → R H × W that projects each pixel from dchannels to a scalar by θ , sort denotes the sort operator ordering all values.
SWD does not introduceextra parameter, and can be readily realized by a 1 × 1 convolutional kernel with quicksort algorithm.
To realize style and entity transfer, we use SWD to minimize the divergence of the empirical internal 182 distribution of syntheses and reference.
The style loss is defined as
[Equation]
Φ style is a set of convolutional layers from pre-trained lpips network to extract spatial features.
Note 184 that the generator learns the style of ˆ y rec rather than that of y ref to prevent the style of the entityfrom leaking into the other area.
The entity loss is defined as
[Equation]
Φ ent is also a set of convolutional layers from pre-trained lpips net, m ↑ is upsampled by m to getthe same resolution as the final synthesis.
Taking the reconstructed entity m ↑ rec ⊙ y rec
as the targetcan be viewed as an implicit augmentation technique to avoid overfitting.
[Figure]
4.4 Manifold regularization
The training losses, particularly the style loss, inevitably distort the contents of source syntheses, 191 and change their relative relationship.
For achieving the cross-domain correspondence, we use thevariant Laplacian regularization L V LapR to alleviate the distortion.
Since we do not tune the mapping 193 network of StyleGAN, the W space coupled with source generator G s is not changed, we define
[Equation]
where ϕ is a pre-trained semantic extractor, here we use the pre-trained CLIP image encoder [32] 195 that is sensitive to both style and content variation.
L V LapR can be interpreted from two perspectives.
The first one is the smoothness of function: Because the integrand also represents the smoothnessof the residual ϕ ( G t ( w ♯ ))
− ϕ ( G s ( w ♯ )) , minimizing L V lapR expects syntheses from G s and G t 198 across W have a smooth semantic difference.
In the idea case, the integrand is nearly zero, thus forvarious w in a local area, G t ( w ♯ ) and G s ( w ♯ ) have similar difference in terms of style and entity.
The second perspective is the geometric structure preservation:
According to the theory of Laplacian 201 regularization [3], the above integral form in Eq.
(8) can be estimated by a discrete form,
[Equation]
(9a)
=
tr ( R T LR ) , (9b) where R =
[ ϕ ( y 1 ) T − ϕ ( x 1 ) T ; . . .
; ϕ ( y n ) T − ϕ ( x n ) T ] , w i,j = e − ∥ w ♯i − w ♯j ∥ /σ , L is the Laplacianmatrix (see Sec. 3).
In Eq.
(9a), if w i and w j are close in latent space, w i,j will be a large scalarencouraging (cid:13)(cid:13) ϕ
( y i ) − ϕ
( y j ) (cid:13)(cid:13) and ∥ ϕ
( x i ) − ϕ ( x j ) ∥ to be same.
It means the relative distance ofadjacent syntheses before and after adaption shall be isometric in the feature space.
In practice, theloss can be efficiently computed by Eq.
(9b), the derivation is provided in supplementary materials.
Relation between L V lapR and L CDC
Recently the influential work FSGA [30] introduces the 208 cross-domain distance consistency loss L CDC to achieve cross-domain correspondence.
For arbitraryw i and w j , without loss of generality FSGA defines the conditional probability p
j | i and q j | i in Eq.(10) to form the similarity distributions, then minimizes their Kullback-Leibler divergence Eq.
(11) 211 to encourage syntheses of G t have the identical similarity property as that of G s .
[Equation]
[Equation]
In the above equation, ϕ denotes the feature map from the corresponding generator.
From the perspective of manifold learning, p ·| i and q ·| i depict the neighborhood structure of x i and y i respectively,
[Table]
L CDC tries to place x i in a new space so as to optimally preserve neighborhood identity.
Actually,we find that this idea is equivalent to the Stochastic Neighbor Embedding [12, 41], which is also apopular method to preserve the manifold structure.
L CDC has the following disadvantages.
Firstly,it needs a large batch that is nearly unacceptable for high-resolution GAN to estimate p ·| i and q ·| i .
L V LapR directly constrains the distance and performs well even when the batch is 2 .
Secondly,
L CDC is non-convex and hard to optimize, in practice the loss value is usually around 1e-5, implying
L CDC does not provide an effective penalty to the differences.
Finally, because the softmax formof p j | i and q j | i is scale-invariant about the inputs, L CDC cannot guarantee the isometric relationship of source and adapted syntheses like L V LapR , thus it is hard to avoid mode collapse.
In the
Supplementary, we show replacing L CDC with L V LapR can hugely remedy the collapse of FSGA.
Experiments
Our implementation is based on the official code of StyleGAN 2 .
If m ref ̸ = 0 , the total epoch is2000 , we adopt the Adam optimizer with learning rate 1 e − 3 , β 1 = 0 , β 2 = 0 .
The cosineannealing strategy of learning rate is adopted to reduce the learning rate to 1 e − 4 gradually.
In
Eq.
(2), λ 1 = 10 , λ 2 = 0 .
2 , λ 3 = 2 , λ 4 = 1 , and in Eq.
(4) λ 5 = 100 .
We use the pre-trainedlpips network with vgg architecture [39] containing five convolutional blocks.
In Eq.
(6) and Eq.(7), Φ style = { vgg 1:4 , vgg 1:5 } , Φ ent = { vgg 1:1 , vgg 1:2 , vgg 1:3 , vgg 1:4 } , m = 1 .
In Eq.
(9b),
n = 2including the samples from reconstruction and internal distribution learning, σ for calculating thegraph weights is 128 .
We use the Monte Carlo simulation that randomly sampling 256 vectors on theunit sphere to compute the integral in Eq.
We only augment the reference by the horizontal flip.
If m ref = 0 , we change λ 2 to 2 , λ 4 to 0 .
5 , and epoch to 1000 for speeding up training.
5.1 Comparison with SOTA methods
We take the few-shot adaption method FSGA [30], one-shot adaption methods MTG [54] and OSCLIP[21], one-shot face style transfer method JoJoGAN [9] to compare on the commonly studied facedomain.
Since these works can not deal with the entity yet, for a fair and comprehensive comparison,as shown in Fig.
3 we select three portraits without entity (Sketch, Disney, Arcane) from previousworks [35, 9] and three portraits with entities (hat, Zelda decorations, and mask) from AAHQ [25].
Qualitative results Fig.
3 shows the qualitative results.
The top row images are the source naturalfaces, they are inverted by e4e [40] to obtain the latent codes.
From the results, we can draw thefollowing conclusions.
Firstly, our syntheses are of competitive style as the given references.
Thoughshown in thumbnails, they have more obvious high-frequency details to look sharp and realistic,the results of JoJoGAN are most fuzzy.
Secondly, our method achieves the strongest cross-domaincorrespondence that preserves the content and shape of the source image, which is a great advancecompared with other methods without a doubt.
It also implies our method can keep the diversity ofthe source model without content collapse.
Thirdly, our method can generate high-quality entities,which are adaptive to the various face shapes and look harmonious with the rest area of the syntheses.
Experiments prove that our method has sufficient visual advantages over the competition.
[Figure]
Quantitative results
Previous works mainly conduct user studies to compare the visual quality.
Following them, we survey user preferences for different methods.
We first randomly sample 50 latentcodes in W space and feed them into various models.
We provide the references, source syntheses, 255 and adapted syntheses to volunteers, they will consider both style and content and subjectively vote fortheir favorite adapted syntheses.
Moreover, to objectively evaluate the cross-domain correspondence, 257 we use the face alignment network [7] to extract facial landmarks and calculate the Normalized meanerror (NME) of landmarks between source syntheses and adapted syntheses.
[Equation]
lmk denotes the 68 points two-dimension landmarks extractor, n is set to 1000.
NME reflectsthe face shape difference before and after adaption.
We further report the identity (ID) similaritypredicted by the Arcface [10] to metric the preservation of identity information.
The model of thelowest NME and highest ID has the best cross-domain corresponding property.
The quantitativeresults have been shown in Table 1.
Our method undoubtedly obtains the best scores in terms of NMEand ID.
It performs stably and gets similar NME in different domains.
FSGA and OSCLIP obtainrelatively worse scores because of the overfitting as shown in Fig.
3, their common characteristicis that the GAN discriminator is used for training.
MTG performs very unsteadily especially whenthe target domains have strong semantic style like Disney and Zelda.
Compared with our approach
JoJoGAN does similar well on NME, but poorly on ID.
For the user study, we finally collect validvotes from 53 volunteers, and a user averagely spends 4 seconds on one question.
It shows ourmethod obtains about 60 % of the votes on each adapted model, whereas other methods evenly acquirethe rest of the votes, proving that our method is much more popular with users.
Training and inference time Our method costs about 12 minutes for m ref ̸ = 0 and 3 minutes form ref = 0 on NVIDIA RTX 3090.
FSGA, MTG, JoJoGAN, OSCLIP cost about 48, 9, 2, 24 minutesrespectively.
All of them take about 30ms to generate an image.
5.2 Ablation study
We start from the basic reconstruction loss L rec and continuously improve the framework to observethe qualitative changes of syntheses.
As illustrated in Fig.
4, the target exemplar is a hard casedepicting a man with a beard and cap, and the style fixation can bring a rough but remarkableimprovement to the style of synthesis.
Although L style can enhance the style of the face, it deterioratesthe hair and introduces the noise like the beard to the female face.
L ent plays a critical rolein the generation of entity, without it the cap can not be synthesized.
Notably, L V LapR has asignificant suppression of the noise in previous syntheses.
Finally, we remove L style and stylefixation respectively and the syntheses are no longer of the target style.
It means that both of themplay vital roles in style transfer.
We also provide the quantitative ablation in Table 1 to study theimpacts of L style and L V LapR .
It shows that removing L style has a positive effect on NME and
ID while removing L V LapR has an opposite effect.
The quantitative results confirm that L V LapR isnecessary for preventing content from being distorted by style loss.
[Figure] [Figure]
5.3 More resultsAdaption to other domains
In addition to the success of the face domain, our framework canbe applied to other source domains.
We take StyleGANs pre-trained on 512 × 512 AFHQ dog,cat dataset [8] and 256 × 256 LSUN church dataset [49] as exemplars.
As shown in Fig.
5, foreach source domain, we provide the images with entities and without entity, the source images arerandomly sampled from the source space.
For the images without entity, the adapted syntheses arerealistic and have a strong corresponding relationship in visual.
For the other case, our approach alsoachieves satisfactory results, for example, the cartoon eyes of the dogs are in the right place and verysimilar to that of the reference.
It proves that our framework is valid and has a wide universality.
Image manipulation
Because our adapted model retains the geometric structure of the sourcegenerative space, it can efficiently perform semantic editing on image using the semantic directionfound in the source space.
As shown in Fig.
6, since our model is able to accurately reconstructthe target exemplar, we can edit both the exemplar and synthesis.
We select three representativesemantic directions [37] to change smile, age, and pose rotation, the results show that the adaptedgenerator achieves exact control not only on the face but also on the entity.
In addition, our modelcan also change the style by altering the style code, which is hard for previous works.
Moreover, ourframework can be applied to remove the entity with ˆ G t , and the result can be manipulated similarly.
We consider these functions will be helpful for artistic creation.
Conclusion and limitations
In this paper, we construct a new GAN adaption framework for the generalized one-shot GANadaption task.
Because our framework relies heavily on the learning of internal distribution, it hassome limitations inevitably.
We think the most important one is that it cannot precisely controlthe position of the entity, which may lead to failures when the pose changes too much.
Anotherlimitation is that the entity cannot be too complex, otherwise it is difficult to learn by patch distribution.
However, these extreme cases require additional consideration.
In general, our framework is effectivein various scenarios.
We believe the exploration of the novel task and the introduction of manifoldregularization are meaningful for the later work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[No] The images we use are widely used in previous works or from
open dataset.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] The images we use are widely used in previous
works or from open dataset.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]