Generalized One-shot Domain Adaptation of Generative Adversarial Networks
[Figure] g e
[Figure]
Abstract
The adaptation of a Generative Adversarial Network (GAN) aims to transfer a pre-trained GAN to a target domain with limited training data.
In this paper, we focus on the one-shot case, which is more challenging and rarely explored in previous works.
We consider that the adaptation from a source domain to a target domain can be decoupled into two parts: the transfer of global style like texture and color, and the emergence of new entities that do not belong to the source domain.
While previous works mainly focus on style transfer, we propose a novel and concise framework to address the generalized one-shot adaptation task for both style and entity transfer, in which a reference image and its binary entity mask are provided.
Our core idea is to constrain the gap between the internal distributions of the reference and syntheses by sliced Wasserstein distance.
To better achieve it, style ﬁxation is used at ﬁrst to roughly obtain the exemplary style, and an auxiliary network is introduced to the generator to disentangle entity and style transfer.
Besides, to realize cross-domain correspondence, we propose the variational Laplacian regularization to constrain the smoothness of the adapted generator.
Both quantitative and qualitative experiments demonstrate the effectiveness of our method in various scenarios.
Code is available at https: //github.com/zhangzc21/Generalized-One-shot-GAN-Adaptation .
Introduction
Beneﬁting from numerous excellent pre-trained GANs ( e.g., StyleGAN [16] and BigGAN [6]),
GAN adaptation has become an important research topic, which leverages the knowledge of GANs pre-trained on large-scale datasets to mitigate the lack of data and speed up the training on a new domain.
The adaptation can be divided into few-shot, one-shot and zero-shot [11] cases.
The few-shot (usually ≥ 10 ) case has been extensively studied in the past years [39, 38, 24, 22].
The latest works [27, 40, 43] seek the cross-domain correspondence that keeps the shapes or contents of syntheses before and after adaptation invariant.
Recently, the more challenging one-shot case has been explored by several works [44, 51, 20], which mainly focus on adapting GANs to the target domain of similar style as the given image.
By utilizing GAN inversion [42], the adapted models can be applied to some creative tasks like image style transfer and manipulation [51].
However, we perceive there exist limitations in previous task settings [44, 51, 20], due to the fact that an exemplar contains rich information more than artistic style including color and texture.
As shown in Fig.
1, the source domain refers to natural faces, and the exemplars of target domains are artistic portraits with some entities ( e.g., hat and accessories).
Previous works only concern the artistic style transfer while the entities are neglected.
This may lead to two problems: 1)
In some cases, the entities considered as an important part of domain feature, should be transferred with the style simultaneously.
The entities of large size, like the mask in Fig.
1, easily bias the style extraction of face region and cause undesired artifacts in adapted syntheses.
Therefore, we perfect the task settings with the aid of an extra binary mask, which labels the entities we are interested in and helps to deﬁne the target domain more exactly and ﬂexibly.
We name the new task generalized one-shot GAN adaptation .
The scenarios in previous works can be taken as the special case when the binary mask is full of zeros.
We believe the exploration of the novel task is quite meaningful, which not only serves to artistic creation for more general and complex scenarios, but also facilitates further discovery and utilization of the knowledge in pre-trained models.
To tackle the new adaptation problem, we propose a concise and effective adaptation framework for StyleGAN [17], which is a frequent base model in previous works [44, 51, 20].
Firstly, we modify the architecture of the original generator to decouple the adaptation of style and entity , where the new generator comprises an additional auxiliary network to facilitate the generation of entities, and the original generator is dedicated to generating stylized images.
Secondly, unlike previous works using CLIP similarity [51] or GAN loss [44] to learn the domain knowledge, our framework directly minimizes the divergence of the internal distributions of the exemplar and syntheses by sliced Wasserstein distance [5] for both style and entity transfer.
Combined with the style ﬁxation that attaches all syntheses with exemplary style by the transformation in latent space, our framework is efﬁcient and potent enough to obtain compelling results.
Thirdly, inspired by the cross-domain consistency [27] and classic manifold learning [12, 2], we propose the variant
Laplacian regularization that smooths the network changes before and after adaptation to preserve the geometric structure of the source generative manifold, and prevent the content distortion during training.
We conduct extensive experiments on various references with and without entities.
The results show that our framework can fully exploit the cross-domain correspondence and achieve high transfer quality.
Moreover, the adapted model can handily perform various image manipulation tasks.
Related works
GAN adaptation leverages the knowledge stored in pre-trained GANs to mitigate the lack of data and speed up the training on a new domain.
[39] is the ﬁrst to study and evaluate different adapting strategies.
It shows that transferring both pre-trained generator and discriminator can improve the convergence time and the quality of the generated images.
Since then, more and more adapting strategies [26, 38, 30, 50] are proposed for various GAN architectures [14, 6 ? ].
As StyleGAN [16] and its variants [17, 15] continue to make breakthrough progress in the generation of various classes of data, designing the adapted strategies of StyleGAN has gained a lot and sustained attention.
Speciﬁcally, [28, 19] have explored the zero-shot GAN adaptation, which transfers StyleGAN into the target domain deﬁned by the text prompts with the help of CLIP [29].
[22, 27, 21, 40, 43, 18] study to transfer the StyleGAN to the domain deﬁned by the given few-shot images.
Among the above works, our work has a close relation to FSGA [27], which proposes the crossdomain consistency (CDC) loss to maintain the diversity of the source generator, and makes the transferred and source samples have a corresponding relation.
Although CDC loss is ineffective for
FSGA when the training data is only one-shot, its mechanism inspires us to consider a more effective regularization to maintain the relation, which will be discussed in Sec. 4.4.
Recently, [51, 20, 9, 44] have explored the one-shot adaptation task.
[51, 20] align the adapted samples and reference image in the CLIP feature space.
[9] learns a style mapper by constructing a substantial paired dataset.
[44] introduces the extra latent mapper and classiﬁer to the original GAN.
In contrast, we focus on both style and entity adaptation which has never been studied before.
And we prove that under the decoupling of style and entity, aligning the internal distributions with the proper regularization is effective for GAN adaptation.
Preliminaries StyleGAN
A canonical StyleGAN [17] can be summarized in two parts: Firstly, given a noise distribution P ( z ) , a mapping network transforms the noise from space Z = { z | P ( z ) (cid:54) = 0 } ⊆ R 1 ×into the replicated latent space W = { [ F ( z ); . . .
; F ( z )] | z ∈ Z} ⊆ R 18 × 512 , where F is a fullyconnected network.
Secondly, a synthesis network G composed of convolutional blocks transfers
W space into image space M = { G ( w ) |
w ∈ W} ⊆
R H
× W × 3 .
Another important concept is the extended latent space W + = { [ F ( z 1 ); . . .
F ( z 18 )]
| z i ∈ Z} .
For ∀ w 1 , w 2 ∈ W , the latent code manipulation merges their information by the linear combination diag ( α 1 ) w 1 + diag ( α 2 ) w 2 ∈ W + , α 1 , α 2 ∈ R 18 .
Empirically, W + space is inferior to W in generation ﬁdelity and editability, but superior in GAN inversion that ﬁnds a code w ref to reconstruct a reference
y ref ∈ R H
× W × 3 .
In this paper we only focus on tuning G , and ﬁx F for keeping the distribution of source latent spaces.
Manifold learning Considering the sample matrix
X =
[ x T 1 ; . . .
; x Tn ] from source manifold M s ,
W = [ w i,j ] ni,j =1 is the weight matrix, where w i,j is the weight of x i and x j usually deﬁned as e −(cid:107) x i − x j (cid:107) 2 /σ .
Given a task-speciﬁc function f and y = f ( x ) , manifold learning requires the transformed samples Y =
[ y T 1 ; . . .
; y Tn ] on the new manifold M t should preserve the geometric structure of M s , which means the data nearby on M s are also nearby on M t , and vice versa.
The requirement is mostly realized by minimizing the manifold regularization term [4] (cid:82) M s (cid:107)∇ M s y (cid:107) 2 dP ( x ) , ∇ M s is the gradient along manifold, and P ( x ) is the probability measure.
If M t is a submanifold of R , the former is equalized with (cid:82) M s y ∆ M s y dP
( x ) by Stokes theorem.
∆ called the Laplace–Beltrami operator, is a key geometric object in the Riemannian manifold, thus the integral is also called Laplacian regularization.
In practice, it is estimated by the discrete form 2 tr ( Y T LY ) =
(cid:80) i,j w i,j (cid:13)(cid:13)
y i − y j
(cid:13)(cid:13) 2 ,
L = D − W is known as the graph Laplacian operator, D is the degree matrix where the diagonal element d
i,i = (cid:80) nj =1 w i,j and other elements are zero.
The convergence between L and ∆ can be found in [3].
Intuitively, the regularization encourages the local smoothness of f .
If data distribute densely in an area, the function should be smooth to avoid local oscillations.
If x i and x j are close, in the discrete form w i,j will impose a strong penalty to the distance between y i and y j .
Task deﬁnition
The generalized one-shot adaptation can be formally described as: Given a reference y ref , a mask m ref ∈ { 0 , 1 } H × W is provided to label the location of entity contained in y ref .
The target domain T is deﬁned to contain images of similar style and entities as y ref .
With the knowledge stored in a generative model G s pre-trained on source domain S , a generative model G t is learned from y ref and m ref to generate diverse images belonging to domain T .
Note that we can set m ref = 0 when there is no interested entities existed in y ref .
Moreover, the adaptation should satisfy the cross-domain correspondence.
i.e. , ∀ w ∈ W , G s ( w ) and G t ( w ) should have similar shape or content in visual.
With the correspondence, adapted models can be applied for not only synthesis, but also the transfer and manipulation of source images.
Methodology 4.1 Overall
Our framework for tackling the generalized one-shot GAN adaptation task is illustrated in Fig.
For the sake of description, in the following part we will take the StyleGAN pre-trained on FFHQ as an example to describe each component of the framework in detail.
We will demonstrate our framework can be applied to various domains in the experiments.
Network architecture
Different from previous works [51, 20] that directly adapt the source generator G s , our target generator G t (as Fig.
2 shows) consists of a generator ˆ G t inherited from
G s , and an auxiliary network ( aux ) trained from scratch.
We are inspired by the fact that G s can 𝒚 𝒓𝒆𝒄 𝒚(cid:3549) 𝒓𝒆𝒄 𝒘 ⋕ 𝒚(cid:3549)
A u x
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k ⋯ ⋯ ` ⋯ 𝑢𝑛𝑒𝑡 𝒚 𝒓𝒆𝒇 𝒙 𝒓𝒆𝒄 𝓛 𝒓𝒆𝒄 𝓛 𝑽𝑳𝒂𝒑𝑹 𝒚
𝓛 𝒔𝒕𝒚𝒍𝒆 (a) Reconstruction (c) Loss (d) Aux (b) Random synthesis
Legend
Conv Block Conv Block
Style injector
StyleGANblock
𝒘 𝒓𝒆𝒇 𝓛 𝒆𝒏𝒕 𝒙 𝑮 𝒔 (cid:4666)𝒘(cid:4667) ⋯
A u x
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k C o n v B l o c k
C o n v B l o c k
Style fixation
Flow of 𝐺 (cid:3047)
Flow of 𝐺(cid:3552) (cid:3047)
Conv Block approximate arbitrary real faces but cannot handle most entities, due to that G s captures the prevalent elements (clear natural faces) of FFHQ rather than the tails (faces with entities like hats or ornaments) of the distribution [46, 25].
The design makes aux serve to entities, and ˆ G t just focus on stylizing clear face to beneﬁt from the prior knowledge stored in G s .
As shown in Fig. 2(d), for each w , its feature map generated by the convolutional block of StyleGAN is fed into aux , where a UNet [31] predicts the feature map f ent of entity and the mask m that exactly labels the position of the entity, then they are merged with f in by Eq.
(1) to get the feature map of face covered with entity.
[Equation]
(cid:12) denotes the Hadamard product.
We hope the shape of the entity is only inﬂuenced by the content of synthesis, hence we place aux after the fourth block of StyleGAN, where the 32 × 32 resolution f in contains the most content information and the least style information of synthesis (see Sec.
Because the global style and local entity are usually independent, the disentangled design also prevents the style of faces from being contaminated by that of entities (see Sec. 4.3).
Training process Our training process can be divided into three main parts: 1 (cid:13) Style ﬁxation and exemplar reconstruction.
In Fig. 2(a), we get the latent code w ref =
[ w cref ; w sref ] of y ref by the
GAN inversion technique [36], and train G t to reconstruct y ref with reconstruction loss L rec .
In
Fig. 2(b), each w will be transformed into the style-ﬁxed code w (cid:93) =
[ w c ; w sref ] to roughly obtain the exemplary style.
2 (cid:13) Internal distribution learning.
We minimize the divergence of internal patch distributions between syntheses and exemplar for style and entity transfer.
Instead of using GAN loss that is hard to optimize, we take sliced Wasserstein distance (SWD) [5] as style loss L style and entity loss L ent to learn internal distribution efﬁciently.
3 (cid:13) Manifold Regularization.
To suppress the content distortion during training, we propose the variational Laplacian regularization L V lapR to smooth the change from G s to G t by preserving the geometric structure of the source manifold.
As shown in Fig. 2(c), the overall loss is
[Equation]
The details of training process will be illustrated and discussed in the following sections.
For convenience of description and self-consistency, some abbreviations have been taken:
y =
G t ( w (cid:93) ) , ˆ y =
ˆ G t ( w (cid:93) ) , y rec =
G t ( w ref ) , ˆ y rec =
ˆ G t ( w ref ) , x = G s ( w (cid:93) ) , x rec =
G s ( w ref ) .
4.2 Style ﬁxation and exemplar reconstruction
It is well known that StyleGAN is equipped with the disentangled latent spaces W and W + [1], where the latter vectors of code mainly determine the style of synthesis whereas the earlier vectors determine the coarse-structure or content of the synthesis.
Taking advantage of this trait, we ﬁrst roughly transfer the exemplary style to other syntheses by ﬁxing the style code.
The exemplar y ref is fed into the pre-trained GAN inversion encoder e4e [36] to get the latent code w ref =
[ w cref ; w sref ] , where w cref ∈ R l × 512 and w sref ∈ R (18 − l ) × 512 encode content and style information of exemplar respectively.
For each w ∈ W , it will be transformed to w (cid:93) before fed into networks,
[Equation]
The content part of w is reserved and style part is replaced by w sref , and we denote the new space as W (cid:93) .
Because style ﬁxation should not change the content of the original synthesis, by using the pre-trained Arcface model [10] to compare the identity similarity before and after style ﬁxation, we ﬁnd setting l = 8 is acceptable which gets 65% average similarity and obtains the adequate exemplary style in visual.
In MTG [51], style mixing assists as a post-processing step to improve the style quality, but the experiments (Fig. 3) show that the style of syntheses is underﬁtting and not consistent.
In our framework, it is used as a pre-processing step to facilitate the following learning.
Because x rec is just the projection of y ref on the source domain, there often exists an obvious distinction between them in visual as shown in Fig.
Hence we adopt the reconstruction loss L rec to narrow the gap,
[Equation]
dssim denotes the negative structural similarity metric [41] and lpips is the perceptual loss [47], m ↑ rec is the upsampled reconstructed mask from aux .
It is worth noting L rec acts on G t rather than the vanilla architecture ˆ G t .
When m ref = 0 , aux does not work and G t is equivalent to ˆ G t .
4.3 Internal distribution learning
Recent works about internal learning [32, 34, 48, 49] prove that the internal patch distribution of a single image contains rich meaning.
Minimizing the divergence of internal distributions works out many tasks like image generation, style transfer, and super-resolution.
It is usually achieved by the adversarial patch discriminator [13].
However, it is well known that the training of GAN is time-consuming, unstable, and requires large GPU memory.
Previous few-shot GAN adaptation works [22, 27] using GAN loss almost have the serious model collapse phenomenon.
We ﬁnd that slice Wasserstein distance ( SWD ) [5] can lead to the same destination but with greater efﬁciency.
For two tensor A , B ∈ R H
× W × d , the SWD of their empirical internal distributions is deﬁned as
[Equation]
where S d = { θ ∈ R d : (cid:107) θ (cid:107) = 1 } , proj : R H × W × d → R H × W that projects each pixel from the d channels to a scalar by θ , and sort denotes the sort operator ordering all values.
SWD can be easily implemented by convolution of a randomized 1 × 1 kernel along with a quick-sort algorithm.
To realize style and entity transfer, we use SWD to minimize the divergence of the empirical internal distribution of the syntheses and reference.
The style loss is deﬁned as
[Equation]
Φ style is a set of convolutional layers from the pre-trained lpips network to extract spatial features.
Note that the generator learns the style of ˆ y rec rather than that of y ref to prevent the style of the entity from leaking into the other area.
The entity loss is deﬁned as
[Equation]
(7) Φ ent is also a set of convolutional layers from the pre-trained lpips net, m ↑ is upsampled by m to get the same resolution as the ﬁnal synthesis.
Taking the reconstructed entity m ↑ rec (cid:12) y rec as the target can be viewed as an implicit augmentation technique to avoid overﬁtting.
4.4 Manifold regularization
The training losses, particularly the style loss, inevitably distort the content of source syntheses and change their relative relationship.
To achieve cross-domain correspondence, we use the variant
[Figure]
Laplacian regularization L V LapR to alleviate the distortion.
Since we do not tune the StyleGAN mapping network, the W space coupled with the source generator G s does not change, so we deﬁne
[Equation]
8) where φ is a pre-trained semantic extractor, here we use the pre-trained CLIP image encoder [29] that is sensitive to both style and content variation.
L V LapR can be interpreted from two perspectives.
The ﬁrst is the smoothness of the function: Because the integrand also represents the smoothness of the residual φ ( G t ( w (cid:93) ))
− φ ( G s ( w (cid:93) )) , minimizing L V lapR expects syntheses from G s and G t across W have a smooth semantic difference.
In the ideal case, the integrand is nearly zero, for various w in a local area, G t ( w (cid:93) ) and G s ( w (cid:93) ) have similar differences in terms of style and entity.
The second perspective is the preservation of geometric structure:
According to the Laplacian regularization theory [3], the integral form in Eq.
(8) can be estimated in discrete form;
[Equation]
9a
[Equation]
(9b)
where R =
[ φ ( y 1 ) T − φ ( x 1 ) T ; . . .
; φ ( y n ) T − φ ( x n ) T ] , w i,j = e − (cid:107) w (cid:93)i − w (cid:93)j (cid:107) /σ , L is the Laplacian matrix (see Sec. 3).
In Eq.
(9a), if w i and w j are close in latent space, w i,j will be a large scalar that encourages (cid:13)(cid:13) φ ( y i ) − φ ( y j ) (cid:13)(cid:13) and (cid:107) φ ( x i ) − φ ( x j ) (cid:107) to be the same.
This means that the relative distances of adjacent syntheses before and after adaptation are isometric in the feature space.
In practice, the loss can be efﬁciently computed using Eq.
(9b), the derivation is provided in the supplementary materials.
Relation between L V lapR and L CDC
Recently the inﬂuential work FSGA [27] introduces the cross-domain distance consistency loss L CDC to achieve cross-domain correspondence.
For arbitrary w i and w j , without loss of generality, FSGA deﬁnes the conditional probability p
j | i and q j | i in Eq.
(10) to form the similarity distributions, then minimizes their Kullback-Leibler divergence Eq.
(11) that encourages the syntheses of G t to have the same similarity property as that of G s . [Equation]
[Equation]
In the above equation, φ denotes the feature map of the corresponding generator.
From the perspective of manifold learning, p ·| i and q ·| i depict the neighborhood structure of x i and y i respectively, L CDC tries to place x i in a new space so as to optimally preserve neighborhood identity.
Actually, we ﬁnd that this idea is equivalent to the Stochastic Neighbor Embedding [12, 37], which is also a popular
[Table]
method for preserving the manifold structure.
L CDC has the following disadvantages.
Firstly, a large batch is almost inacceptable for a high-resolution GAN to estimate p ·| i and q ·| i .
L V LapR directly constrains the distance and performs well even when the batch is 2 .
Second, L CDC is non-convex and hard to optimize, in practice, the loss value is usually around 1e-5, implying that L CDC does not provide an effective penalty for the differences.
Finally, because the softmax form of p j | i and q j | i is scale-invariant about the inputs, L CDC cannot guarantee the isometric relationship of the source and adapted syntheses like L V LapR , thus it is hard to avoid mode collapse.
In the supplementary materials, we show that replacing L CDC with L V LapR can greatly remedy the collapse of FSGA.
Experiments
Our implementation is based on the ofﬁcial code of StyleGAN 1 .
If m ref (cid:54) = 0 , the total epoch is 2000 .
We adopt the Adam optimizer with learning rate 1 e − 3 , β 1 = 0 , β 2 = 0 .
The cosine annealing strategy of the learning rate is adopted to reduce the learning rate to 1 e − 4 gradually.
In
Eq.
(2), λ 1 = 10 , λ 2 = 0 .
2 , λ 3 = 2 , λ 4 = 1 , and in Eq.
(4) λ 5 = 100 .
We use the pre-trained lpips network with vgg architecture [35] containing ﬁve convolutional blocks.
In Eq.
(6) and Eq.
(7), Φ style = { vgg 1:4 , vgg 1:5 } , Φ ent = { vgg 1:1 , vgg 1:2 , vgg 1:3 , vgg 1:4 } , and m = 1 .
In Eq.
(9b), n =including the samples from reconstruction and internal distribution learning.
σ for calculating the graph weights is 128 .
We use the Monte Carlo simulation that randomly samples 256 vectors on the unit sphere to compute the integral in Eq.
If m ref = 0 , we change λ 2 to 2 , λ 4 to 0 .
5 , and epoch to 1000 for speeding up training.
The augmentation of the reference only includes the horizontal ﬂip.
More experimental details and results can be found in the supplementary materials.
5.1 Comparison with SOTA methods
We take the few-shot adaptation method FSGA [27], one-shot adaptation methods MTG [51], OSCLIP [20], and one-shot face style transfer method JoJoGAN [9] for comparison on the commonly studied face domain.
As shown in Fig.
3, since these works cannot deal with the entity yet, for a fair comparison we select three portraits without entity (Sketch, Disney, Arcane) from previous works [ ? 9 ] and three portraits with entities (hat, Zelda decorations, and mask) from AAHQ [23].
Their masks can be obtained via manual annotation, or alternatively by semantic segmentation [ ? ? ].
Qualitative results Fig.
3 shows the qualitative results.
The top row images are the source natural faces, and they are inverted by e4e [36] to obtain the latent codes.
From the results, we can draw the following conclusions: Firstly, our syntheses are of competitive style as the given references.
Though shown in thumbnails, they have more obvious high-frequency details to look sharp and realistic, whereas the results of JoJoGAN are very fuzzy.
Secondly, our method achieves the strongest cross-domain correspondence that preserves the content and shape of the source image, which is a great advance compared with other methods without a doubt.
It also implies our method can keep the diversity of the source model without content collapse.
Thirdly, our method can generate high-quality entities, which are adaptive to the various face shapes and look harmonious with the rest area of the syntheses.
Experiments prove that our method has sufﬁcient visual advantages over competitors.
Quantitative results
Previous works mainly conduct user studies to compare the visual quality.
Following them, we survey user preferences for different methods.
We ﬁrst randomly sample 50 latent codes in W space and feed them into various models.
We provide the references, source syntheses,
[Figure]
and adapted syntheses to volunteers.
They consider both style and content and subjectively vote for their favorite adapted syntheses.
Moreover, to objectively evaluate the cross-domain correspondence, we use the face alignment network [7] to extract facial landmarks and calculate the Normalized Mean Error (NME) of landmarks between source syntheses and adapted syntheses.
[Equation]
where lmk denotes the 68 points two-dimension landmarks extractor, and n is set to 1000.
NME reﬂects the face shape difference before and after adaptation.
We further report the Identity Similarity (IS) predicted by the Arcface [10] to metric the preservation of identity information.
The model with lower NME and higher IS shows better cross-domain corresponding property.
The quantitative results have been shown in Table 1.
Our method undoubtedly obtains the best scores in terms of
NME and IS.
It performs stably and gets similar NME in different domains.
FSGA and OSCLIP obtain relatively worse scores due to overﬁtting as shown in Fig.
3, their common characteristic is that the GAN discriminator is used for training.
MTG performs very unsteadily especially when the target domains have strong semantic style like Disney and Zelda.
Compared with our approach,
JoJoGAN works well on NME, but poorly on IS.
For the user study, we ﬁnally collect valid votes from 53 volunteers, and a user averagely spends 4 seconds on one question.
Our method obtains about 60 % of the votes on each adapted model, whereas other methods evenly acquire the rest of the votes.
It proves that our method is much more popular with users.
Training and inference time Our method costs about 12 minutes for m ref (cid:54) = 0 and 3 minutes for
m ref = 0 on NVIDIA RTX 3090.
FSGA, MTG, JoJoGAN, OSCLIP cost about 48, 9, 2, 24 minutes respectively.
All of them take about 30ms to generate an image.
5.2 Ablation study
We start from the basic reconstruction loss L rec and continuously improve the framework to observe the qualitative changes of the syntheses.
As illustrated in Fig.
4, the target exemplar is a hard case depicting a man with a beard and cap, and style ﬁxation can bring a rough but remarkable improvement to the style of synthesis.
Although L style can enhance the style of the face, it deteriorates the hair and introduces noise like the beard to the female face.
L ent plays a critical role in the generation of the entity. Without it, the cap cannot be synthesized.
In particular, L V LapR has a signiﬁcant suppression of the noise in previous syntheses.
Finally, we remove L style and style ﬁxation respectively, then the syntheses are no longer of the target style.
It means that both of them play vital roles in style transfer.
We also provide the quantitative ablation in Table 1 to study the impacts of L style and L V LapR .
It shows that removing L style has a positive effect on NME and IS while removing L V LapR has an opposite effect.
The quantitative results conﬁrm that L V LapR is necessary for preventing content from being distorted by style loss.
5.3 More results
Adaptation of various domains In addition to the success of the face domain, our framework can be applied to other source domains.
We take StyleGANs pre-trained on the 512 × 512 AFHQ dog, cat data set [8] and the 256 × 256 LSUN church dataset [45] as exemplars.
As shown in Fig.
5, for each source domain, we provide images with entities, and without entities, the source images are
T a r g e t
[Figure]
T a r g e t
Source
T a r g e t
Source
[Figure]
randomly sampled from the source space.
For images without entities, the adapted syntheses are realistic and have a strong corresponding relationship in visual.
In other cases, our approach also yields satisfactory results.
For example, the cartoon eyes of the dogs are in the right place and very similar to that of the reference.
It proves that our framework is valid and has a wide universality.
Image manipulation
Because our adapted model retains the geometric structure of the source generative space, it can efﬁciently perform semantic editing on the image using the semantic direction found in the source space.
As shown in Fig.
6, since our model is capable of accurately reconstructing the target exemplar, we can edit both the exemplar and the synthesis.
We select three representative semantic directions [33] to change smile, age, and pose rotation, and the results show that the adapted generator achieves exact control not only on the face but also on the entity.
In addition, our model can also change the style by altering the style code, which is hard for previous works.
Furthermore, our framework can be applied to remove the entity with ˆ G t , and the result can be manipulated similarly.
We consider these functions will be helpful for artistic creation.
Conclusion and limitations
In this paper, we construct a new GAN adaptation framework for the generalized one-shot GAN adaptation task.
Because our framework relies heavily on learning of internal distributions, it inevitably has some limitations.
We think the most important one is that it cannot precisely control the position of the entity, which may lead to failures when the pose changes too much.
Another limitation is that the entity cannot be too complex, otherwise, it is difﬁcult to learn by patch distribution.
However, these extreme cases require additional consideration.
In general, our framework is effective in various scenarios.
We believe that the exploration of the novel task and the introduction of manifold regularization are signiﬁcant for future work.
Acknowledgements
This paper is supported by the National key research and development program of China (2021YFA1000403), the National Natural Science Foundation of China (Nos. U19B2040), the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDA27000000, and the Fundamental Research Funds for the Central Universities. Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] The images we use are widely used in previous works or from open dataset.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] The images we use are widely used in previous works or from open dataset.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[No] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]