One Model to Edit Them All: Free-Form Text-Driven Image Manipulation with Semantic Modulations
[Figure]
Abstract
Free-form text prompts allow users to describe their intentions during image manipulation conveniently.
Based on the visual latent space of StyleGAN [19] and textembedding space of CLIP [28], studies focus on how to map these two latent spacesfor text-driven attribute manipulations.
Currently, the latent mapping between thesetwo spaces is empirically designed and confines that each manipulation modelcan only tackle one fixed text prompt.
In this paper, we propose a method named Free-Form CLIP (FFCLIP), aiming to establish an automatic latent mapping sothat one manipulation model handles free-form text prompts.
Our FFCLIP has across-modality semantic modulation module containing semantic alignment andinjection.
The semantic alignment performs the automatic latent mapping vialinear transformations with a cross attention mechanism.
After alignment, weinject semantics from text prompt embeddings to the StyleGAN latent space.
Forone type of image (e.g., ‘human portrait’), one FFCLIP model can be learnedwith free-form text prompts.
Meanwhile, we observe that although each trainingtext prompt only contains a single semantic meaning, FFCLIP can leverage textprompts with multiple semantic meanings for image manipulation.
In the experiments, we evaluate FFCLIP on three types of images (i.e., ‘human portraits’, ‘cars’, Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Do not distribute. and ‘churches’ Both visual and numerical results show that FFCLIP effectively produces semantically accurate and visually realistic images.
Introduction
Neural image synthesis has received tremendous investigations since the Generative Adversarial Networks (GANs) [12].
The synthesized image quality is significantly improved via the StyleGANbased approaches [20, 19, 18].
Recently, free-form text prompts describing user intentions have beenutilized to edit StyleGAN latent space for image attribute manipulations [27, 46].
With a single word(e.g., ‘Blue’) or phrase (e.g., ‘Man aged 10’) as an input, these methods edit the described imageattribute accordingly by modulating the latent code in StyleGAN latent space.
The accurate image attribute manipulation relies on the precise latent mapping between the visuallatent space of StyleGAN and the text embedding space of CLIP.
An example is when the text promptis ‘Surprise,’ we first identify its related semantic representations (i.e., ‘Expression’) in the latentvisual subspace.
Then, we modulate the latent code of this identified latent subspace via the textembedding guidance.
Pioneering studies like TediGAN [46] and StyleCLIP [27] empirically identifywhich latent visual subspace corresponds to the target text prompt embedding (i.e., attribute-specificselection in TediGAN and group assignment in StyleCLIP).
This empirical identification confinesthat given one text prompt, they must train a corresponding manipulation model. Different textprompts require different manipulation models to modulate the latent code in latent visual subspaceof StyleGAN. Although the global direction method in StyleCLIP does not employ such a process,the parameter adjustment and edit direction are manually predefined.
To this end, we are motivatedto explore how to map text prompt embeddings to latent visual subspace automatically.
So a singlemanipulation model is able to tackle different semantic text prompts.
In this paper, we propose a free-form method (FFCLIP) that manipulates one image according todifferent semantic text prompts.
FFCLIP consists of several semantic modulation blocks that takethe latent code w in StyleGAN latent space W + [1] and the text embedding as inputs.
Each blockhas one semantic alignment module and one semantic injection module.
The semantic alignmentmodule regards the text embedding as the query, the latent code w as the key, and the value.
Thenwe compute the cross attention separately in both position and channel dimensions to formulatetwo attention maps.
We use linear transformations to perform a latent mapping from text promptembedding and latent visual subspace, where the linear transformation parameters (i.e., translationand scaling parameters) are computed based on these two attention maps.
Through this alignment,we identify each text prompt embedding to its corresponding StyleGAN latent subspace.
Finally, thesemantic injection module modifies the latent code in subspace via another linear transformationfollowing HairCLIP [44].
The modulated semantics are represented as the latent code offset ∆ w ,which is refined progressively through several semantic modulation blocks.
From the perspective of FFCLIP, the empirical group selection of w in [27, 44] is a particular formof our linear transformations in the semantic alignment module.
Their group selection operationsresemble a binary value of our scale parameters to indicate the usage of each position dimensionof w .
On the other hand, we observe that the W + is not disentangled completely, the empiricaldesign could not find the precise mapping between StyleGAN’s latent space and CLIP’s text semanticspace.
In contrast, the scale parameters in our semantic alignment module adaptively modify thelatent code w to map different text prompt embeddings.
The alignment is further improved via ourtranslation parameters.
We evaluate our method on the benchmark datasets and compare FFCLIP tostate-of-the-art methods.
The results indicate that FFCLIP is superior in generating visually pleasantcontent while conveying user intentions.
Related Works
Latent Space Image Manipulation.
The latent space in StyleGAN [20, 19, 18, 24] has demonstratedgreat potential in representing the semantics in the image, motivating many works to disentangle the latent space for controllable image manipulation.
Specifically, investigations [34, 16, 3, 39, 4] predictthe meaningful offsets or directions in the latent space given image annotations as supervision, whilestudies [35, 41, 14, 42] disentangle the latent space in an unsupervised manner to find the semanticdirections.
Although these methods achieve great performance in latent space manipulation, they canonly find limited semantic directions that confine user intentions.
In contrast, we achieve free-formimage manipulation conditioned on arbitrary text prompts, giving users more degrees of editingfreedom.
Our StyleGAN inversion encoder is related to GAN inversion methods [1, 2, 5, 6, 32, 40, 43]that maps the image to the latent code space w ∈ W + [1].
In our work, we use the e4e [40] as
StyleGAN inversion encoder following StyleCLIP [27].
Text-driven Image Generations.
Starting from [31] that leverages text embeddings as the conditionfor GAN-based image training, several works [49, 50, 51] improve the synthesis quality by introducingmulti-scale or hierarchically-nested GANs.
And attention modules are introduced in [47] to matchthe generated images to text.
[8, 26, 25, 38] improves image content fidelity is considered from thenetwork structure and training perspectives.
The text to image generation performance is furtherboosted in DALLE [30, 29] and DiffusionCLIP [21] where text prompts with multiple semanticguidance are translated into images.
Vision transformer [10] and diffusion model [13] also showimpressive results in text-driven image manipulations.
TediGAN [46] transfers the image and textto a shared StyleGAN latent space, and modulates the image latent vector with text.
Recently, CLIP [28] constructs a text-image embedding space to connect the semantics between image and textembeddings.
Combining the CLIP model with StyleGAN becomes promising for text-driven imagemanipulations.
Pioneering works such as StyleCLIP [27] use the CLIP model to map semanticsfrom text to images.
HairCLIP [44] builds on top of StyleCLIP while focusing more on the hairregions with reference images and semantic injections.
FEAT [15] proposes an attention mask toprevent changes to unedited areas .
These methods need to manually select a specific StyleGAN latentsubspace according to the target text.
So they fail to train a single model for different text prompts.
Incomparison, FFCLIP can adaptively match one text embedding to the desired latent subspace, makingone model process multiple text prompts for the corresponding image manipulations.
Proposed Method
[Figure]
Fig.
2 shows an overview of the proposed method.
Given an input image and a text prompt, we canobtain the StyleGAN latent code w and text embedding e t .
We propose k
(we set k = 4 in practice)semantic modulation blocks where each block first aligns semantics between w and e t and then editsthe latent code.
The output of each block is an offset ∆ w which adds to the latent code w .
Finally, a StyleGAN generator decodes the resulting latent code to the manipulated image.
In the following, wefirst briefly review the StyleGAN and the CLIP model, then we illustrate how we align and injectsemantics for latent space editing.
StyleGAN Revisiting.
Studies for StyleGAN [20, 19, 18] generate visually realistic face imageswith various semantics.
In StyleGAN, a latent variable Z ∈ R 512 from the Gaussian distributionis gradually transformed to a disentangled latent code W ∈ R 512 with semantic meanings.
Thelatent code W is then fed into 18 layers for image generation.
Later, investigations [40] find that anextension of W (i.e., W + ∈ R 18 × 512 ) is more effective for identity preservation when manipulatingthe image .
There is a definite relation between the layers of W + and the semantic latent subspace.
So the semantic modulation can be formulated as editing some specific layers of W + .
As a result,image manipulation methods [3, 39, 4, 16] based on StyleGAN invert an input image to a latent codew ∈ W + for semantic modification. In this work, we follow [27] to utilize the StyleGAN2 as ourgenerator.
CLIP Revisiting.
The CLIP model [28] is pretrained from 0.4 billion image-text pairs for crossmodality semantic matching.
This model consists of an image encoder and a text encoder to projectan image and a text prompt into a 512-dimension embedding, respectively.
We leverage the textencoder of CLIP model to produce the text prompt embedding.
This encoder is versatile in perceivingtext prompts with different semantic meanings.
[Figure]
3.1 Semantic Modulation
Fig.
3 shows the framework of a semantic modulation block where there are k blocks in total.
Givenan image latent code w ∈ W + converted from an image with e4e [40] and a text embedding e textracted from the CLIP model, k semantic modulation blocks progressively refine ∆ w .
The ∆ w willmodify w for latent subspace editing.
The output of the i -th block is denoted as ∆ w i .
In each block,there is a semantic alignment module followed by a semantic injection module, which is illustrated asfollows:
Semantic Alignment Module.
The automatic semantic correspondence between e t and w enablesthat one model is able to process text prompts with different semantic meanings.
We establishthis automatic correspondence by aligning w and e t via a linear transformation.
The parametersof this linear transformation are learned based on the cross attention measurement.
Specifically,for the semantic alignment module in the i -th block, we compute a cross attention map betweene t and ∆ w i − 1 ( ∆ w 0 = w ) in both position and channel dimensions similar to the dual attentionnetwork [11].
The scale and translation parameters of the linear transformation are computed basedon the cross attention maps.
For the position dimension, we set the e t as the Query Q p ∈ R 512 and the latent code ∆ w i − 1 as the
Value V ∈ R 18 × 512 and the Key K ∈ R 18 × 512 .
Then, we compute the cross attention map Attention pas a scale parameter S ∈ R 18 .
The scale parameter S models the Value V in position dimension fortransforming the latent space to match the text embedding semantics.
The cross attention computationon the position dimension can be written as:
[Equation]
where W Q , W K , W V ∈ R 512 × 512 .
We use these scale parameters to adjust the contribution ofeach dimension of W + conditioned on e t .
Meanwhile, we observe that the W + space is not fullydisentangled.
One semantic property is reflected not only in a specific element of w , but in otherelements as well.
To this end, we compute the translation parameters in the channel dimension toenhance the semantic alignment.
For the channel dimension, we use the same K and V as those in the position dimension, and a new Query Q c ∈ R 18 × 512 to calculate the cross attention map Attention c in the channel dimension. Then,we reconstruct the Value V with the attention map Attention c and utilize an adaptive average pooling 𝛾 𝛽
Semantic Injection ⊕ 18 ×18 ×512 ×512 ×
Reshape
M a t M u l
S o f t M a x
M a t M u l
A d a p ti v e
A vg P oo li ng 1 ×
T e t e t 18 ×1 ×⊕
K
V V
K
M a t M u l
S o f t M a x 1 ×1 ×18 ×18 ×18 ×∆ w iL a y e r
N o r m e t S 1 ×∆ w i 18 ×
Semantic Alignment
Q p
Q c to compute the translation parameters T ∈ R 512 .
These processes can be written as
[Equation]
where the W K and W V are the same as the those in the position dimension, and the W Qc ∈ R 1 × 18 .
AAP is the adaptive average pooling operation.
After we get the scale and translation parameters, wecan align ∆ w i − 1 to e t with a linear transformation as follows:
[Equation]
where the x i is the output of this semantic alignment module.
Then, we inject e t to this aligned latentsubspace via our semantic injection module in the following.
Semantic Injection Module.
In this module, we inject the semantic from e t to x i following that in
HairCLIP [44].
Specifically, we adopt the fully-connected layers to map the text embedding e t intotwo injection parameters β ∈ R 512 and γ ∈ R 512 , respectively.
Then, we inject the text embeddingsemantics as follows:
[Equation]
where µ x i and σ x i represent the mean and the variance of the input x i , respectively.
With thealigned semantics, our injection produces the ∆ w i to modify StyleGAN latent code.
The ∆ w i isfurther refined through the following semantic modulation blocks to obtain ∆ w for the final imagegeneration.
3.2 Training Objectives
In text-driven image manipulation, we focus on two aspects of the output result.
First, the semanticsof the target object in the image shall be consistent after manipulations.
Second, the output imageshall be semantically relevant to the text prompt.
To this end, we follow [44] and develop two typesof loss functions, i.e., semantic preserving loss and text manipulation loss.
Semantic Preserving Loss.
We aim to preserve the semantic consistency between the input andoutput images.
This semantic shall be represented within the CLIP model.
Specifically, we feed the input and output images through the CLIP image encoder to get two image embeddings.
If the imagesubject is consistent, the distance between these two embeddings should be small.
So we reduce thedistance between these two embeddings.
We call it as image embedding loss and define it as follows:
[Equation]
where the G ( · ) denotes the pretrained StyleGAN generator, cos {·} means cosine similarity, E CLIPI ( · )is the pretrained CLIP image encoder, w and w ′ = w + ∆ w are the input and our edited StyleGANlatent codes, respectively.
Then we utilize the L 1 norm for preserving the irrelevant semantic regions:
[Equation]
Moreover, for ‘human portrait’ images, the background loss and the face identity loss are used toimprove the performance, and the face identity loss is defined as follows:
[Equation]
where R ( · ) indicates the pretrained ArcFace network [9].
Also, the background loss can be formulatedas:
[Equation]
8) where P ( · ) is the facial parsing network [23] and P ( G ( w ′ ) , P ( G ( w )) represents the non-facialregions in the input and output images, respectively.
The overall semantic preserving loss L sp can bewritten as
[Equation]
(9) where λ embd , λ norm , λ id , and λ bg are the weights that adjust the contribution of each loss term.
Weset λ embd = 1 . 0 , λ norm = 1 . 5 , λ id = 1 . 0 , and λ bg = 2 . 0 .
In particular, we only introduce L id and
L bg when we edit face images.
Text Manipulation Loss.
To evaluate the correlation between output images and text promptembeddings, we minimize their cosine distance by using the CLIP model.
It can be written as:
[Equation]
where E CLIPI ( · ) is the pretrained CLIP image encoder.
Overall, the total loss is
[Equation]
11) where λ sp and λ t are set as 1 and 1.5 in our method, respectively.
Experiments
We first illustrate our implementation details.
Then we compare FFCLIP with existing methodsqualitatively and quantitatively.
An ablation study validates the effectiveness of our modules.
Moreresults and a video demo are provided in the appendix and supplementary files, respectively.
We willrelease our implementations to the public.
4.1 Implementation Details
We train and evaluate FFCLIP on the CelebA-HQ dataset [17], the LSUN cars dataset [48], and the
LSUN Church dataset [48].
To invert images into StyleGAN2’s latent codes, we use a pretrainede4e [40] as the image encoder.
The dimensions of the latent code are 18 × 512 , 16 × 512 and 14 × 512for face, car and church images, respectively.
When training our model, we leverage the text encoderfrom CLIP, and use the pre-trained StyleGAN2 to generate edited images.
For each dataset, we usethe corresponding pre-trained StyleGAN2 as the generator.
We use 44 text prompts for face imagescontaining emotion, hair color, hairstyle, age, gender, make-up, etc.
Meanwhile, we follow the textprompts from StyleCLIP [27] when editing LSUN cars and church datasets.
We randomly choose atext prompt for an input image for model training.
In practice, we use a multi-step learning rate withan initial learning rate of 0.0005.
The Adam [22] optimizer is utilized with β 1 and β 2 set to 0.9 and0.999, respectively.
For CelebA-HQ dataset, the total training iterations are 150,000 and the batchsize is 8.
For LSUN church dataset, the total training iterations are 200,000 and the batch size is 4.
For LSUN cars dataset, the total training iterations are 100,000 and the batch size is 8.
We train ourmodel on a workstation with 8 Nvidia Telsa V100 GPUs. Input FFCLIP
B a l d
T
a y l o r S w i f t
StyleCLIP
TediGAN
A ng r y A g e d R e d h a
i r
B ea r d B l ond
M oh a w k
H a i r
Input
FFCLIP
StyleCLIP
TediGAN
4.2 Qualitative Evaluation
We compare our method to state-of-the-art text-guided image manipulation methods TediGAN [46]and StyleCLIP [27].
Then, we show our results with diversified text prompts.
We also show ourmodel is able to manipulate images with a single text prompt containing multiple semantic meanings.
State-of-the-art Comparisons.
Fig. 4 shows the visual comparison results.
We observe that themanipulated content does not exactly match the text prompt in the TediGAN’s results.
An example isshown on the first row with the ‘Angry’ prompt.
The beard appears on the man in the TediGAN’sresult while it does not exists in the input image.
While StyleCLIP improves the manipulation resultof TediGAN, it still limits editing input images with high semantic accuracy.
The hair still presents inthe StyleCLIP result with the ‘Bald’ prompt.
In comparison, our FFLIP is effective in editing imagesbased on the text prompt semantics while maintaining visual realism.
Compared to existing methodsof training each model for one text prompt, our model processes multiple text prompts with onlyone model.
Our superior performance results from the accurate semantic alignment across the textembedding and StyleGAN latent space.
Meanwhile, the results of FFCLIP by using text prompts withmultiple semantics are shown in Section A.1 where our accurate latent mapping empowers FFCLIPto process multiple semantic meanings.
Moreover, the free-form image manipulation is shown in
Section A.2 where FFCLIP performs well in multiple datasets. Finally, more visual comparisonresults are shown in Section A.8. [Figure]
Interpolation Results. FFCLIP can achieve fine-grained image manipulation by interpolating withtwo output latent codes.
As shown in Fig.
5, we generate the intermediate latent code by linearweighting w ′ c = w ′ a + λ ( w ′ b − w ′ a ) , where w ′ c is the intermediate output, w ′ a and w ′ b are the outputsof FFCLIP which correspond to two different manipulation semantics. By gradually increasing the
[Table] blending parameter λ from 0 to 1, we can generate the results between two semantics (e [Table] g., the visualresults between ‘Aged 10’ and ‘Aged 80’).
More interpolation results are shown in Fig.
14 and 15.
4.3 Quantitative Evaluation
There is hardly a straight quantitative measurement to evaluate the image manipulation results.
Nevertheless, in the text-driven image manipulation scenario, we believe the image results shouldcorrespond to the text prompt semantics.
We utilize several text prompts and randomly select [-
 [Table] testing-]1000testing images from the CelebA-HQ dataset.
For each prompt, we produce the results from TediGAN,
StyleCLIP and ours.
Then, we follow [36] to use the multiple semantic classification models tomeasure the text-relevance of these results.
Table 1 shows these evaluation results under the editing performance column.
We use differentconfigurations to compare these results.
When the text prompt is ‘Bald’, we use the PSPNet [52] tolocate the hair region and count the number of pixels within this region.
When the text prompt is ‘Redhair’, we compute the average color difference of the hair region of each result.
When the text promptis ‘Angry’, we use ESR [37] to determine whether the expression of the results belongs to the angrycategory.
When the text prompt is ‘Aged 10’, we use an age classification method [33] to computethe distance between the estimated ages from the results and 10.
For each prompt, we compute thequantitative numbers of all the results.
As shown in this table, our method is more effective to transfertext prompt semantics to the images.
Human Subjective Evaluation.
Besides designing different metrics to evaluate semantic transfer,we conduct human subjective evaluations on the manipulated results from compared methods.
Werandomly collected 54 images which were manipulated based on 6 text prompts.
35 participantswith diverse backgrounds are asked to vote for the best results based on the three equally importantprinciples.
First, they should select the result where the semantic meaning corresponds to the textprompt most.
Second, they should select the result where the human identity is best preserved.
Third,they should select the most visually realistic image.
We tally the votes as shown in Table 1 under thehuman subject evaluation column.
It indicates that most participants favor our results compared toothers, validating the effectiveness of our semantic modulation blocks.
Moreover, we conduct otherhuman subjective evaluations with more text prompts in Section A.4.
4.4 Ablation Analysis
Effect of Semantic Alignment Module.
The proposed semantic alignment module modifies StyleGAN’s latent code w from the position and channel dimensions by the scale and translation parameters.
{+In order to verify the necessity of modulation with scale and translation, we trained three models:without scale parameter, without translation parameter, and without both parameters, respectively. It isworth mentioning that all three models contain a semantic injection module to inject text informationinto the latent space. As shown in Figure 6, without scale or translation parameter, we cannot findthe latent subspace for the text accurately (e.g., in row1 col4, ‘bald’ latent subspace is not found),and the latent subspace we find is not disentangled (e.g., in row1 col3, the face ID changes; in row
 Input+}
Ours w/o S w/o S&T w/o T
B a l d
B o w l c u t
Input
Ours w/o S w/o S&T w/o T
[Figure]
B l ond
M
oh
[Figure]
h a i
[Figure]
[Figure]
col8, the hair color changes; and the result in row1 col9 wears glasses, etc.).
Furthermore, only acomplete model with both scale and translation parameters can be competent given a combinationof texts unseen during training (e.g., beard blond mohawk hair in row2).
The associated numericalresults are shown in Table 2.
Effect of the Number of Semantic Modulation Block.
To ablate the number of modulation block( k in Fig. 2), we train models with k = 0 , 2 , 4 , 6 , 10 respectively and evaluate their performance.
Thevisual results are shown in Fig. When we set the number of modulation blocks as 0, we use themappers in StyleCLIP to train our model, and we find that the visual results cannot reflect the textsemantics. A few modulation blocks will result in poor editing (see k = 2 ).
Increasing the number ofmodulation blocks can improve the performance ( k = 6 ), but larger k will cause unstable training andlead to a higher computational cost (see k = 10 ). We finally select k = 4 in our method to balancethe performance and efficacy.
The numerical analysis results are shown in Table 2.  Concluding Remarks
We propose to manipulate image content according to text prompts with different semantic meanings. Our motivation is that latent mapping of existing approaches is empirically designed between textprompt embedding and the visual latent space, so one editing model tackles only one text prompt. We improve latent mapping by semantic modulations with alignment and injection. It benefits oneediting model to tackle multiple text prompts.
Experiments in multiple datasets show that our FFCLIPeffectively produces semantically relevant and visually realistic results.
A limitation of our method is that we do not completely disentangle the StyleGAN latent code W + ,which still remains an open problem for all the existing methods.
The CLIP is known to encodehumanlike biases [45, 7].
As our text embedding is from CLIP, the manipulated results may alsosuffer from humanlike biases.
Such biases may propagate and have negative effects on minorityrepresentations (i.e., skin color changes drastically after editing).
Moreover, FFCLIP generalizeswhen the training and testing text prompts share similar image attribute descriptions.
It cannot changehair color if no hair-related prompts are utilized during training.
The generalization of other encodersshown in Fig. 21 remains for further investigations. The negative societal impact is that the editedhuman portraits could be used for malicious purposes.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and
scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] The code is proprietary now, and we will release our source code.
The data and instructions are shown in 4.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[No]
A AppendixA.1 Text Prompts with Multiple Semantics.
We observe that although we train our model with single word text prompt, FFCLIP is able toedit images with text prompts containing multiple semantics. Figure 8 and Fig.
9 show the results.
FFCLIP simultaneously transfers multiple text prompt semantics to the input image with realisticappearance. This success is because of our accurate semantic alignment for the text embedding fromthe CLIP model.
W o m a n a g e d M a n a g e d
B a l d M a k e - up
B r o w n a f oh a
i r
B l u e F r o m
[-Increasing the number of modulation blocks can improve the performance( k = 6 ), but larger k will cause unstable training and lead to a higher computational cost (see k = 10 ). We finally select k = 4 in our method to balance the performance and efficacy. The related numericalresults are shown in Table 
 [Table]-]S i x ti e s
B l ac k J ee p G o t h
i c
c hu r c h w it h c l oud s i n t h e s ky
B y za n ti n e c hu
r c h s no w f i e
l d s no w y Input FFCLIP Input
FFCLIP
Input
FFCLIP
Input
FFCLIP
[Figure]
Blue Jeep From Sixties
Gothic church with clouds in the sky Baroque and Byzantine church Input FFCLIP
Input FFCLIP Input FFCLIP
Input
FFCLIP
Input
FFCLIP
Input
FFCLIP
Input
FFCLIP
Input
FFCLIP
Input
FFCLIP
Input FFCLIP Input FFCLIP
Fig. 10, Fig. 11, Fig. 12 and Fig.
13 show our image manipulation results where FFCLIP processtext prompts with different semantic meaning.
For human portrait images, FFCLIP preserves theface identity, produces high-quality edited images, and manipulates images with different semantics. Meanwhile, our results on cars and churches datasets are realistic with the accurate semantictransfer. The various manipulation results in multi datasets prove the robustness of our method. Wepioneeringly develop a single model to process different text prompts for one type of images.
A.3 Comparison with HairCLIP
As shown in Fig. 16 and Fig. 17, we compare with HairCLIP in hair manipulation.
The HairCLIPcannot reflect the manipulation semantics with hair color and style well(e.g.
, see ‘Red hair’ and ‘Bobcut hairstyle’).
In contrast, our FFCLIP can modify the content more correctly with the capability offinding the latent subspace adaptively.
The related numerical comparison as shown in Table 3, for thenumerical text prompt ‘Gray hair’ and ‘Blond hair,’ we use the similar measurement method of ‘Red hair [Table] [Figure]
[Table]
A.4 Human Subject Evaluation for More Text Prompts
We conduct another Human Subject Evaluation with four more text prompts in Table 4. Except forthe text prompts, the other settings in this Human Subject Evaluation are the same as Table 1. We canfind that our FFCLIP outperforms other baselines.
A.5 Discussion for StyleCLIP, HairCLIP, and TediGAN
As shown in Fig.
18, StyleCLIP and HairCLIP need experiential behaviors to match the text semanticand latent space.
While the latent space W in StyleGAN is not disentangled completely, theexperiential behaviors cannot find the correct latent subspace for the target text prompt, so
the
StyleCLIP needs to train a specific mapper to find the desired latent subspace for specific text.
The
HairCLIP can manipulate the image with different texts, but it just edits the hair regions.
In contrast,
FFCLIP proposes a semantic alignment module to align the text semantic and latent space, and wecan find the target latent space with text adaptively after alignment.
For TediGAN, it makes the text
[Figure]
has the capability of generating photo-realistic and text-relevant results.
feature close to the latent code and uses the style-mixing operation in StyleGAN to manipulate theimage content.
Meanwhile, the style-mixing operation needs to replace the specific layer in latentcode with text feature, the text semantic cannot align latent space well with this experiential behavior,so it cannot manipulate the image with some interesting text prompts (i.e., ’Taylor Swift’ or ’Beard
Blond.’).
Moreover, TediGAN just manipulates the human portrait because the coarse correspondencebetween specific semantic and specific layers in latent code is hard to reproduce for other datasets.
[Figure]
A.6 Unseen text prompts and different inversion encoder
[Figure]
We show more visual results for unseen text prompts in Fig.
19 and Fig.
The FFCLIP performswell in these text prompts.
Meanwhile, as shown in Fig.
21, although the FFCLIP is trained with e4eencoder, it can edit image correct with High-fidelity [43] and Restyle [5] encoders, respectively. Thesephenomenons prove the robustness of our model and the effectiveness of the semantic alignmentmodule. [Figure] A.7 Large CLIP Model
[Figure] We show the visual comparison results between different CLIP models in Fig. We find that thelarge CLIP model ViT-L/14 has no noticeable impact on performance. [Figure]
[Figure] We show more visual comparison results between StyleCLIP, TediGAN and FFCLIP in Fig. 23 and
Fig. And we compare our model with StyleCLIP in hair color manipulation in Fig. Ourmethod can preserve the identity of images and ensure well manipulation performance at the sametime. [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
HairCLIP
[Figure]
Ours
HairCLIP Input
B ob c u t
h
a
i r s t y l e
A fr o h
a
i r s t y l e
[Figure]
StyleCLIP
FFCLIP
Image
Inversion ∆w
Text Selection w Grouping Image
Inversion ∆w
Selection w Grouping HairCLIP Text
Image Inversion ∆w w Injection Injection coarse medium fine mapper
Alignment [Figure] y [Figure]
s
G
[Figure]
ng e
[Figure]
h
[Figure]
i r
Input
FFCLIP
Input
FFCLIP
Input
FFCLIP
G i ng e r h a i r
Input
FFCLIP
Input
FFCLIP
Input
FFCLIP
[Figure]
[Figure]
g e
B l ond h a i r r e d b ea r d
Input
[Figure]
FFCLIP
Input
[Figure]
e d h a i r
R e d h a
i r
Input
FFCLIP Input
[Figure]
Input FFCLIP
B l ond h a i r r e d b ea r d
R a g e
Input
FFCLIP
[Figure]
FFCLIP
Input
[Figure]
CLIP ViT-B/
CLIP ViT-L/
[Figure]
Input
FFCLIP
StyleCLIP
TediGAN
[Figure]
ng r y
A g e d
B a l d
R e d h a
i r
Input
FFCLIP
StyleCLIP TediGAN D i s gu s t
B l u e e y e s
D r ea d l o c
k
s h a i r s t y l e
J e w fr o h
a
i r s t y l e
[Figure]
W h il e h
a
i r [Figure] l [Figure]
h a i r
W h il e h
a
i r
W h il e h
a
i r
W h il e h
a
i r
B l ond h a i r
B l ond h a i r
[Figure] r