S TYLE A LIGN : A NALYSIS AND A PPLICATIONS OF A LIGNED S TYLE GAN M ODELS
A BSTRACT
In this paper, we perform an in-depth study of the properties and applications of aligned generative models .
We refer to two models as aligned if they share the same architecture, and one of them (the child ) is obtained from the other (the parent ) via ﬁne-tuning to another domain, a common practice in transfer learning.
Several works already utilize some basic properties of aligned StyleGAN models to perform image-to-image translation.
Here, we perform the ﬁrst detailed exploration of model alignment, also focusing on StyleGAN.
First, we empirically analyze aligned models and provide answers to important questions regarding their nature.
In particular, we ﬁnd that the child model’s latent spaces are semantically aligned with those of the parent, inheriting incredibly rich semantics, even for distant data domains such as human faces and churches.
Second, equipped with this better understanding, we leverage aligned models to solve a diverse set of tasks.
In addition to image translation, we demonstrate fully automatic cross-domain image morphing.
We further show that zero-shot vision tasks may be performed in the child domain, while relying exclusively on supervision in the parent domain.
We demonstrate qualitatively and quantitatively that our approach yields state-of-the-art results, while requiring only simple ﬁne-tuning and inversion.
I NTRODUCTION
Transfer Learning (TL) refers to the process in which a parent model, pretrained for some source domain/task, is used to improve the performance of a child model on a different target domain and/or task (Pan & Yang, 2009).
The assumption underlying TL is that some knowledge learnt by the parent model is transferable to the new domain or task (Pan & Yang, 2009; Torrey & Shavlik, 2010;
Yosinski et al., 2014).
The most common TL approach is ﬁne-tuning, where the parent’s parameters are used to initialize those of the child.
Next, the child’s parameters, or sometimes just a subset of them, are trained on the target domain/task.
Once TL is completed, the child posseses some of the parent’s knowledge, despite the fact that the model parameters may have changed.
Existing TL literature typically examines the performance of the child model, e.g., in terms of classiﬁcation accuracy (He et al., 2019), or FID score (Karras et al., 2020a), without paying much attention to the relationship between parent and child models, induced by the transfer process.
Typically, the child model is simply applied to the task it was trained on, while the parent model is no longer used, having fulﬁlled its purpose.
In this work, we provide a complementary perspective, which focuses on analyzing and leveraging the shared knowledge between the two models.
Speciﬁcally, we consider the case where the TL is performed by ﬁne-tuning the same architecture.
We refer to models obtained in this manner as aligned models .
Several recent works (Pinkney & Adler, 2020; bryandlee, 2020; Kwong et al., 2021; Song et al., 2021; Gal et al., 2021) use aligned models in a novel manner.
In all cases, an unconditional GAN, speciﬁcally StyleGAN2 (Karras et al., 2020b) is ﬁne-tuned from domain A to domain B .
However, instead of applying the child model as an unconditional generator, it is used in conjunction with the parent model to form an image translation pipeline.
First, an image from domain A is embedded into the latent space of the parent StyleGAN2 model.
The resulting latent code is then fed either into the child model (bryandlee, 2020; Song et al., 2021; Gal et al., 2021), or into a hybrid model created by layer swapping , i.e., by combining layers from the parent and the child (Pinkney & Adler, 2020;
Kwong et al., 2021).
These methods have achieved great results in image-to-image translation between several domains.
Most notably, translating real human face images to a variety of styles such as cartoons and oil paintings (Pinkney & Adler, 2020;
Kwong et al., 2021; Song et al., 2021), but also translating humans to dogs and cats to wildlife (bryandlee, 2020; Gal et al., 2021).
However, they focus on a speciﬁc application (image-to-image translation) and do not explore or leverage aligned models further.
As a result, many questions arise but remain unanswered.
For example, which parts of the network change in the TL process, and which knowledge is inherited by the child from its parent?
To which degree do the answers to these questions depend on the similarity between the parent and child domains?
And, is knowledge not used by the child model completely lost or could it be recovered?
Finally, what further applications, besides image-to-image translation, can be solved using aligned models?
In this work, we delve deeper into model alignment.
In light of previous works, we speciﬁcally focus on the state-of-the-art unconditional GAN architecture, StyleGAN2 (Karras et al., 2020b).
The process of obtaining aligned models is incredibly simple: we start with a parent StyleGAN model trained on domain A and ﬁne-tune it fully for domain B , yielding an aligned child model.
We divide the investigation of model alignment into two parts.
First, in Section 3, we perform the ﬁrst empirical analysis of the phenomenon, answering the questions posed above, as well as others.
This analysis provides some surprising and novel insights that shed light on aligned StyleGAN models.
For example, we discover that when ﬁne tuning to a similar target domain, the parts of the model that change the most are the feature convolution weights in the synthesis network.
In contrast, the changes in the mapping network and afﬁne layers are negligible (see Figure 1).
This crucially implies that the learned latent spaces W and S are barely affected by the ﬁne-tuning.
This explains our next discovery – that semantically meaningful directions in the latent space of the parent model, retain the same (or similar) semantics in the child model (see Figure 2).
As the data domains become more distant, the mapping network and afﬁne layers become more affected, which results in a weaker degree of semantic alignment .
However, even in extreme cases, such as human faces and churches, some semantic alignment occurs.
Another surprising discovery is that the semantic latent controls that seemingly disappear after transfer to the child model, are in fact merely hidden, rather than forgotten, and reappear if the child is retrained back to the parent’s domain.
Second, in Section 4, we use aligned models to solve several popular Computer Vision and Computer
Graphics tasks.
We start with the aforementioned image-to-image translation task (Section 4.1), examine a number of alternatives, and show that aligned models obtain state-of-the-art results for a variety of scenarios.
This is especially impactful, as using aligned models for image translation is incredibly simple compared to dedicated methods for the same task, each devising its custom architecture and losses.
Next, we explore additional tasks, for which aligned models have not been used before.
In Section 4.2 we describe a simple method for fully automatic image morphing between fairly dissimilar domains, such as human to dog faces, which previously necessitated sophisticated methods (Aberman et al., 2018; Fish et al., 2020).
Examples of smooth morphs are included in the accompanying video.
In Section 4.3, we use aligned models to solve zero-shot classiﬁcation and regression tasks in domain B , where the supervision is available strictly in domain A .
Conceptually, our method reduces a task in a zero-shot or few-shot setting to the same task in a different data domain where supervision is plentiful.
In summary, while several previous works took advantage of aligned models implicitly, ours is the ﬁrst work to conduct a thorough empirical study of this phenomenon.
Our study reveals various interesting properties that we then use to further leverage aligned models for a variety of applications, almost effortlessly achieving state-of-the-art performance.
R ELATED W ORK
Latent Space of GANs: With the rapid evolution of GANs (Goodfellow et al., 2014) in recent years, understanding and controlling their latent representation has attracted considerable attention.
Speciﬁcally, it has been shown that the intermediate latent space of StyleGAN (Karras et al., 2019; 2020b;a) possesses appealing properties, such as being semantically rich, disentangled and smooth.
Many recent works have proposed methods to interpret the semantics encoded in that space and its extensions and apply them to image editing (Jahanian et al., 2019;
Shen et al., 2020a; H¨ark¨onen et al., 2020; Tewari et al., 2020;
Abdal et al., 2020; Wu et al., 2020; Patashnik et al., 2021).
In order to beneﬁt from these properties in real images, it is necessary to obtain the latent code from which a pretrained GAN can reconstruct the original input image.
This task, commonly referred to as GAN Inversion , has been tackled by numerous recent works, either by using: (i) optimization (Abdal et al., 2019; Karras et al., 2020b); or (ii) an encoder (Guan et al., 2020; Pidhorskyi et al., 2020; Richardson et al., 2021;
Tov et al., 2021); or (iii) a hybrid approach using both (Zhu et al., 2016; Baylies, 2019; Zhu et al., 2020a).
See Xia et al.
for a more thorough review.
Image-to-Image translation: The seminal pix2pix work by Isola et al.
(2017), ﬁrst introduced the use of conditional GANs to solve various supervised image-to-image translation tasks.
Since then, their work has been extended to allow image synthesis in various different settings: high-resolution (Wang et al., 2018a), semantic image (Park et al., 2019; Zhu et al., 2020b; Liu et al., 2019b), multidomain (Choi et al., 2018), multimodal (Zhu et al., 2017b), and using a pre-trained generator (Nitzan et al., 2020; Richardson et al., 2021;
Luo et al., 2020).
Another scenario that has received signiﬁcant attention is unsupervised image-to-image translation (Liu et al., 2017; Zhu et al., 2017a;
Kim et al., 2017; Choi et al., 2020; Lee et al., 2020b), where no paired data samples are given.
Regardless of the setting, all of the aforementioned works train an neural network, designed explicitly for the translation task.
Recently, several works (Pinkney & Adler, 2020; bryandlee, 2020;
Kwong et al., 2021; Song et al., 2021; Gal et al., 2021) have taken a different approach towards image-to-image translation.
They observe that signiﬁcant correspondence between generated images in different domains exists when an unconditional generator, such as StyleGAN2 (Karras et al., 2020b), is ﬁne-tuned between the two domains.
Accordingly, these works take a two-step approach towards image-to-image translation.
First, they invert a given image into the latent space of StyleGAN in domain A and then forward the output latent code through a StyleGAN model for domain
B .
The latter model is obtained either by directly ﬁne-tuning from the former model (bryandlee, 2020;
Song et al., 2021; Gal et al., 2021) or by layer swapping (Pinkney & Adler, 2020; Kwong et al., 2021), i.e., forming a model whose layers are partially those of the ﬁne-tuned model and partially those of the model for domain A .
In this work, we delve deeper into this phenomenon, which we refer to as model alignment , and go beyond the image-to-image translation task.
For example, we demonstrate that the alignment property goes beyond high-level properties, such as pose, and that multiple ﬁne-grained latent semantics are also aligned.
We leverage this property for tasks such as morphing and zero-shot classiﬁcation.
Fine-tuning and Catastrophic Forgetting: Fine-tuning was proven advantageous across ﬁelds, settings and tasks and therefore became a standard practice in the deep learning literature.
Prominent advantages of ﬁne-tuning are enabling few-shot tasks such as classiﬁcation (Chen et al., 2019) and unconditional generation (Wang et al., 2018b; Mo et al., 2020; Wang et al., 2020; Li et al., 2020;
Ojha et al., 2021), improved performance in a wide variety of tasks (Devlin et al., 2018;
Radford et al., 2018;
He et al., 2020) and faster training convergence (Wang et al., 2018b; He et al., 2019).
While ﬁne-tuning can be an effective technique for solving a new task, it has been well known for over 30 years (McCloskey & Cohen, 1989) that in the process the model “forgets” how to solve the original task, a phenomenon referred to as Catastrophic Forgetting (CF).
For example, once a GAN for a certain domain A , is ﬁne-tuned to another domain B , the resulting model can only generate images in domain B (Seff et al., 2017; Zhai et al., 2019).
In settings such as continual learning and multi-task learning, CF is undesirable.
In recent years, there has been progress in mitigating it using dedicated methods (Kirkpatrick et al., 2017; Kemker et al., 2018).
CF has been also studied in the context of GANs (Liang et al., 2018; Li et al., 2020; Thanh-Tung & Tran, 2020).
Aforementioned previous works devised methods to obtain a better child model using ﬁne-tuning.
From a ﬁne-tuning perspective, this means the model would perform better on the new task.
From a CF perspective, this means the model’s performance on the previous task should not be impaired.
We differ from these works signiﬁcantly, as we make no deliberate effort to affect what happens during training of the child model.
Instead, we investigate the relationship between the parent and child models after na ¨ ıve ﬁne-tuning, and then use it to solve a variety of applications.
A NALYSIS OF A LIGNED S TYLE GAN M ODELS
As explained earlier, several previous works observed that a signiﬁcant correspondence exists between images in different domains, generated from the same latent code by a parent StyleGAN
[Figure]
ting the weights of different components in child models (Mega, Dog) to their initial values, which come from the parent model (FFHQ).
Resetting the feature convolution weights causes the most drastic changes.
Also see Figure 8.
model and a child model obtained from it via ﬁne tuning.
Our goal is to further understand the relation between the parent and the child models.
Below, we explore several aspects.
Which parts of the network change during transfer?
Recall that the StyleGAN2 model is composed of a mapping function ( Z to W ), afﬁne transformations ( W to S ), feature convolution layers, and tRGB convolution layers that transform feature maps to RGB images.
We transfer a parent
StyleGAN2 model pretrained on FFHQ to the Mega cartoon dataset (Pinkney & Adler, 2020) and to AFHQ dog faces dataset (Choi et al., 2020), using ADA (Karras et al., 2020a).
After the transfer, we reset the weights of each of the above components in the child models (Mega, Dog) to their initial values in the parent model.
The results of this experiment are shown in Figure 1.
We observe that the greatest effect on the generated results is caused by resetting the feature convolution layers, which changes the content and structure.
Resetting the weights of other components, results in milder changes in both children.
This implies that feature convolution layers change the most during transfer.
The results also suggest that for the dog model, the afﬁne and tRGB layers have changed signiﬁcantly more than for the cartoon model.
We attribute this difference to the distance between the data domains, and additional experiments in the appendix (Figure 8) support this hypothesis.
While resetting the mapping network has a stronger effect on the dog model, note that the changes are fairly subtle in both datasets, implying that the mapping network changes very little.
Effectively, this means that the same z ∈ Z is mapped to similar codes in the W spaces of the parent and the child; in other words, the two W spaces are point-wise aligned .
This is a crucial observation as it explains the success of previous works (Pinkney & Adler, 2020; bryandlee, 2020; Kwong et al., 2021) in performing image translation based on aligned models.
Simply put, the two latent spaces may be viewed as a single shared latent space.
Thus, inversion serves as an encoder from the source domain to this latent space, and the generator is a decoder to the target domain.
Viewed in this light, alignment-based image translation resembles several previous image translation approaches (Liu et al., 2017;
Huang et al., 2018; Liu et al., 2019a), which are based on shared latent spaces.
Semantic alignment for similar domains.
In addition to point-wise alignment, we ﬁnd that the
W and S latent spaces of the child model are also semantically aligned with those of the parent model.
By semantic alignment, we refer to the property that latent space controls that affect various semantic attributes of images generated by the parent, have the same (or analogous) effect in the child model.
This phenomenon is demonstrated below, both qualitatively and quantitatively.
We demonstrate alignment on closely related domains, by ﬁrst ﬁne-tuning a parent pretrained on
FFHQ (Karras et al., 2019) to the Mega cartoon face dataset (Pinkney & Adler, 2020) and the
Metface portrait dataset (Karras et al., 2020a).
Next, we apply a variety of latent semantic controls learnt by the parent to the child models.
The controls are either individual channels in StyleSpace S , identiﬁed by Wu et al.
(2020), or directions in W space, from InterFaceGAN (Shen et al., 2020b).
We manipulate images using these controls “as is” in the parent and child models.
The initial latent code is obtained by inverting a real image with an e4e encoder (Tov et al., 2021).
As may be seen in Figures 2 and 9, regardless of the edited property, or the latent space used, the semantic controls affect the parent and the child models in exactly the same manner.
Also see Figures 10 and 11.
To perform a quantitative evaluation we measure the alignment by calculating the overlap between semantic controls found independently in the parent and child models.
Since latent directions in W are afﬁnely related to channels in S , we only examine overlap between style channels.
Concretely, we follow Wu et al.
(2020) to discover localized channels in both models, and report the number of localized channels for each semantic region in Table 1(a).
As can be seen, there is consistently large
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
amount of channels in
We that this overlap is not coincidental: performing the same experiment for two unaligned FFHQ models (trained from different random initializations) shows that they have much fewer overlapping channels (see Table 4).
To the best of our knowledge, we are the ﬁrst to quantitatively measure the ﬁne-grained semantic alignment phenomenon.
Our experiments indicate that aligned models for related domains are indeed strongly semantically aligned .
This phenomenon enables many applications based on transferring knowledge and supervision between aligned models.
E.g., zero-shot editing as demonstrated in Figure 2 and zero-shot classiﬁcation/regression as discussed in Section 4.3.
Semantic alignment for more distant domains.
To examine the degree of semantic alignment across a wider domain gap, we consider StyleGAN2 models transferred from FFHQ to AFHQ dog faces (Choi et al., 2020).
Figure 3 demonstrates that, even in this case, there are still multiple single-channel controls that retain their semantic meaning (e.g., big eyes, black hair, short hair).
Furthermore, there are also multi-channel editing directions in latent space that exhibit the same behavior, such as curly hair or small face (from StyleCLIP (Patashnik et al., 2021)), as well as pose (from InterFaceGAN (Shen et al., 2020b)).
This appears to be the case for visual attributes that are common to both domains, while controls for attributes that are not present in the target domain (such as glasses, lipstick, or beard) seem to have no effect on the child model.
However, as we discuss later, the relevant knowledge is not lost; rather, it is only hidden.
The retained controls reﬂect some interesting analogies between the domains: for example, controls for hair color and curliness in humans, control fur color and curliness in dogs, while hair length translates to length of dog ears.
Interestingly, psychologists have also observed a correlation between
[Figure]
FF HQ
D og
FF HQ 2 D og
D og
C a t the hair length in women and the ear shape of their preferred dog breeds (Coren, 1999), which is consistent with the folk belief that people look like their dogs.
The gradual emergence of some of these analogies is clearly revealed when examining the samples generated by the model as it evolves during the transfer process.
Figure 4 shows images obtained for the same latent vector z ∈ Z (in each row), as the training progresses.
In the top row, we can see how human hair gradually evolves into dog ears, and the human nose and mouth gradually evolve into the dog’s nose and muzzle, while the pose remains mostly unchanged.
A similarly smooth transition may be observed when transferring from AFHQ dogs to cats, as shown in the bottom row.
Using the same quantitative evaluation method as before, we further quantify the alignment between a parent FFHQ model and a child AFHQ dogs model.
Results are displayed in Table 1(b).
As can be seen, a smaller number of channels preserve their semantics when transferred to AFHQ dog as compared to MetFace.
Nevertheless, we still observe semantic alignment, albeit weaker, as human ears and hair overlap with dog ears, human cloth control the dog torso, etc.
We next experiment with even farther domains, with barely any similarity between parent and child, such as human faces and churches, which were also examined by Ojha et al.
Despite lack of commonality, the latent direction that controls face pose in the parent still controls the church pose in the child model (see Figure 14).
We further examine a double transfer, with FFHQ as parent,
AFHQ dog as child and LSUN bedroom as grandchild.
The pose direction in FFHQ still controls the pose in the grandchild bedroom model, as shown in Figure 14.
Are latent semantics forgotten or hidden?
As shown in Table 1(b), when transferring between distant domains, only a small portion of the localized controls retain a similar semantic function.
An interesting question that arises is: are the remaining controls completely “forgotten” during the transfer learning, or do they simply become inactive?
To examine this, we retrain the child
AFHQ dog model back to the FFHQ domain, thereby obtaining a grandchild model, and report the alignment between the original parent and the grandchild models (both for FFHQ) in Table 3.
It may be seen that the effect of many of the localized controls are restored.
For example, out of the 41 channels that control the ears in FFHQ, only 2 retain a similar function in AFHQ dogs, butregain their function in the grandchild model.
This implies that these channels were merely hidden, but not forgotten, during the ﬁrst transfer learning stage.
It should be emphasized that there is barely any such alignment between two unrelated models, even when they are trained on the same dataset, as shown in Table 4.
Thus, the signiﬁcant alignment between parent and grandchild (in Table 3) cannot be attributed to re-learning when ﬁne tuning from the child to the grandchild.
Locality bias in semantics transfer.
We explore this aspect and conclude that only some of the semantic alignment can be attributed to locality bias (see the discussion in appendix Section A.3).
[Figure]
A PPLICATIONS
We next apply aligned models to solve three kinds of tasks: image-to-image translation (Sec. 4.1), cross-domain image morphing (Sec. 4.2) and zero-shot classiﬁcation and regression (Section 4.3).
Efﬁcient training of generators for different resolutions is described in the appendix (Sec.
A.4).
4. C ROSS -D OMAIN I MAGE T RANSLATION
As demonstrated earlier, aligned models generate images with similar high-level semantic attributes, given the same latent code.
This makes it trivial to translate images between the domains of the parent and the child models, even when these domains are more distant than realistic faces and cartoons or paintings of human faces.
For example, it is easy to translate between faces of different species, which typically involves signiﬁcant changes in both structure and appearance.
Furthermore, there’s no need for task-speciﬁc training or losses; all that is needed is a pair of aligned models and an inversion method to embed real images into the latent space of the source domain StyleGAN.
In Section A.5 we perform a systematic study of which inversion methods (encoder or latent optimization), and which latent spaces ( W / W + / Z / Z + ), are most effective for image translation.
Some previous works (Pinkney & Adler, 2020; Kwong et al., 2021) that considered only similar domains have used the W / W + spaces.
We ﬁnd that Z space yields same level of results for similar domains, but superior results for distant domains, qualitatively and quantitatively.
This could be directly explained with a previous observation.
For both settings the Z space is trivially shared, as it is a non-learned space.
However, only for similar domains are the W / W + spaces aligned and shared.
In Figure 5 we compare our I2I results to two state-of-the-art methods, CUT (Park et al., 2020) and
F-LSeSim (Zheng et al., 2021).
It may be seen that our method produces realistic and natural looking results, while these two previous methods exhibit severe artifacts, and attempt to follow the shape in the source image too closely, yielding unrealistic results.
The table in Figure 5 provides quantitative support for our qualitative observations, yielding signiﬁcantly lower FID and KID scores for both cat2dog and dog2wild translations.
Figure 21 demonstrates our method’s ability to perform image translation between dissimilar domains.
In addition to the I2I scenario examined above, aligned models are also able to perform referencebased image translation, where the resulting image combines the content of a source image with the style from a second (reference) image (Huang et al., 2018; Choi et al., 2020).
StyleGAN inherently supports content and style disentanglement through style mixing.
Speciﬁcally, we combine the
Source
Reference
Ours
StarGAN2 OverLORD
Source
Reference
Ours
StarGAN2 OverLORD
StarGAN
OverLORD
Ours ( [Figure]
) dog2cat 15.
wild2dog 37.
cat2dog 44.
dog2wild 17.
(a) FID
StarGAN
OverLORD
Ours ( Z opt ) 12.
[Figure]
early latent code (below 32 × 32 resolution) from a source image, with the late latent code (above or equal to 32 × 32 resolution) from a target domain reference image, and feed it to the target model to generate the result, as demonstrated in Figure 22.
Figure 23 and Table 6 show that here, as well as for I2I, inversion via Z opt works better than other inversions/spaces for multi-modal image translation.
Figure 6 demonstrates that our results are better than those of current state-of-the-art methods, StarGAN-v2 (Choi et al., 2020) and OverLORD (Gabbay & Hoshen, 2021).
4. C ROSS -D OMAIN I MAGE M ORPHING
Image morphing is a popular visual effect of smoothly transitioning between a pair of input images (Wolberg, 1998), which typically requires either manual or automatic correspondences, in order to deﬁne a warp ﬁeld.
Cross-domain morphing, where the two images are from different domains,
A and B , is particularly challenging (Aberman et al., 2018; Fish et al., 2020).
However, using a pair of aligned StyleGAN models for the two domains, it is possible to perform cross-domain image morphing automatically without the need for correspondences, or any other input!
The two input images are ﬁrst embedded into the W + space of the corresponding generators, using e4e encoders (Tov et al., 2021).
Next, a smooth transition is obtained by linearly interpolating between the resulting latent codes, while also interpolating between the model weights.
Wang et al.
(2019) previously proposed interpolating model weights in order to obtain a smooth transition between the “effects” of two different networks.
We note that they do not discuss morphing real images, which is a slightly different setting, and requires also interpolating latent codes as we propose here.
Layer swapping , proposed by Pinkney & Adler (2020), is an alternative approach to morph between domains.
We discuss the differences between the two approaches in Section A.7.
Concisely, our proposed method ensures a continuous smooth transition, while layer swapping performs the transition as a series of discrete steps, rather than continuously.
We demonstrate automatic morphing between dog and cat faces in Figures 24 and 25, and dog and human faces in Figures 26 and 27.
Interpolating the model weights (along each column) yields a smooth transition between domains (different species, but the same pose and fur color), while interpolating the W + latent codes (along each row) smoothly transitions inside each domain (same species, varying pose and fur color).
In fact, any trajectory in this 2D interpolation space yields a smooth morph sequence between two input images.
We simultaneously interpolate along both dimensions to create the sequences shown in the supplementary video.
[Figure]
Negative
B l ac k
F u r
C u r l y
F u r
D o w n
E a r
K NOWLEDGE
Vision tasks on human faces have been researched for years.
Consequently, numerous datasets with detailed annotations exist.
For example, images in the CelebA dataset (Liu et al., 2015) are labeled with 40 attributes such as “Young”, “Curly hair”, “Smiling”, etc.
Such annotations are not available for almost any other domain, such as animal faces, severely limiting the range of tasks that can be solved.
As discussed earlier, this issue is a prominent motivation for transfer learning.
However, common transfer learning approaches are not applicable in the “zero-shot” setting, where there is abundant labeled data in the source domain, but strictly unlabeled data in the target domain.
We next show that this problem can be solved effectively for directly comparable attributes across domains by leveraging aligned models.
Consider the case of head pose (speciﬁcally, yaw): a clear and comparable attribute for both humans and dogs, however for humans there is abundant labeled data and for dogs there is none.
As demonstrated earlier, the latent pose semantics are aligned in the two models, and the parent’s yaw editing direction continues to edit yaw in the child.
As shown earlier, this holds for additional attributes.
Thus, despite a major gap between the two domains in image space, the gap in the latent space is considerably smaller, enabling transfer of knowledge between these domains.
While na¨ıvely applying a model trained on the source images to the target images would fail, this approach works well when applied on the latent representation.
To demonstrate this approach, we solve several zero-shot classiﬁcation and regression tasks using models trained in the latent space of the parent StyleGAN model.
For regression tasks, we use LARGE (Nitzan et al., 2021), which demonstrated that the distance in W + space to the decision hyperplane associated with a semantic property, gauges the degree of that attribute in image space.
See appendix (Section A.6) for more details.
Zero-shot yaw regression results are depicted in Figures 28 and 29.
As evident, the estimated yaw not only captures the correct tendency, but also produces a value that qualitatively seems reasonably close to actual yaw degree.
For classiﬁcation tasks we take a similar approach.
We simply replace the linear regression model with a logistic regression model and use a cutoff value of 0.5.
As shown in Figure 7, our method can turn classiﬁers for human faces to classiﬁers for dog faces.
These results also demonstrate that the attributes are not required to be exactly identical (pose to pose) but could be comparable in a more broad sense (long hair in humans to down-pointing ears in dogs).
C ONCLUSION
In this work, we performed the ﬁrst extensive investigation of the properties of aligned generative models .
We initially answered several open questions, crucial for their understanding.
The ﬁndings demonstrated impressive and surprising properties, such as semantic alignment across distant domains and knowledge being “hidden” instead of being forgotten.
We then leveraged our new insights to apply aligned models for a multitude of tasks.
Interestingly, we obtain state-of-the-art results for those tasks with a single, simple ﬁne-tuning based method.
We hope that our work can inspire others to consider aligned models as a simple paradigm for solving a wide range of tasks.
E THICS S TATEMENT
This work performs an extensive study of the properties of aligned generative models and applies such models for several computer vision tasks.
In general, generative models and learning-based algorithms raise several concerns.
Notably, generative models may be used to produce deceiving or offending content, e.g. deepfakes (Wikipedia, 2021), and data-driven algorithms may perpetuate biases exiting in their training sets.
However, these concerns are general to the entire ﬁelds and are not ampliﬁed by this work.
R EPRODUCIBILITY
Throughout the paper we provide detailed information facilitating reproduction of our results.
For example, in each experiment we specify the choices of latent space, speciﬁc layer and inversion method (e.g., Sections 4.1, 4.2 and A.5).
Similarly, when applying latent editing directions we specify with which method were they identiﬁed and in what space (e.g., Section 3, A.3).
Additionally, we provide in the appendix (Section A.8)
the information required to reproduce the child models.
We expect these to be sufﬁcient for independent replication of our main ﬁndings. Separately, source code and pretrained models have been made available in the project’s repository.
A CKNOWLEDGMENTS
We thank Daniel Cohen-Or for helpful discussions and encouragement and the anonymous reviewers for their comments. This work was supported in part by a gift from Adobe, by the Israel Science Foundation (grant no. 2492/20), and the Joint NSFC-ISF Research Grant Program (3611/21).
A A PPENDIX A. E FFECT OF TUNING ON INDIVIDUAL FEATURE CONVOLUTION LAYERS
As part of our investigation of which parts of the network change during ﬁne-tuning we also examine in more detail the effect of resetting the weights of individual feature convolution layers on the generated images.
We reset one layer at a time and measure the perceptual change in an image using
LPIPS.
Results are displayed in Table 2.
As may be seen each layer has an effect on the image but resetting the middle resolution layers (32, 64, 128) induces the greatest LPIPS change.
A. F URTHER SEMANTIC ALIGNMENT ANALYSIS
Since many semantic attributes cannot be controlled by a single style channel nor by a single manipulation direction in W we also compare the effect of different semantic manipulation directions in StyleSpace which are discovered for the parent model using CLIP (Patashnik et al., 2021).
Figures 10 and 11 demonstrate that these compound manipulations, e g., expressions and hair styles, also retain their semantics in the child models.
A. L OCALITY BIAS IN SEMANTICS TRANSFER .
We have demonstrated that a variety of localized controls retain their function during transfer learning between FFHQ and AFHQ.
However, since the faces in these two datasets are roughly aligned, it is interesting to examine whether this occurs due to overlap between the corresponding semantic regions.
To examine this, we perform transfer learning from a model pretrained on FFHQ (at 256 × 256 resolution) to three different versions of the same dataset: (i) shifted 60 pixels to the right, (ii) shifted 60 pixels downward, and (iii) ﬂipped upside down.
Figure 15 shows that the shifts, and particularly the ﬂip, affect the identity/appearance of the images generated from the same latent codes in Z , however other high-level characteristics, such as gender, age, or hair length, remain similar.
We also show the effect of manipulating ﬁve different style channels across different layers and different semantic regions.
For the horizontally shifted dataset, all ﬁve channels retain their function.
For the vertical shift, four out of the ﬁve channels retain their function (channel 15 45 that controls lipstick loses its effect).
For the upside down ﬂip, two out of ﬁve channels ( 9 409 for gaze and 12 479 for blond hair) retain their function.
In summary, for 11 out of 15 cases, the function of a channel was transferred despite a signiﬁcant change in the locality.
Thus, locality bias cannot explain all of the alignment that occurs.
There are, however, some interesting examples of strong locality bias.
Channel 6 501 controls smiling in the parent FFHQ model, but after an upside-down ﬂip it controls receding hairline, this implies locality bias does contribute to channel-wise semantics transfer, since the forehead of the ﬂipped faces overlaps the mouth location in the original images.
A. S EMANTIC ALIGNMENT BETWEEN DIFFERENT RESOLUTIONS
Given a high resolution StyleGAN2 model, an aligned lower resolution model may be easily obtained by simply removing the high resolution layers, and ﬁne-tuning to convergence.
The ﬁnetuning is necessary, as without it the model generates low-contrast images, as shown in Figure 30.
This works well because StyleGAN2 inherently supports multi-resolution synthesis, with the generator containing ToRGB layers and the discriminator containing corresponding FromRGB layers that directly operate in image space for different resolutions.
Assuming a high resolution ( 1024 × 1024 ) model is already available, creating a low-resolution model ( 512 × 512 ) in this way is computationally efﬁcient, requiring less than 2 days of ﬁne-tuning on a single GTX1080Ti GPU, compared to more than one month of training from scratch.
Figure 30 shows that the resulting low-resolution model is highly aligned with the original: the same latent code z ∈ Z generates nearly the same image, and the semantic controls in the parent model have the same effect in the child model.
One of the important consequences of such alignment is that there’s no need to spend weeks of GPU time to re-discover the semantic StyleSpace controls (Wu et al., 2020).
Furthermore, given an inversion model (Tov et al., 2021) for the parent model, it may be ﬁne-tuned for the child model within a few
GPU hours, instead of 2-3 days of training from scratch.
A. M ETHODS AND SPACES FOR IMAGE TRANSLATION
To determine which latent space and inversion method (encoder or optimization) is best suited for translation of real images, we explore a number of alternatives.
We modify the pSp encoder (Richardson et al., 2021) to embed images into W , Z , and Z + (Song et al., 2021) spaces.
For W + we use the e4e encoder (Tov et al., 2021), which is based on pSp, but generates W + codes with better alignment with the latent manifold.
We also modify the latent optimization method from the ofﬁcial StyleGAN2 implementation (Karras et al., 2020b) to embed into these different spaces.
For
Z / Z + , it is crucial to use the truncation trick for both image inversion and generation, otherwise the translation results might exhibit strong artifacts (we use a truncation coefﬁcient of 0.7).
Inversion results corresponding to these different methods are shown in Figure 16 for AFHQ dogs and cats.
Examples of I2I translation (dog2wild and cat2dog) using these different inversion methods are shown in Figure 17.
While inversion of source domain images to W + yields arguably the best reconstructions, when translating to the target domain via W or W + , the color palette of the results seems wrong, especially for the dog2wild translation.
We attribute this to the fact that the mapping function (from Z to W ) changes when ﬁne tuning the parent to the child, which affects the color palette, and translating using W / W + latent codes ignores this change.
Translations via Z + or
Z + opt inversion also suffer from occasional color artifacts (mainly in the dog2wild examples).
Both Z and Z opt , on the other hand, yield satisfactory translation results.
We prefer Z opt because it tends to produce a vivid color palette and to maintain a stronger resemblance of the source images, in terms of pose, shape, and colors.
Quantitatively, translating via Z opt inversion achieves best FID and KID over the other plausible alternatives, as reported in Table 5.
Therefore we use translation via Z opt as our preferred method.
We perform similar study for translation between nearby domains (FFHQ and cartoon) in Figure 18, 19 and 20.
In our subjective opinion, translation via Z opt still achieves the most cartoonish look.
However, translations via W bear closer resemblance to the input portrait, while still achieving a satisfactory cartoonish look.
As discussed in the text, this may be attributed to the fact that, for similar domains, the mapping function changes little during ﬁne-tuning, resulting in pointwise alignment of the W spaces of the parent and child models.
A. Z ERO - SHOT REGRESSION
To leverage aligned models for regression tasks, we use LARGE (Nitzan et al., 2021), which demonstrated that the distance in W + space to the decision hyperplane associated with a semantic property, gauges the degree of that attribute in image space.
As their method is designed for a few-shot setting, we simplify it slightly for our setting where the training data in the source domain is abundant.
Concisely, we simply use the distances calculated in speciﬁc layers known to control certain attributes as the input features for the regression model.
We demonstrate this approach for head pose regression and use the ﬁrst four layers, which are known to control the pose in StyleGAN (Karras et al., 2019;
Nitzan et al., 2021).
At inference time, we use e4e (Tov et al., 2021) to encode images of the target domain into the W + space of the child model, compute the distances of the ﬁrst four layers from the decision hyperplane, and input them to the human face yaw estimation model.
The zero-shot yaw regression results for AFHQ dogs and cats are depicted in Figures 28 and 29.
As can be seen, the estimated yaw not only captures the correct tendency, but also produces a value that qualitatively seems reasonably close to actual yaw degree.
A. M ETHODS TO BLEND ALIGNED MODELS
Layer swapping was introduced by Pinkney & Adler (2020) as a method to generate images of a new domain by “blending” together two existing data domains.
It does that by creating a hybrid model contains layers from two aligned models.
Speciﬁcally, the ﬁrst (coarse) layers are taken from one model and the last (ﬁne) layers are taken from another.
We note that this method blends the two data domains in a speciﬁc manner.
Thanks to the hierarchical structure of StyleGAN (Karras et al., 2019), the created model inherits the structure (coarse layers) from one model and texture from another (ﬁne layers).
Pinkney & Adler (2020) also mentioned that the ﬁne layers could be interpolated between the models, however this idea wasn’t applied in practice.
The layer swapping method was shown to produce visually pleasing results on the task of stylizing human portraits (Pinkney & Adler, 2020; Song et al., 2021).
However, there are a few disadvantages to this method.
First, when domains are more distant (e.g. faces of humans and dogs), the results obtained by this approach are less intuitive and visually pleasing (see Figures 31 to 33).
This coincides well with our observation from Figure 8.
Since the feature convolution layers change much more signiﬁcantly when transferring to a distant domain, the layers of a layer-swapped model are more alien to each other.
Second, the number of intermediate steps is limited by the number of convolution layers in the generator, which is at most 18.
This prevents the application of layer swapping for creating a smooth transition between images from different domains.
In our morphing application (Section 4.2), we present an alternative method to blend two aligned models.
There we propose to perform a simple linear interpolation of all model weights to achieve a gradual transition.
We compare the results of this approach with layer swapping in Figures 31 to 33.
Please note that our proposed method is able to obtain “blended” images that seem more smooth and natural.
A. F INE - TUNING IMPLEMENTATION DETAILS
Given a model pretrained on the parent domain, we ﬁne-tune it on the child domain.
Speciﬁcally, we use model conﬁg-f and the default hyper-parameters from the ofﬁcial Nvidia StyleGAN2 and
StyleGAN2-ADA implementations in tensorﬂow.
We use the augmentations of StyleGAN2-ADA only when the child domain is AFHQ or Metface.
Note that StyleGAN2-ADA implementation chooses the conﬁg based on input image resolution.
It uses conﬁg-f for image resolution above 512 × 512 , and conﬁg-e for other resolutions.
To use conﬁg-f without worrying about image resolution, one can specify the ﬂag --cfg stylegan when using tran.py, and change line 179 in train.py from spec.fmaps = 1 if res > =else 0.5 to spec.fmaps = 1 .
A. D ETECTING LOCALIZED CHANNELS
We follow Wu et al.
(2020) to discover localized channels in the StyleGAN model.
To reduce noise, we only consider channels to be localized if they have the strongest gradient in the same semantic region over 75% of sampling images, rather than 50% used in the original paper.
For FFHQ and
Metface models, we use the semantic segmentation maps from BiSeNet (Yu et al., 2018) pretrained on CelebAMask-HQ (Lee et al., 2020a).
For AFHQ dogs, we using the semantic segmentation maps from a uniﬁed parsing network (Xiao et al., 2018) pretrained on Broden+ (Bau et al., 2017).
[Figure]
[Table] [Table]
[Table]
This experiment serves as a negative control to show that a large number of overlap channels only occurs when the two and relation, as is the case in Table
Original
Bangs
Smile
Gaze
Pose
Age
Gender
FF
HQ
M e g a
M e t f ace
Original
Angry
Skinny
Bald
Bob Cut
Hi-top Fade
FF
HQ
M e g a
M e t f ace
[Figure]
[Figure]
[Figure]
[Figure] [Figure]
Original
FF
HQ
S h
i f t
D o w n
S h
i f t
R i gh t up - do w n ﬂ i p
FF
HQ
S h
i f t
D o w n
S h
i f t
R i gh t up - do w n ﬂ i p
[Figure]
Original
W
W +
Z
Z +
Z opt
Z + opt
Original
W
W +
W opt
W + opt
Z opt
Z + opt
Original
W
W +
W opt
W + opt
Z opt
Z + opt
[Table] [Table]
[Figure]
[Table]
hair becomes longer so does the dog’s fur the dog’s ears change ﬂame” to folded ( color of the
Source Reference         w+ enc
Z+ enc
Z enc z opts dog2cat 10.
wild2dog 44.
cat2dog 42.
dog2wild 37.
[Figure] FID w+ enc
Z+ enc
Z enc z opts 4.
[-(b) KID × 
-][Figure] [Figure] t [Figure]
t
[Figure]
=
[Figure] .t [Figure] = 0 .t 1 = 0 .t 1 = 0 .t 1 =
t=t=
.t=
.t= 
 .t=
.t=
t [-1 =t 1 = 0 .t 1 = 0 .t 1 = 0 .t 1 = 0 .t 
-][Figure] t [Figure]
=
[Figure]
.t [Figure] = 0 .t 1 = 0 .t 1 = 0 .t 1 = t=t=
.t=
.t= 
 .t=
.t=
t 1 =t 1 = 0 .t 1 = 0 .t 1 = 0 .t 1 = 0 .t
=
t=t=
.t=
.t= 
 .t=
.t=
t 1 =t 1 = 0 .t 1 = 0 .t 1 = 0 .t 1 = 0 .t
=
t=t=
.t=
.t=
.t= 0 .t=
[Figure] [Figure]
Original
Before FT
Smile
Gaze
Blond Hair
Lipstick p a r e n t 102c h il d c h il d
A
B
L a y e r
S w a
pp i
ng : A
B
A
B
L a y e r
S w a
pp i
ng : B
A
A 0.
B
W e i gh t
I n t e r po l a ti on
A
B
L a y e r
S w a
pp i
ng : A
B
A
B
L a y e r
S w a
pp i
ng : B
A
A 0.
B
W e i gh t
I n t e r po l a ti on
A
B
L a y e r
S w a
pp i
ng : A
B
A
B
L a y e r
S w a p p
i
ng : B A
A 0.
B
W e i gh t
I n t e r po l a ti on
Weight interpolation instead linearly interpolates the weights of layers interpolation
A
B
L a y e r
S w a
pp i
ng : A
B
A
B
L a y e r
S w a
pp i
ng : B
A
A 0.
B
W e i gh t
I n t e r po l a ti on
A
B
L a y e r
S w a
pp i ng : A B
A
B
L a y e r
S w a
pp i
ng : B
A
A 0.
B
W e i gh t
I n t e r po l a ti on
A
B
L a y e r
S w a
pp i
ng : A
B
A
B
L a y e r
S w a
pp i
ng : B
A
A 0.
B
W e i gh t
I n t e r po l a ti on
A
B
L a y e r
S w a
pp i
ng : A
B
A
B
L a y e r
S w a
pp i
ng : B
A
A 0.
B
W e i gh t s
I n t e r po l a ti on
A
B
L a y e r
S w a
pp i
ng : A
B
A
B
L a y e r
S w a
pp i
ng : B
A
A 0.
B
W e i gh t s
I n t e r po l a ti on
A
B
L a y e r
S w a
pp i
ng : A
B
A
B
L a y e r
S w a
pp i
ng : B
A
A 0.
B
W e i gh t s
I n t e r po l a ti on
[Figure] [Figure]
Original
Smile
Gaze
Blond Hair
Lipstick
P a r e n t
G r a nd c h il d
P a r e n t
G r a nd c h il d
P a r e n t
G r a nd c h il d
P a r e n t
G r a nd c h il d
[Table]