A C OMPREHENSIVE O VERHAUL OF D ISTILLING U NCONDITIONAL GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative adversarial networks (GANs) have achieved impressive results on various content generation tasks.
Yet, their high demand on storage and computation impedes their deployment on resource-constrained devices.
Though several GAN compression methods have been proposed to address the problem, most of them focus on conditional GANs.
In this paper, we provide a comprehensive overhaul of distilling unconditional GAN, especially for the popular StyleGAN2 architecture.
Our key insight is that the main challenge of unconditional GAN distillation lies in the output discrepancy issue, where the teacher and student model yield different outputs given the same input latent code.
Standard knowledge distillation losses typically fail under this heterogeneous distillation scenario.
We conduct thorough analysis about the reasons and effects of this discrepancy issue, and identify that the style module plays a vital role in determining semantic information of generated images.
Based on this ﬁnding, we propose a novel initialization strategy for the student model, which can ensure the output consistency to the maximum extent.
To further enhance the semantic consistency between the teacher and student model, we present a latent-direction-based distillation loss that preserves the semantic relations in latent space.
Extensive experiments demonstrate the effectiveness of our approach in distilling StyleGAN2, outperforming existing GAN distillation methods by a large margin.
Code and models will be released.
I NTRODUCTION
GAN compression (Wang et al., 2020; Li et al., 2020; Liu et al., 2021) has been actively studied to enable the practical deployment of powerful GAN models (Karras et al., 2018; 2019; 2020) on mobile applications and edge devices.
Among these techniques, knowledge distillation (Hinton et al., 2015) is a widely adopted training strategy for GAN compression.
The objective of GAN distillation is to transfer the rich dark knowledge from the original model (teacher) to the compressed model (student) so as to mitigate the performance gap between these two models.
There are two distillation manners, i.e., pixel-level and distribution-level.
The former minimizes the distance between generated images of two models, while the latter minimizes the distance between distributions.
In this work, we focus on the ﬁrst setting considering its prevalence in GAN compression literature (Chen et al., 2020; Wang et al., 2020; Li et al., 2020; Liu et al., 2021)
The majority of contemporary GAN distillation methods focus on conditional GANs (cGANs) while the distillation of unconditional GANs (uncGANs) is relatively under-explored.
Since there is a large difference between the learning dynamics of these two types of GANs, distillation methods tailored for cGANs cannot be directly applied to the unconditional setting.
Hence, in this work, we comprehensively investigate the distilling of uncGANs, especially StyleGAN2 (Karras et al., 2020), considering its popularity in both academia and industry.
We ﬁnd that the main difﬁculty of uncGAN distillation lies in the output discrepancy between the teacher and student model.
In fact, the implicit prerequisite of KD is that teacher and student should have similar outputs for the same input, otherwise the mimicking supervision is no longer meaningful.
This prerequisite is easier to be satisﬁed in cGAN, because the output space of cGANs can be narrowed down by the given conditional input, especially when the condition is strong (Zhu et al., 2017; Isola et al., 2017) Consider horse → zebra task as an example.
An input horse image determines which region should be added with zebra stripes and which region is background that should not be changed.
Two generated images in cGAN may differ in some low-level details such as the shape of zebra stripes, but would largely resemble in their structure.
Unlike cGAN, as shown in our experiments, it is impossible for a uncGAN student with random initialization to learn similar mapping function to the teacher, even though we leverage distillation loss to enforce the agreement between the outputs of the two models.
To address the aforementioned output discrepancy problem, we carefully examine each component of the StyleGAN2 student model through comparative experiments.
We identify that the style module plays a crucial role in deciding the semantic information of the generated images.
Based on this ﬁnding, we reach to a simple yet effective initialization strategy for the student model, i.e., inheriting the weights from the teacher style module and keeping the remaining convolutional layers randomly initialized.
Such initialization strategy can work well even in heterogeneous distillation where the student architecture is obtained by neural architecture search (NAS) or manual design, and is totally different from the teacher model.
After resolving the output discrepancy problem, we further design a more effective mimicking objective tailored for uncGAN distillation.
As opposed to most of existing GAN distillation approaches that merely transfer the knowledge within single image, we propose a novel latent-direction-based relational loss to fully exploit the rich relational knowledge between different images.
Speciﬁcally, we exploit the good linear separability property of StyleGAN2 in latent space and then augment each latent code w by moving it along certain direction such that the resulting image only differs in a single semantic factor.
Then, we compute the similarity matrix between original images and augmented images and take it as the dark knowledge to be mimicked by the student network.
The latent-direction-based augmentation disentangles various semantic factors and makes the learning of each factor easier, thus yielding better distillation performance.
Our contributions are summarized as follows: 1) To the best of our knowledge, this is the ﬁrst work that uncovers the output discrepancy issue in uncGAN distillation.
Through carefully designed comparative experiments, we identify that the style module is the determining factor to ensure output consistency.
We propose a concise yet effective initialization strategy for the student model to resolve the output discrepancy problem, demonstrating signiﬁcant gains upon conventional uncGAN distillation.
We further propose a latent-direction-based distillation loss to employ the rich relational knowledge between different images, and achieve state-of-the-art results in StyleGAN distillation, outperforming the existing state-of-the-art CAGAN (Liu et al., 2021) by a large margin.
R ELATED W ORK
GAN Compression.
We highlight a few recent methods among the many GAN compression methods (Shu et al., 2019; Chang & Lu, 2020; Chen et al., 2020; Wang et al., 2020; Li et al., 2020).
GAN
Slimming (Wang et al., 2020) integrates model distillation, channel pruning and quantization into a uniﬁed framework.
GAN Compression (Li et al., 2020)
ﬁrst searches a compact student architecture via NAS, and then requires the student to mimic the intermediate outputs and synthesized results of the teacher simultaneously.
A common characteristic shared by these works is that they all focus on the compression of cGANs such as Pix2pixGAN (Isola et al., 2017) and CycleGAN (Zhu et al., 2017).
The more recent Content-Aware GAN compression (CAGAN) (Liu et al., 2021) shifts the attention to compressing unconditional GANs.
It ﬁrst estimates the contribution of each channel in each layer to the generated human faces and eliminates channels with little contribution.
Subsequently, the pruned model inherits the well-trained parameters from the original network for both style module and convolutional layers, and are ﬁnetuned with adversarial loss and distillation loss afterwards.
Though CAGAN involves the compression of uncGAN, it bypasses the issues of model heterogeneity between the teacher and student model by allowing the student to inherit the well-trained parameters.
Such an requirement assumes the student to inherit the main structure of the teachers too despite pruning.
As will be shown in the experiments, the performance of CAGAN will greatly degrade in heterogeneous distillation.
The proposed mimicking loss cannot guarantee the student to learn a similar mapping as the teacher.
Moreover, we ﬁnd that the content-aware pruning strategy in
CAGAN is not an optimal solution for student initialization.
With our proposed initialization strategy, the student model does not need to inherit any weights from convolutional layers of the teacher but achieve better results.
Knowledge Distillation.
Knowledge distillation (KD) (Hinton et al., 2015) is originally proposed to achieve model compression (Buciluundeﬁned et al., 2006) for image classiﬁcation, whose target is to transfer the dark knowledge from one or multiple cumbersome networks (teacher) to a small compact network (student).
Vanilla KD (Hinton et al., 2015) proposes to match the outputs of two classiﬁers by minimizing the KL-divergence of the softened output logits.
Besides the output logits, other intermediate outputs such as feature maps (Romero et al., 2014), attention maps (Zagoruyko & Komodakis, 2017), Gram matrices (Yim et al., 2017), pre-activations (Heo et al., 2019), relation (Peng et al., 2019; Tung & Mori, 2019) and self-supervision signal (Tian et al., 2020; Xu et al., 2020) can also serve as the dark knowledge.
However, it should be careful when adapting KD from classiﬁcation tasks to generation tasks.
The output consistency prerequisite is naturally satisﬁed in image classiﬁcation since the supervision of labels guarantees different models to converge to similar mappings.
As discussed in Sec. 1, the consistency prerequisite does not naturally hold for uncGANs.
Therefore, a special distillation technique tailored for uncGANs is required to cope with the output discrepancy problem.
M ETHODOLOGY 3. P RELIMINARIES
StyleGAN2.
There are two modules in StyleGAN2 (Karras et al., 2020), i.e., a style module S ( · ) that maps Gaussian noise z to the style vector w and a convolution backbone C ( · ) that takes w as input and generate images.
The style vector w is fed into the backbone C ( · ) through the modulated convolution (ModConv) layer (Huang & Belongie, 2017; Karras et al., 2020).
And StyleGAN allows the use of different w vectors in different ModConv layers.
The image generation process can be formulated as:
[Equation]
where L is the number of ModConv layers in the backbone, and the i -th ModConv layer uses w i that comes from z i .
We deﬁne the output consistency condition as:
[Equation]
where the s and t represent student and teacher, respectively.
Equation 2 suggests that the generated images of two models should be the same if they use the same z at corresponding layers.
StyleGAN2 Compression.
A typical StyleGAN2 compression approach (Liu et al., 2021) contains two steps, i.e., pruning and ﬁnetuning.
In the pruning stage, unimportant / unnecessary channels will be removed according to some heuristics (Hu et al., 2016;
Li et al., 2017; He et al., 2018; Liu et al., 2021).
Note that pruning is only applied to the convolution backbone C ( · ) and the style module
S ( · ) is kept unchanged .
The pruned model will inherit the well-trained weights from the original model for both the style module and the convolution backbone (Liu et al., 2021).
In the ﬁnetuning stage, besides the normal adversarial loss, the pruned model is also required to mimic the original model’s output to compensate the performance degradation brought by channel reduction.
A typical mimicking loss includes RGB loss and LPIPS loss (Zhang et al., 2018):
[Equation]
where F is a well-trained frozen network that computes the perceptual distance between two images.
L rgb and L lpips require that the generated image of student should be close to that of teacher in RGB space and perceptual space, respectively.
The ﬁnal loss function in the ﬁnetuning stage is:
[Equation]
where λ ∗ is the loss weight of each item.
3. F RAMEWORK O VERVIEW OF U NCONDITIONAL GAN D ISTILLATION
Knowledge distillation is a common strategy that can bring improvements in classiﬁcation tasks.
However, in generation tasks, its prerequisite, namely the student and teacher having consistent outputs for the same input, is rarely mentioned.
In the absence of this prerequisite, the inﬂuence of mimicking losses to the training of student remains largely unknown.
Here, we hypothesize [Figure] that RGB or LPIPS loss is not compatible with GAN loss when the output discrepancy occurs and distillation will also bring no beneﬁt to the student.
We examine this hypothesis both qualitatively and quantitatively.
Note that the three losses in Eq.
4 serve different roles.
L GAN requires the student to generate realistic images while L rgb and L lpips encourage similarity between the generated images by student and those of the teacher.
Intuitively, if the generated image of the student are totally different from that of teacher for the same input, L rgb and L lpips will result in images that are slightly closer to teacher but with much less realism.
To examine this hypothesis, we remove the GAN loss in Eq. and keep only RGB or LPIPS loss.
We also cut off the gradient backward path between the student generator and generated images.
In this condition, the gradient of RGB/LPIPS loss directly works on the images.
The change of synthesized images reﬂects how RGB/LPIPS loss inﬂuences the generation process.
We select two student models, i.e., student-1 that has similar output with teacher for the same input and student-2 that has totally different outputs from teacher.
The effects of RGB/LPIPS loss are shown in Fig.
1a.
We can ﬁnd that the intermediate results are a mixup of source and target images to some extent.
If the source image is in the neighbor of the target image (1st and 2nd rows), the intermediate results are still perceptually realistic.
However, if the source image has a large distance to the target image (3rd and 4th rows), the intermediate results are no longer realistic.
Though RGB and LPIPS losses are reducing the distance between source and target images, they cannot guarantee a smooth and face-like interpolation in dissimilar setting.
And this unrealistic intermediate results naturally contradict with GAN loss.
From quantitative perspective, we wish to prove that RGB/LPIPS loss is not compatible with GAN loss in the heterogeneous setting by gradient analysis.
In the training process, for each batch, we perform backward propagation for GAN loss, RGB loss and LPIPS loss, respectively, and obtain three gradients of these losses.
We then compute the cosine distance between GAN gradient and
RGB/LPIPS gradients.
As shown in Fig.
1b and Fig.
1c, the cosine distance between GAN gradient and RGB/LPIPS gradients of dissimilar student is always negative, suggesting that RGB/LPIPS gradients are competing with GAN gradients.
On the contrary, the cosine distance of similar student is positive, indicating that the distillation loss is driving the model in the same direction as the adversarial loss.
Our analysis above suggests that distillation is not beneﬁcial in heterogeneous setting.
Having similar outputs for the same input z is the prerequisite for uncGAN distillation.
3. E FFECT OF THE STYLE MODULE
As we will show in the experiments, if the student is randomly initialized, it cannot learn consistent outputs as teacher even though we leverage RGB/LPIPS loss to force the agreement between the outputs of two models.
We hypothesize that the style module S ( z ) plays a key role in determining whether two models can have consistent outputs.
If the gap between style modules of student and teacher is too large, it is hard for the student to learn outputs consistent with the teacher.
We prove this hypothesis using a proof by contradiction.
Suppose the student has a different style module from the teacher and the consistency condition (Eq. 2) is also satisﬁed.
For the convenience of the following discussion, we deﬁne:
[Equation]
where all the ModConv layers use w 1 except that the k -th layer uses w 2 .
The consistency condition of Eq.
2 requires that:
[Equation]
As shown in StyleGAN (Karras et al., 2019), for a well-trained model, the w latent space consists of linear subspaces.
It should be possible to ﬁnd direction vectors that consistently correspond to individual factors of variation.
An example is shown in Fig.
Some individual semantic factors such as pose, glasses and hair color can be controlled by moving the style vector w of certain layer along a certain direction.
Suppose the direction p at k -th layer controls the hair color of the generated face.
The only difference between C t ( w 0 , w 1 + p ; k ) and C t ( w 0 , w 1 ; k ) is that they are the same faces with different hair colors.
The movement of w from w 1 to w 1 + p corresponds to a consecutive change of hair color of the generated face.
If we map the w back to the noise space:
[Equation]
obviously, the line segment in w space corresponds to a curve in z space with two end points z 1 and z 2 due to the nonlinearity of S t ( z ) .
We denote this curve as (cid:95) z
1 z 2 .
Then { G t ( z 0 , z ; k ) | z ∈ (cid:95) z
1 z 2 } represents a cluster of faces with different hair colors.
According to the consistency constraint, { G s ( z 0 , z ; k ) | z ∈ (cid:95) z 1 z 2 } should be the same cluster as { G t ( z 0 , z ; k ) | z ∈ (cid:95)
z 1 z 2 } .
We feed z 0 , z ∈ (cid:95) z 1 z 2 into the student style module S s ( · ) :
[Equation]
Since S s ( · ) is different from and independent of S t ( · ) , the result
[Equation]
is still a curve.
Thus, the semantic factor of hair color in student model is controlled by a complex curve in w space, which contradicts the property of StyleGAN2 that various semantic factors are decoupled well in w space.
Hence, having different style modules and consistency condition cannot hold at the same time.
We further conduct experiments to examine our hypothesis.
Speciﬁcally, we select four students according to whether the style module is from teacher or not and whether the convolution is from teacher or not.
We use GAN loss, RGB loss and LPIPS loss to train these models. The style module and convolution are updated together.
The results are shown in Fig.
The student that inherits weights from the teacher’s style module can learn a mapping that aligns well with the teacher’s output, no matter how the convolution C ( · ) is initialized.
However, for the student whose style module is randomly initialized, there are no meaningful connections between student’s and teacher’s outputs.
The analysis above clearly shows that the output consistency between student and teacher is determined by the style module.
3. S TYLE M ODULE C ONSISTENCY IN GAN D ISTILLATION
We have shown that the consistency between student and teacher outputs is the prerequisite of the distillation, and the style module determines whether two generators can have consistent outputs.
Hence, to make distillation meaningful, it is necessary to impose extra constraints to guarantee the consistency between two style modules.
The simplest way is to keep the architecture of the style module unchanged and inherit teacher’s parameters directly.
In fact, the parameters of style module account for only 7.5% of the parameters of the convolution backbone.
The FLOPs of style module account for about 0.005% of the FLOPs of the convolution backbone.
Preserving style module is thus feasible in practice.
If there is a strong demand on the compression of style module, one can perform a two-stage training to ensure a small gap between student and teacher style modules.
In the ﬁrst stage, the student style module is forced to mimic outputs of the teacher style module:
[Equation]
where D ( · , · ) is a distance metric.
Considering S t ( · ) and S s ( · ) are both shallow MLPs, the training cost of this stage is negligible (0.59% of the normal GAN training in the second stage).
In the second stage, the style module and generator backbone are ﬁnetuned together using the loss in Eq.
We will explore the effects of compressing the style module in Sec. 4.1.
3. L ATENT - DIRECTION - BASED RELATION DISTILLATION
Under the premise that consistency condition is satisﬁed, we further propose to incorporate relation mimicking into GAN distillation.
Conventional relation-based distillation (Tung & Mori, 2019) in classiﬁcation task computes feature similarity matrices using the samples in a minibatch.
Here, we tailor it to better cater to StyleGAN2.
Speciﬁcally, for a given well-trained teacher model, we compute all its meaningful latent directions (LD) that control a single semantic factor and store them in a dictionary { d 1 , d 2 , · · · , d m } .
Note that the latent direction is related to a speciﬁc layer.
For example, if d i is computed in k -layer, then only
C t ( w, w + d i ; k ) has single semantic factor difference with C t ( w ) .
C t ( w, w + d i ; j ) j (cid:54) =
k does not has this property.
In the training stage, we feed a batch of noise { z i } i =1: N into the style module and obtain { w i } i =1: N .
For each w i we randomly sample a latent direction d j from the dictionary.
Thus,
C t ( w i ) and C t ( w i , w i + αd j ; k ) ( k is the layer related to d j ) are two images with single semantic factor difference with α controls the moving distance.
We denote the intermediate features of C t ( w i ) and C t ( w i , w i + αd j ; k ) as f i and f (cid:48)
i , respectively.
Then the similarity matrix M between original view and augmentation view can be computed as A i,j = f i · f (cid:48)
j .
We then convert the similarity into probability via the softmax operation and minimize the distance using KL-divergence loss:
[Equation]
The ﬁnal learning objective is the combination of Eq.
4 and L LD .
E XPERIMENTS
We conduct experiments mainly on StyleGAN2 since it is the most powerful unconditional GAN so far.
We use the FFHQ (Karras et al., 2019) dataset.
Following CAGAN (Liu et al., 2021), we adopt Fr´echet Inception Distance (FID), Perceptual Path Length (PPL) (Karras et al., 2019) and
PSNR/LPIPS (Liu et al., 2021) between real and projected images as evaluation metrics.
More qualitative results are shown in the appendix.
[Table] For the ablation study in Sec. 4.1, we train the models on resolution 256 × 256 and use a smaller batch size of 8 to save the computation cost.
For the comparison with state-of-the-art methods in
Sec.
4.2, we train the models on both resolutions of 256 × 256 and 1024 × 1024.
We also use a batch size of 16 that is the same as CAGAN (Liu et al., 2021) to ensure a fair comparison.
4. A BLATION S TUDY
The Initialization of the Student Model.
Previous works usually treat StyleGAN2 as an integral module and initialize style module and convolution backbone in the same way (from scratch or inherits teacher parameters).
Based on our analysis in Sec. 3.3 that the style module plays a key role in determining the semantics of generated images, here we separate style module S ( z ) from convolution backbone C ( w ) and test three initialization strategies: 1) both S ( z ) and C ( w ) are randomly initialized, 2) both S ( z ) and C ( w ) are initialized with teacher weights, 3) only S ( z ) inherits teacher weights and C ( w ) is randomly initialized.
The results are shown in Table.
For the setting where S ( z ) and C ( w ) are both randomly initialized, RGB loss can only bring marginal improvement.
RGB+LPIPS even performs worse than
No-Mimic, indicating that distillation cannot work well when output discrepancy occurs.
If S ( z ) and C ( w ) both inherit teacher weights, the mimicking loss can achieve 1-2 FID improvement.
To explore the effect of style module, we also try inheriting only S ( z ) and surprisingly ﬁnd that this initialization obtains the best result.
And loading C ( w ) hampers the performance of RGB+LPIPS and RGB+LPIPS+LD.
This result contradicts with the conclusion in CAGAN (Liu et al., 2021).
It shows that the general pruning strategy, i.e., determining which channels should be removed, is not important.
Randomly initialization of convolution layers is the optimal solution as long as the style module is kept.
The Effects of Style Module Compression.
We conduct experiments to investigate how to deal with the style module in StyleGAN2 compression.
Speciﬁcally, we consider three settings: 1) student has the same style module architecture as teacher but with random initialization, 2) student style module has a different architecture and uses the two-stage training strategy, 3) student has the same architecture and inherits the weights from the teacher style module.
For all the settings, the convolution backbones are randomly initialized.
For the two-stage setting, we also explore how the architecture of the style module and mimicking loss in Eq.
9 affect the ﬁnal performance.
To emphasize the importance of style module, we also list the average L1 distance between S s ( z ) and
S t ( z ) before entering the normal GAN training stage.
The results are shown in Table.
‘Random Initialization’ obtains the worst FID because the output discrepancy makes the distillation ineffective.
The ‘Two-Stage’ strategy improves the results by narrowing the gap between S s ( z ) and S t ( z ) .
From several two-stage settings, we can ﬁnd that L [Table] is a better mimicking loss than L2 and reducing the number of layers is better than reducing the number of channels in each layer.
It is also worth noting that there is a strong positive correlation between | S s ( z ) − S t ( z ) | and FID, indicating that the gap between S s ( z ) and S t ( z ) determines the output consistency and further determines the inﬂuence of distillation.
Single View brings marginal improvement.
Random Offset even has negative effect.
Our LD loss consistently improves the performances of both
RGB and RGB+LPIPS.
Mimicking Loss
[Table]
RGB
[Table]
RGB +
[Table]
Offset
KL
[Table]
RGB + Single View KL 9. RGB + LD L 9.
RGB + LD
KL 9.
RGB + LPIPS
N/A 8.
RGB + LPIPS + LD
L 8.
RGB + LPIPS + LD
KL 8.
Though the two-stage strategy brings performance gains, there is still a large gap between it and the ‘Inheriting’ variant.
Thus, we conclude that the modiﬁcation to the style module will greatly harm the ﬁnal performance and the two-stage strategy can only mitigate the degradation to a certain degree.
Considering that the scale of the original S t ( z ) is negligible compared to the convolution backbone, the best practice in StyleGAN2 compression is to preserve the style module architecture and inherit the weights from the teacher style module.
The Effects of Latent-Direction-Based Distillation Loss.
The proposed latent-directionbased loss is essentially a relation loss.
We are interested in whether the beneﬁt brought by L LD comes from relation mimicking or from the latent-direction-based augmentation.
Speciﬁcally, we consider three variants: 1) Single View, namely the similarity is computed inside the normal samples rather than between normal samples and augmented samples, 2) Random Offset, namely we move w along a random direction to get f (cid:48)
i instead of along the latent direction, 3)
Our latent-direction-based method (abbreviated as LD).
The results are shown in Table.
RGB+Single
View obtains comparable results with RGB loss, indicating that naive relation mimicking cannot bring improvements.
RGB+Random Offset performs even worse than RGB loss, demonstrating that the way of augmentation is critical for relation mimicking.
Improper augmentation may be counterproductive.
LD improves RGB by around 0.3 FID.
And LD can still bring 0.16 FID improvement to the RGB+LPIPS variant.
Among two kinds of
L LD , KL-divergence loss is slightly better than L2 loss.
4. C OMPARISON WITH STATE - OF - THE - ART METHODS
We compare our method with the GAN Slimming (Wang et al., 2020) and CAGAN (Liu et al., 2021) methods.
Since our method does not focus on the pruning, we directly adopt the student architecture used in CAGAN, i.e., a network that is the same as teacher but with fewer channels.
We also compare with CAGAN in heterogeneous setting where the student is not a subnet of the teacher.
Speciﬁcally, we modify the kernel size of the second convolution layer in each resolution block from 3 to 1, thus inheriting teacher convolution parameters is infeasible.
Since CAGAN did not notice the output [Table] discrepancy issue and always initialize the style module and convolution backbone in the same way, we assume it does not inherit weights from teacher in heterogeneous setting.
The results are shown in Table.
Our method outperforms CAGAN on FID by 0.65 and 0.41 on resolution 256 × 256 and 1024 × 1024, respectively, showing that our method can generate more realistic images.
Note that these improvements are not marginal considering the images generated by CAGAN already have high quality.
For the PPL metric that measures the smooth degree of latent space, we outperform CAGAN by 0.008 on resolution 256.
The gap is even larger (0.029) on resolution 1024.
For PSNR and LPIPS that are related to image projection ability, our method also surpasses CAGAN, demonstrating that our method can model the face distribution in real world better.
Our superiority is much more signiﬁcant in heterogeneous setting, showing that our method can be applied in a more general situation where the student is not necessary to be a subnet of the teacher.
C ONCLUSION
In this paper, we uncover the output discrepancy issue in uncGAN distillation.
Through comparative experiments, we ﬁnd that the style module is the key to the output discrepancy and propose a novel initialization strategy of student, which can help resolve the output discrepancy issue.
The proposed latent-direction-based distillation loss further improves the distillation efﬁcacy and we achieve stateof-the-art results in StyleGAN2 distillation, outperforming the rival method by a large margin on image realism, latent space smoothness and image projection ﬁdelity.  L IMITATIONS
The analysis and method in this work are tailored for StyleGAN2-like generator, which may affect its universality.
It is unknown whether our analysis is still valid for those uncGANs that do not have style module such as BigGAN (Brock et al., 2019) and PGGAN (Karras et al., 2018).
Whether conventional mimicking loss such as RGB/LPIPS loss can solve the output discrepancy and how to initialize them are not clear. Besides, we mainly conduct experiments on FFHQ which is a wellaligned face dataset. Experiments on more and harder datasets such as ImageNet, outdoor scenes and rooms should be included to examine the effectiveness of our method. Finally, we only consider the output discrepancy issue in unconditional GANs.
In fact, this problem also exists in conditional setting when the condition is not strong enough (e.g., the conditional input is the class label).
How to analyze the output discrepancy issues of uncGANs and cGANs in a more general form is also a direction worth exploring.
A PPENDIX A I MPLEMENTATION DETAILS
Training Hyperparameters.
For the style module mimicking (Eq. 9) of the ﬁrst stage, we use
Adam optimizer with a initial learning rate of 0.05.
We train for 50k steps with a batch size 4096.
For the normal GAN training of the second stage, we use Adam optimizer with a initial learning rate of 0.002 and 450k iterations.
For the α that controls the offset along latent direction, we sample it from a Gaussian distribution N (0 , 5) .
We set λ GAN , λ rgb , λ lpips and λ LD to be 1, 3, 3 and 30, respectively.
Evaluation Metrics.
Fr´echet Inception Distance (FID) is commonly used metric to evaluate the realism of generated images.
The generated images and real images are fed into a inception network and then a Fr ´ echet distance is computed between their corresponding feature maps.
We use the implementation of FID in CAGAN (Liu et al., 2021). Speciﬁcally, we use 50K real images and 50K generated images to compute statistics, respectively.
Perceptual Path Length (PPL) is proposed in StyleGAN (Karras et al., 2019) to measure the smoothness of latent space.
We adopt the PPL implementation in CAGAN (Liu et al., 2021) for a fair comparison.
PSNR and LPIPS are proposed by CAGAN to evaluate the image projection ability.
A given real image is ﬁrst mapped back to the latent space through optimizer such as L-BFGS.
The projected image is obtained by feeding this resulting latent code to the generator.
Then the PSNR and LPIPS distance are computed between projected image and the original image again.
A smaller value indicate that the generator can model the distribution in real world better.
We compute these two metrics using our own implementation.
B S TYLE GAN2 L INEAR SEPARABILITY
A well-trained StyleGAN2 model is linear separably in latent space.
An example is shown in Figure.
C I MAGE E DITING
We demonstrate the superiority of our method on image editing, including style mixing and interpolation.
Given two real face images I A , I B , we ﬁrst project them back to the latent space and get w A , w B .
Both w A and w B are of shape L × D , where L is the number of convolution layers and
D is the dimension of latent code.
For style mixing, we replace the i − th vector in w A with that from w B .
We set i ∈ [1 , 3] , i ∈ [5 , 8] and i ∈ [10 , 13] for coarse, middle and ﬁne style mixing, respectively.
For interpolation, we linearly combine the latent code with β controls the weight: w = β · w A + (1 − β ) · w B , and then feed w into generator to get the interpolation results.
We edit the images on resolution 256 × 256.
The results are shown in Fig.
For style mixing, CAGAN always has artifacts in face shape (coarse style) and skin color (middle shape).
While our results are more realistic and correspond better with two source images.
In coarse style case, our result corresponds well on face shape and sense organs with source B.
In ﬁne style case, our result corresponds well on lighting and skin color with source
B. For interpolation, we also observe a smoother change than CAGAN, showing that our method learns a better structure in latent space.
D I MAGE P ROJECTION
We show image projection results of our method in Fig.
All the real images come from Helen
Set55 (Liu et al., 2021) and are not seen in the training stage.
Our model reconstructs them with high quality.
E G ENERATION R ESULTS
We show more generation results of resolution 256 × 256 and resolution 1024 × 1024 in Fig.
6 and
Fig. 7, respectively.
(a
In coarse style mixing, CAGAN generates glasses, which does not appear in both source images.
CAGAN also produces blurry images in middle style case.
In contrast, our style mixing results are more realistic and more similar to teacher.
Our method also shows smoother transition of mouth and haircut in interpolation.
b) In
coarse style mixing
, our
result corresponds better with source B on mouth and face shape.
In ﬁne style mixing,
our
result
corresponds better with source B on skin color.
CAGAN
also generates artifacts on hair in middle and ﬁne style cases.
In interpolation, our
methods are more consistent
with teacher than CAGAN.
c) CAGAN generates lighting artifacts
in coarse case and skin color artifacts in ﬁne
case while our results are more realistic
In
interpolation of CAGAN,
the earrings disappear in β = 0 .
while appear again in β =
In contrast, our results are much smoother