D ISTILLING GAN S WITH S TYLE -M IXED T RIPLETS FOR X2I T RANSLATION WITH L IMITED D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Conditional image synthesis is an integral part of many X2I translation systems, including image-to-image, text-to-image and audio-to-image translation systems.
Training these large systems generally requires huge amounts of training data.
Therefore, we investigate knowledge distillation to transfer knowledge from a high-quality unconditioned generative model (e.g., StyleGAN) to a conditioned synthetic image generation modules in a variety of systems.
To initialize the conditional and reference branch (from a unconditional GAN) we exploit the style mixing characteristics of high-quality GANs to generate an inﬁnite supply of style-mixed triplets to perform the knowledge distillation.
Extensive experimental results in a number of image generation tasks (i.e., image-to-image, semantic segmentation-to-image, text-to-image and audio-to-image) demonstrate qualitatively and quantitatively that our method successfully transfers knowledge to the synthetic image generation modules, resulting in more realistic images than previous methods as conﬁrmed by a signiﬁcant drop in the FID.
I NTRODUCTION
Conditional image synthesis, also X2I translation , maps from an input domain (e.g. text, audio, segmentation maps, etc.) to the image domain.
Beneﬁting from GANs (Goodfellow et al., 2014) and its follow-up improved versions (Gulrajani et al., 2017; Kang & Park, 2020; Salimans et al., 2016), they obtain remarkable performance on a wide variety of image synthesis tasks: image to image (I2I)
(Lee et al., 2018; Zhu et al., 2017), audio to image (Chen et al., 2017; Wang et al., 2020a), text to image (Hu et al., 2021;
Li et al., 2019; 2020; Radford et al., 2021;
Zhang et al., 2017a) and semantic segmentation map to image (Isola et al., 2017; Wang et al., 2018).
Despite impressive leaps forward for a variety of image synthesis tasks, there are still important challenges.
Speciﬁcally, to obtain good results, existing works rely on large labelled datasets.
Labeling these datasets is both laborious and time-consuming, considerably reducing the practical impact of these methods.
It is noteworthy to observe that many of these models (Zhang et al., 2017a) apply transfer learning to the text and audio encoders (e.g. using a pretrained LSTM (Reed et al., 2016) model for text and pretrained GRU (Merkx et al., 2019) model for audio), however they train the image synthesis decoder from scratch.
This happens because there are no established methods to transfer pretrained GANs to conditional image decoders; an omission which we aim to address in this paper.
In this paper, we investigate knowledge transfer for a variety of conditional image synthesis tasks.
Traditional knowledge transfer for conditional image synthesis is often not possible, because there might not be a pretrained network available for the desired translation task (e.g. at the moment no high-quality pretrained network for segmentation map-to-image translation is available).
It would therefore be preferable if the wide variety of high-quality GANs available for image generation could be exploited for X2I. Recent works (Wang et al., 2021; 2020b) leveraged a pretrained GAN to initialize an I2I translation model, managing to transfer knowledge to different image synthesis tasks.
These methods, however, suffer from three problems: (I) They can only be used for I2I translation and do not generalize to other conditional image synthesis tasks.
(II)
They are GAN architecture-speciﬁc approaches, requiring the GAN architecture within the X2I system to be exactly the same as that of the pretrained GAN.
This limits transfer to current state-of-the-art X2I systems for which no similar GAN architecture exists (like for example StarGANv2 (Choi et al., 2020) for
I2I).
(III) X2I systems are based on a conditional GAN, however existing methods for knowledge transfer for I2I do not initialize the conditional branch during the transfer, and therefore this has to
[Figure]
be learned from scratch during the ﬁnetuning on the target dataset.
Learning this on small target datasets can be problematic.
To address the aforementioned problems, we propose several improvements for knowledge transfer to X2I systems: (I) we are the ﬁrst to investigate knowledge transfer for X2I translation.
Therefore, we propose a novel, uniﬁed transfer learning method, which can be used for varying kinds of conditional image synthesis tasks (Figure 1) which is based on generated images and therefore does not require any real data.
(II)
The student generator does not need to have the same architecture as the pretrained GAN.
Therefore, we can use well-devised speciﬁc image synthesis architectures (e.g.,
SPADE (Park et al., 2019) and StarGANv2 (Choi et al., 2020)) by distilling knowledge from the pretrained teacher GAN (e.g., StyleGAN) to the task-speciﬁc student GAN.
(III)
We use the style mixing characteristic of StyleGAN to create style-mixed triplet data, which are used to transfer the knowledge efﬁciently to both I2I and X2I translation models.
Furthermore, we propose a semantic diversity loss based on the style-mixed triplet, which contributes to learn the semantic information of the output image.
We perform experiments on a wide variety of image synthesis tasks, including text-to-image, audioto-image, segmentation map-to-image and I2I translations.
We demonstrate the efﬁciency of the proposed knowledge distillation method, providing qualitative and quantitative results.
We prove that the single pretrained GAN model can be universally
used in varying speciﬁc task model
Additionally, leveraging the style mixing character of StyleGAN, further improves I2I translation performance
R ELATED WORK
GAN-based Conditional Image Synthesis.
Beneﬁting from the advances in GANs and its variants in recent years, conditional image synthesis (also called X2I translation) research has developed rapidly.
Two typical approaches have been investigated for GAN-based image synthesis, namely, unsupervised (Kim et al., 2017;
Yi et al., 2017) and supervised image generation (Park et al., 2019;
Zhang et al., 2017b).
The latter inputs conditional information (e.g. text, audio, image, segmentation map etc.) to synthesize images which contain the corresponding semantic information (i.e. the conditional information).
Speciﬁcally, text-to-image translation (Hu et al., 2021; Li et al., 2020;
Zhang et al., 2017a) aims to synthesize high-realistic images which are semantically consistent with the text descriptions.
Recent work (Hu et al., 2021) introduces semantic-spatial batch normalization to better exploit the text information.
Similar to text-to-image translation, both audio-to-image translation (Chen et al.,
; Wang et al., 2020a) and segmentation map-to-image translation (Bau et al., 2020; Park et al., 2019) aim to learn a mapping from the audio/segmentation map to the output image.
Different to the above image synthesis tasks, image-to-image translation (Park et al., 2020;
Zhu et al., 2017) performs projection from the source to the target image domain.
In this paper, we explore transfer learning from GANs to a variation of conditional image synthesis tasks.
Transfer learning.
A considerable research effort has investigated transferring knowledge for both discriminative (Donahue et al., 2014; Hinton et al., 2014) and generative tasks (Noguchi & Harada, 2019;
Zhao et al., 2020).
There also exist several approaches (Goetschalckx et al., 2019; Jahanian et al., 2020) which focus on the image manipulation based on the pretrained GAN. Given a target
[Figure]
semantic attribute they aim to manipulate the output image of a pretrained GAN.
However, these methods do not focus on transfer learning for target data.
Some other methods (Abdal et al., 2019;
Zhu et al., 2020a) reverse the given image into the input latent space of the pretrained GAN (e.g.,
StyleGAN), and manage to restructure the target image by optimization of the latent representation.
Recent work (Shocher et al., 2020; Wang et al., 2021; 2020b) performed knowledge transfer from a pretrained classiﬁcation model (e.g., VGG (Simonyan & Zisserman, 2014)) or the discriminator (BigGAN (Brock et al., 2019)) for I2I translation.
However, both DeepI2I (Wang et al., 2020b) and
TransferI2I
(Wang et al., 2021) require that the GAN architecture is identical with the generator used in the I2I architecture.
As a consequence, these methods cannot be applied to well-designed
I2I translation architectures (like starGANv2 (Choi et al., 2020)) since they do not use a standard
GAN architecture.
The proposed method could address these problems.
K NOWLEDGE T RANSFER FOR X2I
Problem setting.
Our goal is to transfer knowledge from a pretrained high-quality unconditional
GAN to an X2I translation system for the case when training data is limited.
The proposed method consist of two stages.
In the ﬁrst stage we perform the data-free knowledge transfer method explained in Sec. 3.2 and 3.3.
This stage does not require any target data.
In the second stage, we apply a standard ﬁnetuning of this distilled model on the target dataset.
Distillation can be used to perform knowledge transfer to other architectures, however, there is little target data available.
Fortunately, the pretrained GAN can generate inﬁnite data for data distillation, meaning that we do not require access to any real data (i.e., data-free ).
Also, we transfer knowledge from an unconditional
GAN (e.g., StyleGAN) to a conditional GAN (e.g., I2I translation); requiring us to propose techniques to initialize the conditional branch.
Exploiting the style-mixed characteristic (Sec. 3.1) of
StyleGAN we propose two solutions: one for I2I
(Sec. 3.2) and another one for X2I
(Sec. 3.3).
Our method consists of two stages:(1) transfer leaning without any real data (i.e., style-mixed triplets in
Sec. 3.1, I2I translation in Sec.
3.2 and X2I translation in Sec. 3.3 ) and (2) ﬁnetunning with the real target data.
3. S TYLE - MIXED TRIPLETS
Beneﬁting from the style mixing ((Figure 2 (a))) characteristic of StyleGAN, we are able to create an inﬁnite amount of triplet images from any of the two domains.
Let z t ∈ R Z indicates the input noise of the pretrained generator G T (teacher).
given the input noises z 1t and z 2t , the mapping network
M T of the StyleGAN generator encodes them to a style vector s 1t =
M T ( z 1t ) and s 2t =
M T ( z 2t ) .
We deﬁne G T ( z )
=
G (cid:48) T ( M T ( z ))
where G (cid:48)
T takes styles vectors as an input to each of its layers.
We further feed these style vectors to the teacher generator G (cid:48) T to output x 1t = G (cid:48) T ( s 1t ) and x 2t =
G (cid:48) T ( s 2t ) (where x 1t , x 2t ∈ X and image domain X =
R H
× W × 3 ) respectively.
Here the function Φ( s 1t , s 2t ) selects to forward s 1t to the layers of G (cid:48)
T important for the content of the generated image (the bottom layers), and s 2t to those important for the style (the top layers) 1 .
Interestingly, a new image y t =
G (cid:48) T (Φ( s 1t , s 2t ))
(where y t ∈ Y and image domain Y =
R H
× W × 3 ) contains mixed information with respect to content and style of both inputs x 1t and x 2t .
Based on this observation above, we propose to leverage these style-mixed triplets ( x 1t , x 2t , y t ) to perform distillation for the
I2I and X2I translation models without the need of any real triplet training data.
3. D ATA - FREE KNOWLEDGE DISTILLATION FOR I2I TRANSLATION
We aim to leverage the style-mixed triplets to perform knowledge transfer for the I2I model.
Let x t ∈ X and y t ∈ Y indicate the two domains.
We expect to map an image x t from domain X into another sample y t which mimics to Y .
Note we never get access to any real data.
Figure(b) shows the knowledge distillation framework for I2I translation.
Our framework is composed of six neural networks: a pretrained teacher GAN (consisting of G T and D T ), and an I2I model with a student generator G S , a student discriminator D S , a student encoder E S that extracts the content information from the input image x 1t , and a reference encoder E R that extracts the style information from the reference image x 2t .
Similarly to StarGANv2, we investigate two cases of I2I translation: the style representation is from noise (i.e., latent-guided synthesis ) and from the reference image (i.e., reference-guided synthesis ).
Here we mainly focus on reference-guided synthesis , since latent-guided synthesis can be adapted easily by replacing the reference encoder with several fully connection layers, which take the noise as input.
Both the teacher discriminator D T and the student encoder E S take the input image x 1t as input, extracting the hierarchical representation (cid:8) D T ( x 1t ) l (cid:9) and (cid:8) E S ( x 1t ) l (cid:9) .
Here D T ( x 1t ) l represents the output of the l th ResBlock 3 .
We encourage them to be aligned via the feature-based knowledge distillation loss:
[Equation]
(1) where parameters γ l are scalars to balance the terms.
We set them to 0.1.
When the teacher and student dimensions are not the same, an additional layer can be introduced to map the teacher output to the desired dimensions (similar as the hint distillation used in (Romero et al., 2015)).
Next, we take the image x 2t for both the pretrained discriminator D T and the reference encoder E R , extracting the hierarchical representation (cid:8) D T ( x 2t ) l (cid:9) and (cid:8) E R ( x 2t ) l (cid:9) , both of which we encourage to be aligned via the feature-based knowledge distillation loss
[Equation]
where parameters τ l are scalars which balance the terms.
We set them to 0.1.
Finally, taking the output of the student encoder E S ( x 1t ) and the output of the reference encoder E R ( x 2 t ) , for the student generator G S we obtain the hierarchical representation (cid:8)
G S ( E S ( x 1t ) , E R ( x 2t )) l (cid:9) and the output image ˆy s =
G S ( E S ( x 1t ) , E R ( x 2t )) , both of which we encourage to align with the hierarchical representation { G (cid:48) T (Φ( s 1t , s 2t )) l } .
We also align the output image y t =
G (cid:48) T (Φ( s 1t , s 2t )) with y t :
[Equation]
where parameters α l and β are scalars to balance the terms.
We set them to 0.1.
We also employ the following GAN loss (Goodfellow et al., 2014) to optimize this problem:
[Equation]
The full objective function of our model is:
[Equation]
where both λ adv and λ kdl are hyper-parameters to balance the importance of each terms.
We set them to 1.
In conclusion, to improve knowledge transfer for I2I, we have presented two novel contributions.
Firstly, we have shown that the pretrained StyleGAN can be used to generate an unlimited number of style-mixed triplets which we can use to perform the training.
We are the ﬁrst to exploit the stylemixing possibilities of StyleGAN to transfer knowledge to I2I systems.
Secondly, we have proposed a distillation approach to transfer knowledge from the GAN to the I2I model.
We name this datafree distillation because no real data is required and it is based on generated data.
our method can transfer knowledge between generators with different architectures.
3. D ATA - FREE KNOWLEDGE DISTILLATION FOR X2I TRANSLATION
Here we show how the proposed data-free knowledge distillation can be generalized to X2I translation problems.
We will perform the distillation based on a noise input to the student generator.
For X2I translation, the conditional information c determines the structural information and the semantic information.
For instance, in text2image translation on the birds dataset, we take as an input text descriptor (information c ), and expect to generate an image which mimics the shape and the appearance of a real bird.
However, during the knowledge transfer, the conditional information c is not available.
Therefore, we propose to use the teacher GAN discriminator to extract a semantic encoder of the image that can be used to replace the conditional information c .
However, when directly applying this for knowledge distillation, we found that this information can still be ignored by the network (see e.g. Figure 13).
We therefore introduce two additional techniques.
We again use the style-mixed triplets introduced in the previous section to perform the knowledge transfer and we propose a novel semantic diversity loss to diversify the semantic information of the output when varying the condition c .
As shown in Figure 2(c), we additionally introduce a semantic encoder S E to extract a pseudocondition vector c , which assists in distilling the knowledge from the unconditional GAN G T to the conditional GAN G S .
The semantic encoder S E is identical to the teacher discriminator after removing the last fully connection layer, and initialized by the teacher discriminator, which is welloptimized on the input images y t and x 1t .
Utilizing the style-mixed triplets (Sec. 3.1), we extract teacher output x 1t =
G (cid:48) T ( s 1t ) and y t =
G (cid:48) T (Φ( s 1t , s 2t )) , and the corresponding hierarchical representation (cid:8) G (cid:48) T ( s 1t ) l (cid:9) and (cid:8) G (cid:48) T (Φ( s 1t , s 2t ))
l (cid:9) , respectively.
Note that G (cid:48) T ( s 1t ) 1 and G (
cid:48) T (Φ( s 1t , s 2t ))are identical, since x 1t provides the structural information of y t .
Semantic encoder S E extracts the semantic attributes from the input images x 1t and y t respectively, termed as c x = S E ( x 1t ) and c y = S E ( y t ) respectively.
For the student generator G S , we take as input the teacher’s latent representation at the ﬁrst layer, the output the semantic encoder and the noise.
Note, the teacher’s latent representation 4 is summed with the corresponding feature of the student generator.
When the input of the semantic encoder
S E is x 1t , we generate the hierarchical representation (cid:8) G S ( z s , G (
cid:48) T ( s 1t ) 1 , c x ) l (cid:9) and ﬁnal student output ˆx 1s =
G S ( z s , G (cid:48) T ( s 1t ) 1 , c x ) .
Similarly, when y t is the input of semantic encoder S E ,
we
S ( s
G (cid:48) T (Φ( s
, s 2t ))
1 , c y )
G z s , G (cid:48) T (Φ( s 1t , s 2t )) 1 , c y ) .
By conditioning the student generator with the teacher representation, the student generator can output a similar image (this would otherwise be impossible).
The loss is deﬁned as:
[Equation]
where parameters α l and β are scalars to balance the terms.
Yang et al.
(2019) proposed a loss to promote diversity of the generated images with respect to changes of the input noise z .
Inspired by this work, to address the lack of diversity with respect to the conditional vector, we propose the semantic diversity loss is:
[Equation]
Other than (Yang et al., 2019), our loss promotes diversity with respect to changes of the conditional vectors (i.e., c x and c y ) while we retain the input noise vector z s .
Since the conditional vector inﬂuences the semantic information (e.g., gender, age and hair style), we call our loss the semantic diversity loss.
The full objective function of our model is:
[Equation]
(8) where λ adv , λ kdl and λ srl are hyper-parameters to balance the importance of each term, and L adv is deﬁned in Eq.
The proposed initialization procedure is not applicable to the case when the conditioning is a onehot spatial map (we found the semantic encoder was not able to predict these).
Therefore, for this case, we do not initialize the conditional branch and we set c are 1 in the student generator and 0 in the student discriminator during the distillation.
We found that even for these cases the knowledge transfer led to large performance gains (Table 2).
E XPERIMENTS
In Sec. 4.1, we ﬁrstly investigate knowledge transfer for I2I translation.
We explore how the proposed method affects image synthesis by using conditional information c (i.e., audio, text and semantic segmentation) in Sec. 4.2.
For the experimental evaluation, we apply our proposed knowledge transfer method explained in the previous section (i.e the ﬁrst stage) and use this distilled model for the downstream task where we ﬁnetuned it on the target data (i.e. the second stage)..
More detailed information and visualization results can be found in the Appendix A and C.
The code is provided in Supp.
Mat.
4. D ATA - FREE KNOWLEDGE DISTILLATION FOR I2I TRANSLATION
We ﬁrst evaluate our method for I2I translation with a reference image, named reference-guided I2I translation .
Then we verify the proposed method without reference image, called latent-guided I2I translation , in which we use random noise to control the style of the output image.
We take stateof-the-art StarGANv2 as the student network.
Except for the well known metrics: FID (Heusel et al., 2017) and KID (Bi´nkowski et al., 2018), we also train a real ( RC ) and a fake classiﬁer ( FC )
(Shmelkov et al., 2018) to evaluate the ability to generate class-speciﬁc images.
RC is trained on real data and evaluated on the generated data and vice versa for FC.
Reference-guided I2I translation.
We conduct multi-class I2I translation on three datasets: Animal faces (Liu et al., 2019), Birds (Van Horn et al., 2015) and Foods (Kawano & Yanai, 2014). The
[Figure] [Figure] [Table] Animal faces dataset contains 1,490 images and 149 categories in total, Birds has 48,527 images and 555 classes , Foods consists of 31,395 images and 256 classes.
We compare to StarGANv2 (Choi et al., 2020).
We also explore a wide variety of conﬁgurations for our approach, including: no knowledge distillation ( No KD ), knowledge distillation only using L1 distance between the teacher generator and the student generator ( KD (w-L1, w/o-adv.) ), knowledge distillation only using the adversarial loss ( KD (w/o-L1, w-adv.) ) and knowledge distillation with both losses ( KD (w-L1, w-adv.) ).
We also ablate the style-mixed triplets.
We replace both the reference image x 2t and the style-mixed output y t with the input image x 1t (indicated by KD* (w-L1, w-adv.) ). [Figure]
Knowledge distillation and style-mixed triplets.
Figure 4 presents a comparison between several variants of our method in terms of mean FID (mFID) on three datasets.
Taking Animal faces as an example, adding either L1 distance or the adversarial loss improves the I2I translation task in general compared to the I2I model trained from scratch (mFID: 134.7).
Finally combining both losses obtains the best score (mFID: 96.2), indicating that the proposed method largely improves I2I translation performance.
Without the style-mixed triplets, the model performance largely drops (mFID:104.1), even with both the L 1 distance and the adversarial loss.
This is probably because the generator fails to perform disentanglement, and the reference encoder does not learn the style information.
In conclusion, this experiment validates the effectiveness of the proposed style-mixed triplets.
Results.
Table 1 ( reference-guided I2I translation ) reports results for baseline and our method on
Animal faces , Birds and Foods datasets with image resolution 256 × 256 .
StarGANv2 training from scratch obtains lower results (e.g., mFID:134.7 on Animal faces dataset), indicating training I2I model from scratch is challenging when given limited data.
Our method achieves a signiﬁcant ad-
[Figure]
is made of white petals and a pistol that is yellow in color
[Figure]
This flower is white and pink in color, with petals that have small veins
This small bird has a red head and a curved beak.
A small bird with a red cheek and pointed beak.
[Figure] [Figure]
vantage (mFID: 96.2 on Animal faces dataset), demonstrating that the proposed knowledge transfer facilitates I2I translation.
Figure 3 compares the results of mapping the input to the target image domain with a reference image.
Given both the input and reference image our method manages to generate higher quality images than StarGANv2, that retain the pose of the input and keep a similar style as the reference image.
It clearly indicates that our method improves the image synthesis model.
Note that the existing approaches which address transfer learning for I2I (Wang et al., 2020b; 2021) cannot be applied to this case, since they require the I2I image synthesis architecture to be the same as the pretrained GAN architecture.
Latent-guided I2I translation.
To further evaluate the generality of our method for I2I translation, we validate the proposed method without reference image.
We compare to StarGANv2 (Choi et al., 2020), SDIT (Wang et al., 2019), DRIT++ (Lee et al., 2020b), DMIT (Yu et al., 2019),
DeepI2I (Wang et al., 2020b) and TransferI2I
(Wang et al., 2021).
We use random noise to control the style information of output images.
As reported in Table 1 ( latent-guided I2I translation ), all baselines training from scratch (from 4rd to 6th row) suffer a signiﬁcant disadvantage on small dataset.
StarGANv2 obtains the best result (131.2 mFID on Animal faces ).
However, using knowledge transfer TransferI2I performs signiﬁcantly better (103.5 mFID on Animal faces ), and achieves the best score on Foods dataset.
Finally, our method obtains the best results on both Animal faces and
Birds datasets, and competitive performance on the Foods dataset.
This could be due to the fact that the Foods dataset has more training data, and the importance of transfer learning is negligible.
We also evaluate our method with higher resolution images (256 × 256).
As shown in the last two rows of Table 1 ( latent-guided I2I translation ), we still retain a large advantage comparing to StarGANv2, demonstrating that our method still obtains better performance for higher resolution images.
Neither
DeepI2I nor TransferI2I is applicable, since there is no the released pretrained model for usage (they are based on the BigGAN).
The synthesized images on three datasets are in the Appendix C.
[Figure]
Eq. 7) from ﬁnal objective (Eq. 8).
4. D ATA - FREE KNOWLEDGE DISTILLATION FOR X2I TRANSLATION
Our approach can be applied to a wide variety of conditional image synthesis tasks.
Given the small target datasets, we perform image synthesis using conditional information c (i.e., text, audio and segmentation maps).
We explore four variants of our method: knowledge distillation only using L1 distance between the teacher generator and the student generator ( Ours (w-L1, w/o-adv.) ); knowledge distillation only using the adversarial loss ( Ours (w/o-L1, w-adv.) ); next we set the condition c
= 1 in the student generator and 0 in the student discriminator ( ours † ); ﬁnally we remove the semantic diversity loss in Eq.
7 from the ﬁnal objective in Eq.
8 ( ours ‡ ).
More details on training and hyperparameters can be found in Appendix A.
Text-to-image.
We evaluate the proposed method on the CUB bird dataset (Welinder et al., 2010).
It contains 8,855 training images (150 species) and 2,933 test images (50 species), following 10 text descriptions for each bird.
Here we use 10 images per class for training, and verify our method on the test dataset.
We compare with SSA-GAN (Hu et al., 2021) which is one of state-of-the-art methods.
Results are depicted in Figure 1, Figure 5 (a) and Table 2 (left).
Audio-to-image.
We evaluate on the commonly-used dataset: Oxford-102 (Nilsback & Zisserman, 2008), where 82 categories and 10 images per category are selected for training, and 20 categories and 1,155 images for test.
Note that we split the training and test categories following
S2IGAN (Wang et al., 2020a).
Results are depicted in Figure 1, Figure 5 (b) and Table 2 (middle).
Semantic segmentation-to-image.
Here we conduct experiments by conditioning on segmentation map on the CeleAMask-HQ (Lee et al., 2020a) dataset containing 30,000 image and segmentation mask pairs.
We follow the same train/test split as SEAN (Zhu et al., 2020b).
We randomly select 500 pairs of data from the train set for training, and 2,000 pairs for testing.
To evaluate our method, we compare to three baselines:
SPADE (Park et al., 2019), OASIS (Simonyan et al., 2013) and
CoCosNetv2 (Zhou et al., 2021), which are trained from scratch.
We report the generator performance using the pretrained StyleGAN optimized on AFHQ (Choi et al., 2020)(animal face dataset).
Results are depicted in Figure 1, Figure 5 (c) and Table 2 (right, w/o reference).
We also compare to SEAN (Zhu et al., 2020b) which additionally uses a reference image to control the style of the output image, the results are shown in Figure 5 (d) and Table 2 (right, w reference).
As reported in Table 2, our approach outperforms all baselines without using knowledge distillation on several metrics.
The semantic diversity loss gets the best performance except for the semanticmap-to-image task.
The qualitative results in Figure 1 and 5 indicate that our method can generate high-quality images with limited data due to the beneﬁt of the knowledge transferred from the expert teacher network.
Figure 6 shows the output images of the X2I generator after knowledge distillation and shows that we synthesize diverse outputs given different condition c .
In conclusion, our method is an efﬁcient, general purpose mechanism for conditional image synthesis with limited data.
C ONCLUSIONS
Conditional image synthesis tasks require vast amounts of training data.
This requirement can be alleviated by transfer learning.
Therefore, we proposed to transfer knowledge from a unconditional pretrained GAN model to conditional image synthesis tasks.
We perform the distillation with style-mixed triplets that can be automatically computed from high-quality GANs.
Our experiments conﬁrmed that the proposed transfer learning method obtains state-of-the-art results and allows to train of high-quality X2I systems from fewer labeled samples.
A A RCHITECTURE AND T RAINING D ETAILS
The proposed method is implemented in Pytorch (Paszke et al., 2017).
We transfer knowledge from
StyleGAN to X2I translation.
We introduce here the architecture and model details for each task.
Note we perform the knowledge distillation on one GPU(Quadro RTX6000) with 24GB VRAM.
A. K NOWLEDGE DISTILLATION FOR TEXT - TO - IMAGE TRANSLATION
For text-to-image translation the training is composed of two stages: data-free distillation from the unconditional prerained StyleGAN to text-to-image translation model, and ﬁnetuning text-to-image translation model using the target data and label.
Model details for the ﬁrst stage.
The student model is the same as SSA-GAN (Hu et al., 2021), which is composed of three sub-networks: a text encoder, a student generator G S , a student discriminator D S .
The text encoder is the pretrained one provided by (Xu et al., 2017).
The generator takes random noise as input, following one fully connection ( 100 × 8192 ) and 7 ResBlocks 5 .
The input feature size of the ﬁrst ResBlock is B × 512 × 4 × 4 ( B is the batch size).
We extract the hierarchical representation from StyleGAN (Karras et al., 2019).
The ﬁrst feature from StyleGAN is B × 512 × 8 × 8 , and fed into the student generator, which is summed with the student generator feature representation which has the same dimension.
The student discriminator contains one convolution layer and 6 ResBlocks.
We optimize the model using Adam (Kingma & Ba, 2014) with batch size of 16.
The learning rates of the generator and the discriminator are set as 0.0001 and 0.0004 with exponential decay rates of ( β 1 , β 2 ) =
The model is trained for 300 epochs for knowledge distillation.
In the second stage, we iterate 300 epochs, with same batch size, learning rate and exponential decay rate.
In Eq.both α l and β are identical.
For the speciﬁc features of which the dimension is less than 128, we set them 0.1.
In other case they are 0.01.
In Eq.
8 λ adv and λ kdl are 1, and λ srl is 0.1.
Model details for the second stage.
We ﬁnetune the well-initialized student network learned in the ﬁrst stage.
In this stage, we get access to the target data.
We keep all the details on training and the hyperparameters of SSA-GAN (Hu et al., 2021).
We refer readers for more detailed information to (Hu et al., 2021).
A. K NOWLEDGE DISTILLATION FOR AUDIO - TO - IMAGE TRANSLATION
For audio-to-image translation the training is composed of two stages: data-free distillation from the unconditional prerained StyleGAN to audio-to-image translation model, and ﬁnetuning audioto-image translation model using the target data and label.
Model details for the ﬁrst stage.
We perform knowledge distillation for audio-to-image based on
S2IGAN (Wang et al., 2020a).
To obtain stable training, S2IGAN proposed a progressive mechanism to generate multi-scale images.
S2IGAN introduced a two-stage study, i.e., a speech semantic embedding stage and an image generation stage.
The former is to extract the speech embedding with a pretrained speech extractor, the latter takes as an input both random noise and the speech embedding to synthesize photo-realistic images in a multi-stage (coarse-to-ﬁne) way.
The generator is composed of one fully connected layer ( 228 × 2048 ), one batchnorm, one Gaussian Error Linear
Unit(GLU) (Devlin et al., 2019) and 7 UpsBlocks 6 .
The ﬁrst UpsBlock is B × 512 × 4 × 4 ( B is the batch size).
We extract the hierarchical representation from StyleGAN (Karras et al., 2019).
The ﬁrst feature from StyleGAN is B × 512 × 8 × 8 , and fed into the student generator, which is summed with the student generator feature representation which has the same dimension.
We optimize the model using Adam (Kingma & Ba, 2014) with batch size of 12.
The learning rates of both the generator and the discriminator are set as 0.0002 with exponential decay rates of ( β 1 , β 2 ) =
The model is trained for 300 epochs for knowledge distillation.
In the second stage, we iterate 300 epochs, with same batch size, learning rate and exponential decay rate.
In Eq.
6 both α l and β are identical, and are 10.
In Eq.
8 λ adv , λ kdl and λ srl are 10.
Model details for the second stage
We ﬁnetune the well-initialized student network learned in the ﬁrst stage.
In this stage, we get access to the target data.
We keep all the details on training and the hyperparameters of S2IGAN (Wang et al., 2020a).
We refer readers to check more detailed information to (Wang et al., 2020a).
A. K NOWLEDGE DISTILLATION FOR SEGMENTATION MAP - TO - IMAGE TRANSLATION
For segmentation map-to-image translation the training is composed of two stages: data-free distillation from the unconditional prerained StyleGAN to segmentation map-to-image translation model, and ﬁnetuning segmentation map-to-image translation model using the target data and label.
Model details for the ﬁrst stage.
For segmentation map-to-image task, we use SEAN (Zhu et al., 2020b) as the student model, since it is state-of-the-art.
SEAN is composed of one generator, one style encoder and one discriminator.
The generator contains several SEAN ResBlks.
Each of the ResBlks is followed by a nearest neighbor upsampling layer.
We extract the hierarchical representation from StyleGAN (Karras et al., 2019).
The ﬁrst feature from StyleGAN is B × 512 × 8 × 8 .
In the knowledge distillation stage, we set the learning rates to 0 . 0001 and 0 . 0004 for the generator and discriminator, respectively (Heusel et al., 2017).
For the optimizer, we choose Adam (Kingma & Ba, 2014) with β 1 = 0 , β 2 = 0 .
999 , with batch size 4.
In second stage (segmentation map-toimage translation), we use same setting except for batch size (i.e., 8).
We update 100 epochs for knowledge distillation.
In second stage (image-to-image translation) we iterate 100 epochs.
In Eq.both α l and β are identical.
For the speciﬁc features of which the dimension is less than 128, we set them 0.1.
In other case they are 0.01.
In Eq.
8 λ adv and λ kdl are 1, and λ srl is 0.1.
Model details for the second stage.
We ﬁnetune the well-initialized student network learned in the ﬁrst stage.
In this stage, we get access to the target data.
We keep all the details on training and the hyperparameters of SEAN (Zhu et al., 2020b).
We refer readers to check all detailed information in (Zhu et al., 2020b).
A. K NOWLEDGE DISTILLATION FOR IMAGE - TO - IMAGE TRANSLATION
Model details.
We take StarGANv2 (Choi et al., 2020) as the student model.
StarGANv2 contains 5 sub-networks: a context encoder, a mapping network, a style encoder, a generator and a discriminator.
For both content encoder and generator, we extract hierarchical representations with dimension size range from 16 to 256.
We also extract the corresponding dimension hierarchical representations from the pretrained generator and discriminator respectively.
To align the student style encoder with the teacher discriminator, we extract hierarchical representations for both networks with the dimension size ranging from 4 to 256.
We use the Adam (Kingma & Ba, 2014) with β 1 = 0 , β 2 = 0 .
99 , with batch size 2.
We update 20,000 iterations for knowledge distillation.
In second stage (image-to-image translation), we iterate 100,000.
[Table] [Table]
Model details for the second stage.
We ﬁnetune the well-initialized student network learned in the ﬁrst stage.
In this stage, we get access to the target data.
We keep all the details on training and the hyperparameters of StarGANv2 (Choi et al., 2020).
We refer readers to check all detail information in (Choi et al., 2020).
We are able to explicitly control domains/classes as starGANv2 does.
For the latent-guided synthesis, starGANv2 introduces class-speciﬁc mapping networks (one for each class) to project the class embedding into the shared latent space.
Thus starGANv2 uses the class-speciﬁc mapping networks to control the domains/classes.
In this paper, we follow the same setup as starGANv2.
In the ﬁrst stage of our method, we only learn one mapping network, which is duplicated to initialize all the mapping networks (one for each class) of the second stage.
Then we use the target dataset (second stage) to train the well-initialized class-speciﬁc mapping networks.
B I2I TRANSLATION B. T WO - CLASS I2I TRANSLATION RESULTS To
evaluate the generality of our method, here we validate the proposed algorithm for two-class
I2I translation on a two-category dataset: cat2dog-200 (Wang et al., 2021) with an image size of 256 × 256 .
In cat2dog-200, the training set is composed of 200 images (100 images/per class) and the test set has 200 images (100 images/per class).
We investigate two cases of I2I translation: the style representation is from noise (i.e., latent-guided synthesis) and from the reference image (i.e., reference-guided synthesis).
As reported Tables 3 and 4, compared to the transfer learning methods (i.e., DeepI2I (Wang et al., 2020b) and TransferI2I
(Wang et al., 2021))
we still maintain a large advantage.
B. M ULTI - CLASS I2I TRANSLATION RESULTS
We also further validate the proposed algorithm for Multi-class I2I translation on AFHQ-500 (Choi et al., 2020) with an image size of 256 × 256 .
In AFHQ-500, the training set is composed of 500 images (100 images/per class) and the test set has 1500 images (500 images/per class).
We investigate two cases of I2I translation: the style representation is from noise (i.e., latent-guided synthesis) and from the reference image (i.e., reference-guided synthesis).
As reported Tables 5, compared to StarGANv2 we still maintain a large advantage.
B. V ARIANTS OF THE BASELINES
We adapt both DeepI2I (Wang et al., 2020b) and TransferI2I (Wang et al., 2021) to use a very similar architecture to StarGANv2 (the architectural details are in Table 5, 6, 7 of StarGANv2 (Choi et al., 2020)).
Specially, to devise the generator, we preserve the subnets of the discriminator of the pretrained StyleGAN as image encoder which has 16 × 16 × 512 output, then additionally add
ResBlks like StarGANv2, and ﬁnally use the subnets of the generator of the pretrained StyleGAN as the decoder to output the image.
We keep the mapping network of the pretrained StyleGAN for the
I2I translation generator when performing latent-guided I2I translation.
We adapt the discriminator
[Table]
[Table]
of the pretrained StyleGAN
as the style encoder when performing reference-guided I2I translation.
Similar to StarGANv2, we also use a multitask discriminator, which consists of multiple output branches.
We use cat2dog-200 dataset to compare the adapted DeepI2I (DeepI2I*), adapted TransferI2I (TransferI2I* ) and ours.
As reported in Tables 6 and 7, compared to the transfer learning methods (i.e., Adapting DeepI2I and TransferI2I )
we still maintain a large advantage.
B. A BLATION OF THE ENCODER
In this paper we use the pre-deﬁned discriminator to initialize the encoder, since the discriminator of the StyleGAN is trained on HHFQ with 70k images, which optimizes it to be an effective feature extractor.
The similar technique was also explored in SGD (Shocher et al., 2020) and transferI2I.
To verify that the previously reported results also hold for our method, we also train both the encoder and the reference encoder from scratch in reference-guid translation on Animal faces dataset (10/per class).
we achieve 130 of mFID, which is lower than our method (mFID 96.2) .
If we train the encoder from scratch, the training suffers from overﬁtting with limited training images.
C A DDITIONAL R ESULTS
[Figure]
C. U SER STUDY
We conduct a user study and ask subjects to select results that are more realistic given the target label, and have the same pose as the input image .
We apply pairwise comparisons (forced choice) with 20 users (100 image pairs/user) for I2I translation.
Experiments are performed on images from the Animal faces dataset.
Fig.
7 shows that our method considerably outperforms the other methods.
C. D IVERSE OUTPUTS
Figure 8 shows some examples of the synthesized images on three datasets.
Taking Animal faces as example, given the target class label our method manages to generate high visual quality images.
C. K NOWLEDGE DISTILLATION FOR I2I SYSTEM IN THE FIRST STAGE
Figures 9 and 10 show examples generated after transfer learning on I2I system.
This shows that our method successfully distills the style-mixing characteristics from the teacher generator to the student generator. [Table] [Table] [Figure]
C. K NOWLEDGE DISTILLATION FOR X2I SYSTEM IN THE FIRST STAGE
Figures 11 and 12 show examples generated after semantic diversity loss.
For each triplet, we depict the teacher’s output(left), the student’s output (middle) with condition c x and the one (right) with condition c y .
This shows that our semantic diversity loss manages to diversify the output given the input condition c .
C. A DDITIONAL RESULTS
We additionally depict the generated images in Figures 14- 26.
[Figure]
[Figure] [Figure]
[Figure]
O u t p u t ( t e a c h e r )
O u t p u t ( s t u d e n t , ) O u t p u t ( s t u d e n t , )
O u t p u t ( t e a c h e r )
O u t p u t ( s t u d e n t , ) O u t p u t ( s t u d e n t , )
Input
SSA-GAN
This bird is red and black in color with a black beak and red eye rings
This bird has a red breast, with a red and white belly
This yellow bird's wings appear black and white in color
A small bird with a short, ﬂ at beak and a vibrant blue body
A black bird with a shiny green breast.
This bird has wings that are grey and has a white belly
The bird has a long curved black bill and a tan belly
This bird has wings that are black and has a yellow belly
Ours
GT
[Figure]
The petals on this flower are purple with yellow
[Figure]
A flower with bright red petals and stripes of black
This flower has petals that are yellow and has black lines
The flower shown has thin yellow petals as its main feature
This flower has petals that are white with yellow patch
This flower has light pink petals and a bell shape
The petals are pink and rounded and cover the pistil
This flower has pointy long yellow petals and one sepal
[Figure]
[Figure] [Figure]
I n p u t
R e f e .
O u t p u t
R e c o .
I n p u t
R e f e .
O u t p u t
R e c o .
I n p u t
R e f e .
O u t p u t
R e c o .
[Figure]
I n p u t
R e f e .
O u t p u t
S t a r
G A N v
O u t p u t
O u r s
I n p u t
R e f e .
O u t p u t
S t a r
G A N v
O u t p u t
O u r s
I n p u t
R e f e .
O u t p u t
S t a r
G A N v
O u t p u t
O
[Figure]
r s
I n p u t
R e f e .
O u t p u t
R e c o .
I n p u t
R e f e .
O u t p u t
R e c o .
I n p u t
R e f e .
O u t p u t
R e c o .
I n p u t
R e f e .
O u t p u t
S t a r
G A N v
O u t p u t
O u r s
I n p u t
R e f e .
O u t p u t
S t a r
G A N v
O u t p u t
O u r s
I n p u t
R e f e .
O u t p u t
S t a r
G A N v
O u t p u t
O u r s
I n p u t
R e f e .
O u t p u t
R e c o .
I n p u t
R e f e .
O u t p u t
R e c o .
I n p u t
R e f e .
O u t p u t
R e c o .
I n p u t
R e f e .
O u t p u t
S t a r
G A N v
O u t p u t
O u r s
I n p u t
R e f e .
O u t p u t
S t a r
G A N v
O u t p u t
O u r s
I n p u t
R e f e .
O u t p u t
S t a r
G A N v
O u t p u t
O u r s