W HEN , W HY , AND W HICH P RETRAINED GAN S A RE U SEFUL ?
Anonymous authors
Paper under double-blind review
A BSTRACT
The literature has proposed several methods to finetune pretrained GANs on new datasets, which typically results in higher performance compared to training from scratch, especially in the limited-data regime.
However, despite the apparent empirical benefits of GAN pretraining, its inner mechanisms were not analyzed indepth, and understanding of its role is not entirely clear.
Moreover, the essential practical details, e.g., selecting a proper pretrained GAN checkpoint, currently do not have rigorous grounding and are typically determined by trial and error.
This work aims to dissect the process of GAN finetuning.
First, we show that initializing the GAN training process by a pretrained checkpoint primarily affects the model’s coverage rather than the fidelity of individual samples.
Second, we explicitly describe how pretrained generators and discriminators contribute to the finetuning process and explain the previous evidence on the importance of pretraining both of them.
Finally, as an immediate practical benefit of our analysis, we describe a simple recipe to choose an appropriate GAN checkpoint that is the most suitable for finetuning to a particular target task.
Importantly, for most of the target tasks, Imagenet-pretrained GAN, despite having poor visual quality, appears to be an excellent starting point for finetuning, resembling the typical pretraining scenario of discriminative computer vision models.
I NTRODUCTION
These days, generative adversarial networks (GANs) (Goodfellow et al., 2014) can successfully approximate the high-dimensional distributions of real images.
The exceptional quality of the state-ofthe-art GANs (Karras et al., 2020b; Brock et al., 2019) makes them a key ingredient in applications, including semantic editing (Isola et al., 2017; Zhu et al., 2018;
Shen et al., 2020; Voynov & Babenko, 2020), image processing (Pan et al., 2020; Ledig et al., 2017; Menon et al., 2020), video generation (Wang et al., 2018a), producing high-quality synthetics (Zhang et al., 2021; Voynov et al., 2020).
To extend the success of GANs to the limited-data regime, it is common to use pretraining, i.e., to initialize the optimization process by the GAN checkpoint pretrained on some large dataset.
A line of works (Wang et al., 2018b; Noguchi & Harada, 2019; Zhao et al., 2020; Mo et al., 2020; Wang et al., 2020; Li et al., 2020) investigate different methods to transfer GANs to new datasets and report significant advantages compared to training from scratch both in terms of generative quality and convergence speed.
However, the empirical success of GAN pretraining was not investigated in-depth, and its reasons are not entirely understood.
From the practical standpoint, it is unclear how to choose a proper pretrained checkpoint or if one should initialize both generator and discriminator or only one of them.
To the best of our knowledge, the only work that systematically studies the benefits of pretraining is Wang et al. (2018b).
However, the experiments in (Wang et al., 2018b) were performed with currently outdated models, and we observed that some conclusions from Wang et al. (2018b) are not confirmed for modern architectures like StyleGAN2 (Karras et al., 2020b).
In particular, unlike the prior results, it appears that for state-of-the-art GANs, it is beneficial to transfer from sparse and diverse rather than dense and less diverse sources.
In this work, we thoroughly investigate the process of GAN finetuning.
First, we demonstrate that starting the GAN training from the pretrained checkpoint can significantly influence the diversity of the finetuned model, while the fidelity of individual samples is less affected.
Second, we dissect the mechanisms of how pretrained generators and discriminators contribute to the higher coverage of finetuned GANs.
In a nutshell, we show that a proper pretrained generator produces samples in the neighborhood of many modes of the target distribution, while a proper pretrained discriminator serves as a gradient field that guides the samples to the closest mode, which together result in a smaller risk of mode missing.
This result explains the evidence from the literature that it is beneficial to initialize both generator and discriminator when finetuning GANs.
Finally, we investigate different ways to choose a suitable pretrained GAN checkpoint for a given target dataset.
Interestingly, for most of the tasks, Imagenet-pretrained models appear to be the optimal initializers, which mirrors the pretraining of discriminative models, where Imagenet-based initialization is de-facto standard (Donahue et al., 2014;
Long et al., 2015;
He et al., 2020; Chen et al., 2020a).
Our conclusions are confirmed by experiments with the state-of-the-art StyleGAN2 (Karras et al., 2020b), chosen due to its practical importance and a variety of open-sourced checkpoints, which can be used as pretrained sources.
The main contributions of our analysis are the following:
We show that initializing the GAN training by the pretrained checkpoint can significantly affect the coverage and has much less influence on the realism of individual samples.
We explain why it is important to initialize both generator and discriminator by describing their roles in the finetuning process.
We describe a simple automatic approach to choose a pretrained checkpoint that is the most suitable for a given generation task.
A NALYSIS
This section aims to explain the success of the GAN finetuning process compared to training from scratch.
First, we formulate the understanding of this process speculatively and then confirm this understanding by experiments on synthetic data and real images.
2. H IGH - LEVEL INTUITION
Let us consider a pretrained generator G and discriminator D that are used to initialize the GAN training on a new data from a distribution p target .
Throughout the paper, we show that a discriminator initialization is “responsible for” an initial gradient field, and a generator initialization is “responsible for” a target data modes coverage.
Figure 1 illustrates the overall idea with different initialization patterns.
Intuitively, the proper discriminator initialization guarantees that generated samples will move towards “correct” data regions.
On the other hand, the proper pretrained generator guarantees that the samples will be sufficiently diverse at the initialization, and once guided by this vector field, they will cover all target distribution.
Below, we confirm the validity of this intuition.
−
[Figure]
−
[Figure]
−
[Figure]
−
[Figure]
− 20 − 10 0 10 20 30 −− 20 −
Source-I Generated −−−− −−
Source-II Generated −−
20 −−   3.
Source-Iinit −−
−−   16.
Source-IIinit − 20 −  20 −−−    12.
Randominit
2. S YNTHETIC E XPERIMENT
We start by considering the simplest synthetic data presented on Figure 2.
Our goal is to train a GAN on the target distribution, a mixture of ten Gaussians arranged in a circle.
We explore three options to initialize the GAN training process.
First, we start from random initialization.
The second and the third options initialize training by GANs pretrained on the two different source distributions.
The first source distribution corresponds to a wide ring around the target points, having high coverage and low precision w.r.t.
target data.
The second source distribution is formed by three Gaussians that share their centers with three target ones but have a slightly higher variance.
This source distribution has high precision and relatively low coverage w.r.t.
target data.
Then we train two source GANs from scratch to fit the first and the second source distributions and employ these checkpoints to initialize GAN training on the target data.
The results of GAN training for the three options are presented on Figure 2, which shows the advantage of pretraining from the more diverse model, which results in a higher number of covered modes.
The details of the generation of the synthetic are provided in the appendix.
Dissecting the contributions from G and D .
Here, we continue with the synthetic example from above and take a closer look at the roles that the pretrained generator and discriminator play when finetuning GANs.
Our goal is to highlight the importance of ( 1 ) the initial coverage of the target distribution by the pretrained generator and ( 2 ) the quality of the gradient field from the pretrained discriminator.
We quantify the former by the established recall measure (Kynk¨a¨anniemi et al., 2019) computed in the two-dimensional dataspace with k =5 for 1000 randomly picked samples from the target distribution and the same number of samples produced by the pretrained generator.
To evaluate the quality of the discriminator gradient field, we use a protocol described in (Sinha et al., 2020).
Namely, we assume that the “golden” groundtruth gradients would guide each sample towards the closest Gaussian center from the target distribution.
Then we compute the similarity between the vector field ∇ x D provided by the pretrained discriminator and the vector field of “golden” gradients.
Specifically, we evaluate the cosine similarity between these vector fields, computed for the generated samples.
Given these two measures, we consider a series of different starting generator/discriminator checkpoints ( G i , D i ) , i = 1 , . . .
, N .
The details on the choice of the starting checkpoints are provided in the appendix.
Then we use each pair ( G i , D i ) as initialization of GAN training on the target distribution of ten Gaussians described above.
Additionally, for all starting G i /D
i , we evaluate the recall and the discriminator gradients field similarity to the “golden” gradients.
The overall quality of GAN finetuning is measured as the Wasserstein-1 distance between the target distribution and the distribution produced by the finetuned generator.
The scatter plots of recall, the similarity of gradient fields, and Wasserstein-1 distance are provided in Figure 3.
As can be seen, both the recall and gradient similarity have significant negative correlations with the W 1 -distance between the ground-truth distribution and the distribution of the finetuned GAN.
Furthermore, for the same level of recall, the higher values of the gradient similarity correspond to lower Wasserstein distances.
Alternatively, for the same value of gradient similarity, higher recall of the source generator typically corresponds to the lower Wasserstein distance.
We also note that the role of the pretrained generator is more important since, for high recall values, the influence from the discriminator is not significant (see Figure 3, left).
− 1.0 − 0.8 − 0.6 − 0.4 − 0.2 0.0 0.
∇ D similarity 0.
R e c a ll − 1.0 − 0.8 − 0.6 − 0.4 − 0.2 0.0 0.
∇ D similarity
W 1 - d i s t a n c e 0.
1.0 Recall
W 1 - d i s t a n c e W 1 - d i s t a n c e
This synthetic experiment does not rigorously prove the existence of a causal relationship between the recall or gradient similarity and the quality of the finetuned GANs since it demonstrates only correlations of them.
However, in the experimental section, we show that these correlations can be successfully exploited to choose the optimal pretraining checkpoint, even for the state-of-the-art
GAN architectures.
L ARGE - SCALE EXPERIMENTS 3. E XPLORING PRETRAINING FOR S TYLE GAN
In this section, we confirm the conclusions from the previous sections experimentally with the stateof-the-art StyleGAN2 architecture (Karras et al., 2020b).
If not stated otherwise, we always work with the image resolution 256 × 256 .
Datasets.
We work with six standard datasets established in the GAN literature.
We also include two datasets of satellite images to investigate the pretraining behavior beyond the domain of natural images.
As potential pretrained sources, we use the StyleGAN2 models trained on these datasets.
Table 1 reports the list of datasets and the FID values (Heusel et al., 2017) of the source checkpoints.
We also experimented with four smaller datasets to verify our conclusions in the medium-shot and few-shot regimes.
The details on the datasets are provided in the appendix.
Experimental setup.
Here, we describe the details of our experimental protocol for both the pretraining of the source checkpoints and the subsequent training on the target datasets.
We always use the official PyTorch implementation of StyleGAN2-ADA (Karras et al., 2020a) provided by the authors 1 .
We use the “stylegan2” configuration in the ADA implementation with the default hyperparameters (same for all datasets).
Training is performed on eight Tesla V100 GPUs and takes approximately three hours per 1M real images shown to the discriminator.
Pretraining of source checkpoints.
We pretrain one checkpoint on the Imagenet for 50M real images shown to the discriminator and seven checkpoints on other source datasets from Tablefor 25M images.
A larger number of optimization steps for the Imagenet is used since this dataset is more challenging and requires more training epochs to converge.
For the large LSUN datasets (Cat, Dog, Bedroom), we use 10 6 first images to preserve memory.
For Satellite-Landscapes, we use ADA due to its smaller size.
Then, we always use checkpoints with the best FID for further transferring to target datasets for each source dataset.
Training on target datasets.
For each source checkpoint, we perform transfer learning to all datasets from Table 1.
We use the default transfer learning settings from the StyleGAN2-ADA implementation (faster adaptive data augmentation (ADA) adjustment rate, if applicable, and no
G ema warmup).
ADA is disabled for the datasets containing more than 50K images and enabled for others with default hyperparameters.
In these experiments, we train for 25M real images shown to the discriminator.
Each transfer experiment is performed with three independent runs, and the metrics are reported for the run corresponding to the median best FID (Heusel et al., 2017). [Table]
Metrics.
In the experiments, we evaluate the performance via the four following metrics.
(1) Frechet Inception Distance (FID) (Heusel et al., 2017), which quantifies the discrepancy between the distributions of real and fake images, represented by deep embeddings.
Both distributions are approximated by Gaussians, and the Wasserstein distance between them is computed.
(2) Precision (Kynk¨a¨anniemi et al., 2019), which measures the realism of fake images, assuming that the visual quality of a particular fake is high if it belongs to the neighborhood of some real images in the embedding space.
(3) Recall (Kynk¨a¨anniemi et al., 2019), which quantifies GAN diversity, measuring the rate of real images that belong to the neighborhood of some fake images in the embedding space.
(4) Convergence rate equals a number of real images that were shown to the discriminator at the moment when the generator FID for the first time exceeded the optimal FID by at most 5% .
Intuitively, this metric quantifies how fast the learning process reaches a plateau.
FID is computed based on the image embeddings extracted by the InceptionV3 model 2 .
Precision and Recall use the embeddings provided by the VGG-16 model 3 .
Precision and Recall are always computed with k =5 neighbors.
For FID calculation, we always use all real images and 50.000 generated samples.
For Precision/Recall calculation, we use the first 200K real images (or less, if the real dataset is smaller) and the same amount of generated samples.
Results.
The metric values for all datasets are reported in Table 2, where each cell corresponds to a particular source-target pair.
For the best (in terms of FID) checkpoint obtained for each sourcetarget transfer, we report the FID value (top row in each cell), Precision and Recall (the second and the third rows in each cell), and the convergence rate measured in millions of images (bottom row in each cell).
We highlight the sources that provide the best FID for each target dataset or differ from the best one by at most 5% .
We additionally present the curves of FID, Precision, and Recall values for several target datasets on Figure 9 and Figure 10 in the appendix.
We describe the key observations from Table 2 below:
• In terms of FID, a pretraining based on a diverse source (e.g., Imagenet or LSUN Dog) is superior to training from scratch on all datasets in our experiments.
• The choice of the source checkpoint significantly influences the coverage of the finetuned model, and the Recall values vary considerably for different sources, especially for smaller target datasets.
For instance, on the Flowers dataset, their variability exceeds ten percent.
In contrast, the Precision values are less affected by pretraining, and their typical variability is about 2 − 3% .
Figure 4 reports the standard deviations of Precision/Recall computed over different sources and highlights that Recall has higher variability compared to Precision, despite the latter having higher absolute values.
Pretraining considerably speeds up the optimization compared to the training from scratch
[Table]
C
I F
A RFF H Q
F l o w e r s
G r u m p y
C a t
L .
B e d r oo m
L .
C a t
L .
C
h u r c h
L .
D o g
S .
B u il d i n g s
S .
L a n d s c a p e s
S i m p s o n s
[Figure]
Overall, despite having poor quality (FID =49 . 2 ), the Imagenet-pretrained unconditional StyleGAN model appears to be a superior GAN initialization that typically leads to more efficient optimization compared to alternatives.
This result contradicts the observations in (Wang et al., 2018b) showing that it is beneficial to transfer from dense and less diverse sources rather than sparse and diverse ones, like Imagenet.
We attribute this inconsistency to the fact that (Wang et al., 2018b) experimented with the WGAN-GP models, which are significantly inferior to the current state-of-the-art ones.
3. A NALYSIS
In this section, we perform several additional experiments that illustrate the benefits of pretraining.
Pretraining improves the mode coverage for real data.
Here, we consider the Flowers dataset and assume that each of its 102 labeled classes corresponds to a different distribution mode.
To assign the generator samples to the closest mode, we train a 102 -way flowers classifier via finetuning the linear head of the Imagenet-pretrained ResNet-50 on real labeled images from Flowers.
Then we apply this classifier to generated images from eleven consecutive generator snapshots from the GAN training process on the interval from 0 to 200 kimgs taken every 20 kimgs.
This pipeline allows for tracking the number of covered and missed modes during the training process.
Figure 5 (left) demonstrates how the number of covered modes changes when GAN is trained from scratch or the checkpoints pretrained on FFHQ and Imagenet.
In this experiment, we consider a mode being “covered” if it contains at least ten samples from the generated dataset of size 10 000.
One can see that Imagenet, being the most diverse source, initially covers more modes of the target data and faster discovers the others.
FFHQ also provides coverage improvement but misses more modes compared to Imagenet even after training for 200 kimgs.
With random initialization, the training process covers only a third of modes after training for 200 kimgs .
On the right of Figure 5, we show samples drawn for the mode, which is poorly covered by the GAN trained from FFHQ initialization and well-covered by its Imagenet counterpart.
Pretraining provides more gradual image evolution.
The observations above imply that it is beneficial to initialize training by the checkpoint with a higher recall so that the target data is originally better “covered” by the source model.
We conjecture that transferring from a model with higher recall makes it easier to cover separate modes in the target distribution since, in this case, generated samples can slowly drift to the closest samples of the target domain without abrupt changes to cover previously missing modes.
To validate this intuition, we consider a fixed batch of 64 random latent codes z and a sequence of the generator states G 1 , . .
. , G N obtained during the training.
Then we quantify the difference between consecutive images computed as the perceptual LPIPS distance
Zhang et al.
LPIPS( G i ( z ) , G i +1 ( z )) .
Figure 6 shows the dynamics of the distances for
Flowers as the target dataset and Imagenet, FFHQ, and random initializations.
Since the Imagenet source initially has higher coverage of the target data, its samples need to transform less, which results in higher performance and faster convergence.
Figure 6 indicates more gradual sample evolution when GAN training starts from a pretraining checkpoint.
Here we additionally report the distributions of samples’ trajectories’ lengths quantified by LPIPS.
Namely, for a fixed z and a sequence of the generator snapshots G 1 , . .
. , G N obtained during training, we calculate the length of the trajectory as a sum (cid:80) i LPIPS( G i ( z ) , G i +1 ( z )) .
Figure 7 (left) presents the length distributions for three initializations and Flowers as the target dataset. [Figure]
[Figure] Finally, to track the dynamics of mode coverage of the target dataset, we obtain the class assignments of the generated samples G 1 ( z ) , . . .
, G N ( z ) with a classifier pretrained on the dataset.
Then for the samples G i ( z ) , G i +1 ( z ) generated with the consequent checkpoints, we calculate the probability that the sample changes its class assignment by averaging over 256 latent codes. That is, we evaluate the probabbility that a flower class of a sample G i ( z ) differs from a class of a sample G i +1 ( z ) .
The probabilities of the class change for different source checkpoints are presented in Figure 7, right.
Importantly, training from pretrained sources demonstrates higher class persistence of individual samples.
This indicates that the Imagenet-pretrained generator initially covers the target dataset well enough and requires fewer mode-changing sample hops during training.
Pretraining is beneficial for downstream tasks.
Here, we focus on the task of inverting a real image given a pretrained generator, which is necessary for semantic editing.
We employ the recent
GAN inversion approach (Tov et al., 2021) and train the encoders that map real images into the latent space of generators approximating FFHQ and Bedroom distributions.
For both FFHQ and Bedroom, we consider the generators that were trained from scratch and the Imagenet-based pretraining.
Table 3 reports the reconstruction errors quantified by the LPIPS measure and F -error .
The details of the metrics computation are provided in the appendix.
Overall, Table 3 confirms that higher GAN recall provided by pretraining allows for the more accurate inversion of real images.
C HOOSING PROPER PRETRAINED CHECKPOINT
This section describes a simple recipe to select the most appropriate pretrained checkpoint to initialize GAN training for a particular target dataset.
To this end, we consider a set of natural proxy metrics that quantify the similarity between two distributions.
Each metric is computed in two regimes.
[Figure] GAN Model (source, target) Inverted Domain Best FID
LPIPS
[Table]
error
F
[Table]
error s
[Table]
[Table]
CelebA
[Table]
[Table]
[Table]
CelebA
[Table]
[Table]
[Table]
FFHQ
[Table]
[Table]
[Table]
FFHQ
[Table]
[Table]
Bedroom
[Table]
Bedroom 2.
[Table]
[Table]
.Bedroom
[Table] .Bedroom 2. In the first regime, we measure the distance between the source dataset, consisting of real images used to pretrain the GAN checkpoint, and the target dataset of real images. In the second regime, we use the generated images from pretrained checkpoints instead of the source dataset.
The second regime is more practical since it does not require the source dataset.
As natural proxy metrics, we consider FID , KID (Bi´nkowski et al., 2018), Precision , and Recall measures. Regime/Metric
[Table]
To estimate the reliability of each metric, we calculate the number of target datasets for which this metric does not correctly predict the optimal starting checkpoint.
We consider a starting checkpoint optimal if it provides the lowest FID score or its FID score differs from the lowest by most 5%.
The quality for all metrics is presented in Table 4, which shows that
FID or Recall can be used as a rough guide to select a pretrained source in both regimes.
On the other hand, Precision is entirely unreliable.
This observation is consistent with our findings from Section 2 that imply that Recall can serve as a predictive measure of finetuning quality.
C ONCLUSION
Transferring pretrained models to new datasets and tasks is a workhorse of modern ML.
In this paper, we investigate its success in the context of GAN finetuning.
First, we demonstrate that transfer from pretrained checkpoints can improve the model coverage, which is crucial for GANs exhibiting modeseeking behavior.
Second, we explain that it is beneficial to the usage of both pretrained generators and discriminators for optimal finetuning performance.
This implies that the GAN studies should open-source discriminator checkpoints as well rather than the generators only.
Finally, we show that the recall measure can guide the choice of a checkpoint for transfer and highlight the advantages of Imagenet-based pretraining, which is not currently common in the GAN community.
We opensource the StyleGAN2 checkpoints pretrained on the Imagenet of different resolutions for reuse in future research.
A PPENDIX D ATASETS
Here we provide the details for the used datasets.
Table 5 reports the size, original image resolution (which was always resized to 256 × 256 in our experiments), number of samples used for training, and URL for each of the datasets.
In Tables 6, 7, 8, 9 we report pairwise distances between source and target datasets for different metrics.
Figure 8 illustrates samples from each dataset. [Table] [Table]
https
[Table]
[Table]
[Table]
[Table]
L.Bdr 0.
L.Cat 0.
L.Chr 0.
L.Dog 0.
S.Bld 0.
S.Lnd 0.
C0.
Flw 0.
[Table]
S 0.
L EARNING CURVES
On Figure 9 and Figure 10 we present the learning curves from Table 2 in the main text.
To make the plots readable, for each target dataset, we report only the curves corresponding to training from scratch, training from the Imagenet checkpoint, and from two checkpoints that perform best among the rest as a representative subset of sources.
S YNTHETIC DATA DETAILS
Here we provide the details for the experiment described in Section 2.2.
The synthetic target data is formed by 10 Gaussians with centers on the circle of radius 20 and σ = 0 .
Source-I (blue) is a distribution formed as a sum of a uniform distribution on a zero-centered circle of a radius 20 and the zero-centered Gaussians with σ = 4 .
Source-II (green) is formed by 3 Gaussians with centers that coincide with the consequent centers of three Gaussians of the original data and σ = 0 .
We use the standard GAN loss (Goodfellow et al., 2014) and perform 5000 generator training steps with 4 discriminator steps for every generator step.
We use batch size 64 and Adam optimizers with learning rate 0 . 0002 and β 1 , β 2 = 0 .
The generator has a 64 -dimensional latent space and consists of six consequent linear layers, all but the last followed by batch-norms and ReLUactivations.
The intermediate layers’ sizes are 64 , 128 , 128 , 128 , 64 .
The discriminator is formed by a sequence of five linear layers, each but the last followed by the ReLU-activation.
The intermediate layers’ sizes are 64 , 128 , 128 , 64 .
The starting checkpoints for the Dissecting Contributions experiments are taken from the intermediate checkpoints of the GAN training for Source-I.
We take every 50-th checkpoint, gatheringin total.
We perform fine-tuning to the target distribution with the same parameters as above except the number of steps equal 1000 .
L ONGER TRAINING
In this series of experiments, we run GAN training for a source checkpoint being either Imagenetpretrained or randomly initialized for two times higher number of steps (50 million real images shown to the discriminator).
The results are presented in Table 10.
Generally, Imagenet-pretraining almost always either improves GAN quality or performs equally to the random initialization while speeding up convergence by a large margin. [Table]
T RANSFER FROM AN EARLIER EPOCH
This experiment verifies if it is important to transfer from a well-converged nearly-optimal source checkpoint, or it is sufficient to start from a roughly stabilized checkpoint from the intermediate step of the optimization process.
To address the question, we perform a series of additional experiments with Imagenet and FFHQ as source domains
[Table]
and Flowers as a target domain.
As pretrained checkpoints, we consider the best-FID checkpoint and a checkpoint that passed two times fewer steps.
The results for these runs are presented in Table 11.
Overall, the choice between two options has only a
[Table]
marginal impact on the transfer quality, and one can use the source checkpoint from the middle of training to initialize the finetuning process.
D ETAILS OF EXPERIMENTS ON THE GAN INVERSION
We take the e4e generator inversion approach proposed by (Tov et al., 2021) and train an encoder that maps real data to the GAN latent space.
This scheme is known to be capable of mapping real images to the GAN latent space preserving all generator properties such as latent attributes manipulations.
We follow the original author’s implementation and train an independent encoder model for each generator.
For a generator G we receive an encoder E which is trained to satisfy G ( E ( x ))
=
x for each real data sample x .
We evaluate the encoders with the average LPIPS-distance (Zhang et al., 2018) between a test set real samples and their inversions equal E x ∼ p test LPIPS( x, G ( E ( x ))) .
We also report the average distance between an original image and its reconstruction features of a pretrained features extractor F which is equal E x ∼ p test ∥ F ( x ) − F ( G ( E ( x )))
∥ 2 .
The lower these quantities –, the better reconstruction quality is.
Following (Tov et al., 2021), for FFHQ-target generators, we train the encoder on the FFHQ dataset and evaluate it on the Celeba-HQ dataset and on FFHQ itself.
As for LSUN-Bedroom, we split the original data into a train and a test subset in the proportion 9 : 1 and train e4e on the train set and evaluate on the test set.
As
[Figure] [Figure]