U SING LATENT SPACE REGRESSION TO ANALYZE AND LEVERAGE COMPOSITIONALITY IN GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
In recent years, Generative Adversarial Networks have become ubiquitous in both research and public perception, but how GANs convert an unstructured latent code to coherent, high quality output is still an open question.
In this work, we investigate regression into the latent space as a probe to understand the compositional properties of GANs.
We ﬁnd that combining the regressor and a pretrained generator provides a strong image prior, allowing us to create composite images from a collage of random image parts at inference time and leverage the generator to rectify the inconsistencies.
To compare compositional properties across different generators, we measure the trade-offs between reconstruction of the unrealistic input and image quality of the regenerated samples.
We ﬁnd that the regression approach enables more localized editing of individual image parts compared to direct editing in the latent space, and we conduct experiments to quantify this independence effect.
Our method is agnostic to how image parts are extracted, and does not require explicit labelled knowledge of concepts during training.
Because this approach only uses a single feed-forward pass, it can operate in real-time and also perform a number of related applications, such as image inpainting or example-based image editing.
We demonstrate its effectiveness on several GANs and datasets.
[Figure]
I NTRODUCTION
Unconditional generative adversarial networks are the current state-of-the-art in image generation (Karras et al., 2019b;a; 2017; Radford et al., 2015), but like any deep model, our understanding of this complex mapping from random noise to images remains limited.
How do GANs create realistic scenes from seemingly uninterpretable noise?
What kinds of representations do they learn during training?
And how interchangeable are these learned concepts?
Here, we use a latent space regressor model to probe what GANs know about the world.
Given a pretrained, ﬁxed generator network, we train a regressor to approximate the inverse mapping from image to noise sample.
We then use this mapping to investigate the priors that a GAN enforces and the degrees of freedom that it allows.
For example, from a given church image, we can explore alternative ways that the generator can imagine constructing that scene.
Can the generator maintain the same church but add clouds to the sky?
What if we swap the foreground tree for a different one?
Given only parts of the building, can it realistically ﬁll in the missing portion?
We show that in fact, the pretrained generator – without any additional interventions – already represents these compositional properties in its latent code.
In addition, it applies certain rules that it learns from the dataset, ensuring that the output is always a coherent scene regardless of inconsistencies in the input.
Our approach is simple – using a ﬁxed pretrained generator, we subsequently train a regressor to predict the latent code from an input image, while adding a masking modiﬁcation to learn to handle missing pixels.
To investigate the GAN’s ability to produce alternative versions of a given scene, we input a rough template of the scene we desire, and use the regressor and generator networks to convert it into a realistic image.
Even though our regressor is never trained on these unrealistic templates, the encoder bottleneck projects each image into a reasonable part of the latent space, which the generator maps onto the image manifold.
The approach requires no labels or clustering of attributes; all we need is a single example of approximately how we want the generated image to look.
It only requires a forward pass of the regressor and generator, so there is low latency in obtaining the output image, unlike iterative optimization approaches that can require upwards of a minute to reconstruct each image.
We use the regressor and generator to investigate the compositional capabilities of pretrained GANs across different dataset domains.
Using automatically generated input collages, we leverage the generator to recombine this unrealistic content into a coherent image.
We then investigate the GAN’s ability to independently vary localized portions of a given image.
In summary, our contributions are:
• We propose a latent regression model that learns to perform image reconstruction even in case of incomplete images and missing pixels.
• There is no use of labelled attributes nor test-time optimization, so we can modify images based on a single example of the desired modiﬁcation and reconstruct in real-time.
The reconstruction respects the style of the modiﬁed context, and not just the object identity.
• Together, the regressor and generator form a prior that can create coherent scene composites from rough collages of extracted image parts.
• We use the regressor to probe what parts of a scene can vary independently without explicit conceptual labels, and investigate the difference between image mixing using the encoder and interpolation in latent space.
• The same regressor setup can be used for a variety of other applications, such as multimodal editing, scene completion, or dataset rebalancing.
R ELATED W ORK
Image Inversion.
Given a target image, either the output of the generator network or a real image, the inversion problem aims to recover the latent code of the generator that best corresponds to the target.
Image inversion comes with a number of challenges, including 1) the complex optimization landscape and 2) the generator’s inability to reconstruct out-of-domain images.
To relax the domain limitations of the generator, one possibility is to invert to a more ﬂexible intermediate latent space (Abdal et al., 2019).
However, inverting to an intermediate code may allow the generator to become overly ﬂexible, thus requiring regularizers to ensure that the recovered latent code does not deviate too far from the latent manifold (Pividori et al., 2019; Zhu et al., 2020; Wulff & Torralba, 2020).
An alternative to increasing the ﬂexibility of the generator is to learn an ensemble of latent codes that approximate a target image when combined (Gu et al., 2019a).
Due to the challenging optimization, the quality of inversion depends on good initialization.
A number of approaches propose a hybrid of a latent regression network to provide an initial guess of the latent code with subsequent optimization of the latent code (Bau et al., 2019; Guan et al., 2020) or the generator weights (Zhu et al., 2016;
Bau et al., 2020; Pan et al., 2020), while Huh et al.
(2020) investigates gradient-free approaches for optimization.
Another use case of image inversion through a generator may be to complete partial scenes: using optimization, this is achieved in optimization methods by reconstructing only the known pixels (Bora et al., 2017;
Gu et al., 2019a; Abdal et al., 2020), whereas in feed-forward methods, this missing region must be provided explicitly to the model.
Rather than inverting into pre-trained generator, one can train the generator and encoder jointly; based on modiﬁcations to the Variational
Autoencoder (Kingma & Welling, 2013); Donahue et al.
; Donahue & Simonyan (2019);
Dumoulin et al.
(2016) use this setup to investigate the properties of latent space representations learned during training, while Pidhorskyi et al.
(2020) demonstrates a joint learning method that can achieve comparable image quality to recent GAN models.
Speciﬁcally in our work, we investigate the emergent priors of a pretrained GAN using a masked latent regression objective as an approximate image inverter; the regression network allows us to investigate these GAN priors with a tradeoff of an more approximate reconstruction but much lower latency compared to optimization-based techniques.
Composition in Image Domains.
To join segments of disparate image sources into one cohesive output, early works use hand-designed features, such as Laplacian pyramids for seamless blending (Burt & Adelson, 1983).
To automate this, Isola & Liu (2013) and Hays & Efros (2007) employ nearest-neighbor approaches for scene composition and completion.
More recently, a number of deep network architectures have been developed for compositional tasks.
For discriminative tasks, Tabernik et al.
(2016) and Kortylewski et al.
(2020) train CNNs with modiﬁed compositional architecture for purposes of model interpretability and reasoning about object occlusion in classiﬁcation.
For image synthesis, Mokady et al.
(2019) and Press et al.
(2020) use an autoencoder to encode, disentangle, and swap properties between two sets of images, while Shocher et al.
(2020) proposes mixing images in deep feature space while training the generator.
Rather than creating models speciﬁcally for the image composition or scene completion objective, we investigate the ability of a pre-trained generator to mix-and-match parts of its generated images.
Related to our work, Besserve et al.
(2018) estimates the modular structure of GANs by learning a casual model of latent representations, whereas we investigate the generator’s compositional properties using image inversion.
Image Editing.
A recent topic of interest is the ability to edit images using generative models.
A number of works propose linear attribute vector editing directions to perform image manipulation operations (Goetschalckx et al., 2019;
Jahanian et al., 2019;
Shen et al., 2019; Kingma & Dhariwal, 2018; Karras et al., 2019a; Radford et al., 2015).
One may also identify concepts learned in the generator’s intermediate layers by clustering intermediate representations, either using segmentation labels (Bau et al., 2018) or unsupervised clustering (Collins et al., 2020), and change these representations to edit the desired concepts in the output image.
Suzuki et al.
(2018) use a spatial feature blending approach which mixes properties of target images in the intermediate feature space of a generator.
On faces, editing can be achieved using a 3D parametric model to supervise the modiﬁcation (Tewari et al., 2017; 2020).
In our work, we do not require clusters or concepts in intermediate layers to be deﬁned a priori, nor do we need distinct input or output domains as in image translation tasks (Zhu et al., 2017; Almahairi et al., 2018).
Furthermore, unlike image manipulation using semantic maps (Park et al., 2019; Gu et al., 2019b), our approach respects the style of the manipulation (e.g. the speciﬁc color of the sky), which is lost in the semantic map representation.
Our method shares commonalities with Richardson et al.
(2020), although we focus on investigating composition rather than image-to-image translation.
In our approach, we only require a single example of the approximate target property we want to modify and use regression into the latent space as a fast image prior to create a coherent output.
This allows us to create edits that are not contingent on learned concepts, and we do not need to modify or train the generator network.
M ETHOD 3. L ATENT CODE RECOVERY IN GAN S
While GANs provide a simple mapping from a predetermined input distribution to a complex output distribution, e.g. from a standard normal Z to the image manifold X , they are not easily invertible.
In other words, given an image sample from the output distribution, it is not trivial to recover the sample from the input distribution that generated it.
The image inversion objective aims to ﬁnd the latent code z that best recovers the desired target image x :
[Equation]
using some metric of image distance dist , such as pixel-wise L1 error or a metric based on deep features.
This objective can be solved iteratively, using LBFGS (Liu & Nocedal, 1989) or other optimizers.
However, iteration incurs a substantial time cost – it takes a large number of iterations to converge, is prone to local minima, and must be performed for each target image x independently.
An alternative way of of recovering the latent code z is to train a neural network to amortize the optimization costs.
In this case, the recovered latent code is simply the result of a feed-forward pass through a trained regressor network, z ∗ =
E ( x ) , where E can be used for any x ∈ X .
3. L EARNING A LATENT REGRESSOR FUNCTION
Here, we investigate using latent regressors to probe properties of the generator latent space, such as the ability to combine scenes and complete missing regions.
Our latent regression objective for the regressor network E is:
[Equation]
We sample z randomly from the latent distribution, and pass it through a pretrained generator G to obtain the target image x = G ( z ) .
Between the target image x and the recovered image G ( E ( x )) , we use a mean square error loss to guide reconstruction, and a perceptual loss L p (Zhang et al., 2018) to recover details.
Between the original latent code z and the recovered latent code E ( x ) , we use a latent recovery loss L z .
We use mean square error or a variant of cosine similarity for latent recovery, depending on the GAN’s input normalization.
We provide additional details in Supp.
Sec.
A.1.1.
In all cases the generator is frozen, and we only optimize the weights of the latent space encoder E .
Using ProGAN (Karras et al., 2017), we train the encoder network to directly invert to the latent code z .
For StyleGAN (Karras et al., 2019b), we encode to an expanded W + latent space.
Once trained, the output of the latent regressor yields a latent code such that the reconstructed image looks perceptually similar to the target image and is much faster than iterative optimization methods.
3. I MAGE COMPOSITION USING LATENT REGRESSION
The regressor network E and generator G form an image prior to project any input image x input onto the manifold of generated images X , even if x input / ∈ X .
We take advantage of this to investigate the compositional properties of the latent code.
We extract parts of images (either generated by G or from real images), and combine them to form a collaged image x clg .
This extraction process does not need to be precise and can have obvious seams, so x clg is often not realistic.
We then use the regressor/generator pair to blend the seams and produce a realistic composite output.
To create x clg , we sample base images x i and masks mask i , and combine them via union:
[Equation]
To automate the process of extracting masked images, we use a pretrained segmentation network (Xiao et al., 2018) and sample from the output classes (additional details in Supp.
Sec. A.1.2).
However, the masking network is not speciﬁc to segmentation – we also experiment with a saliency network (Liu et al., 2018) as well as user-deﬁned masks in Supp.
Sec.
A.2.4.
Once we have formed the collage x clg , we reproject via the regressor generator to obtain the recovered composite image x rec =
G ( E ( x clg )) .
Note that the encoder is trained solely for the latent recovery objective (Eqn. 2) and has never seen collaged images.
3. L EARNING WITH MISSING DATA
When automatically creating composite images, not every pixel in x comp will be ﬁlled with part of a base image x i .
In optimization approaches using Eqn. 1, this can be handled by optimizing only over the known pixels.
However, a regressor network cannot handle this naively - it cannot distinguish between unknown pixels and known pixels, and will try to ﬁt the values of the unknown pixels.
This can be mitigated with a slight modiﬁcation to the regression network, by indicating which pixels are known versus unknown as input (Fig. 2), yielding a modiﬁed regression objective:
[Equation]
Instead of taking an image x as input, the encoder takes as input a masked image x m and a mask m , where x m = x ⊗ m , and m is an additional channel of input.
Again, we hold the generator
[Figure]
E
G m ⊗ ̂ x training the with additional binary
[Figure]
ﬁxed and only optimize the weights of the encoder using image samples from the generator.
Similar to before, we train only for the latent recovery objective and not for the image composition task.
Intuitively, this masking operation on the encoder operates similarly to “dropout” Srivastava et al.
(2014) on pixels – it encourages the encoder to learn a ﬂexible way of recovering a latent code that still enables the generator to reconstruct the image.
Thus, given only partial images as input, the encoder is encouraged to map from these known pixels into a part of the latent space that is semantically consistent with the rest of the image (as deeper layers of the generator focus on semantic concepts (Bau et al., 2018)).
This allows the generator to reproduce an image that is both likely under its prior and consistent with the observed pixels.
To obtain the masked image during training we take a small patch of random uniform noise u , upsample to create smooth boundaries, and sample a threshold t from the uniform distribution to create the binary mask.
However, at test time, the exact form of the mask does not matter – the mask simply indicates where the generator should try to reconstruct or inpaint, and does not distinguish between the different image parts of the input (i.e., the masks used to extract image parts in Eqn.are not available to the encoder).
We provide additional details in Supp.
Sec.
A.1.1 and A.2.3.
E XPERIMENTS
Using pre-trained Progressive GAN (Karras et al., 2017) and StyleGAN2 (Karras et al., 2019b) generators, we conduct experiments on CelebA-HQ and FFHQ faces and LSUN cars, churches, living rooms, and horses to investigate the compositional elements that the generators learn from data.
4. I MAGE C OMPOSITION FROM A PPROXIMATE C OLLAGES
By training a latent regressor with a masked objective, we ﬁnd that the encoder and generator pair forms an image prior which can convert unrealistic input images into realistic scenes that maintain properties of the input image.
By mixing parts of different images together and encoding through the generator, we obtain a coherent output that is consistent with each part of the input while ﬁlling in the missing regions.
How well can the generator recombine concepts that it creates, and how does this ability vary across different generator architectures and dataset domains?
In this experiment, we automate the process of creating collages from image parts by randomly sampling the generated images, extract different semantic categories using a pretrained segmentation network, and overlaying these patches using Eqn. 3.
In Fig.
4, we show qualitative examples of these input collages and the corresponding generator output across a variety of domains; note that the input collages are not realistic, and often contain missing detections or misalignments.
However, the learned image prior from the generator and encoder bottleneck ﬁxes these issues to create realistic outputs.
[Figure]
When creating composites from random images, composition is a tradeoff between two factors.
First, we desire the networks to reconstruct the input, which are recombinations of images that each generator creates.
However, due to the automatic nature of the collages, a perfect reconstruction will not be realistic due to the lack of alignment between image parts, so the second criteria is that the reconstructed images should have good quality.
In Fig.
3, we visualize the tradeoff between these two desirable properties of composition by measuring how well each generator can reconstruct randomly mixed parts of images from its own domain using masked L1 distance (lower is better):
[Equation]
and the image quality of the synthesized result (FID score; Heusel et al. (2017).
To isolate the realism of the composite image from the encoder’s reconstruction capability, we compare the difference FID between the reconstructed composites G ( E ( x )) , where x = ∪ i mask i ⊗ G ( z i ) , and re-encoded images
G ( E ( G ( z )) .
In general, encoding to the StyleGAN W + latent space results in better reconstructions of the collaged input, but at the tradeoff of higher FID, potentially due to the increased ﬂexibility of the expanded W + latent code compared to the more restricted z latent code in ProGAN.
However, while the StyleGAN generator on FFHQ faces has the largest increase in FID, we note that the synthesized composites have high density but lower coverage (Naeem et al., 2020), indicating that while the generated composites are close to the real-image manifold, they are missing full variation of natural images.
We show full results on FID, density, coverage, and L1 reconstruction error and additional random samples of input collages and generator output in Supp.
Sec.
A.2.4.
While here we use a pretrained segmentation network to automatically extract parts, our reconstruction method is agnostic to the way in which the parts are extracted.
In Supplementary Sec.
A.2.1 and
A.2.4, we show that these image collages can also be created by extracting output from a saliency network, approximate rectangles, or user-drawn masks.
4. C OMPARING C OMPOSITIONAL P ROPERTIES A CROSS A RCHITECTURES
While using a trained encoder network gives us a fast and simple way of creating a composite image from an input collage, it is not the only way of ﬁnding images, in our case the collaged input, within a pretrained generator network.
Is it only pretrained generators that have this property, and is it contingent on having a feed-forward encoder?
Here, we compare the the different image priors enforced by different image reconstruction methods, using both pretrained generators as well as separately trained generator architectures, where inference involves either optimization or a feed-forward pass.
For comparison, we use the same set of collages to compare the methods, generated from parts of random real images of the church and face domains.
As some methods take several minutes to reconstruct a single image, we use 200 collages for each domain (see Tab. 4 for optimization times of each).
Due to the smaller sample size, we use density as a measure of realism (higher is better), which measures proximity to the real-image manifold (Naeem et al., 2020).
For the church domain, we compare to methods in which the generator and encoder are trained jointly:
DIP (Ulyanov et al., 2018), CycleGAN (Zhu et al., 2017), and SPADE (Park et al., 2019) (additional details in Supplementary Sec.
A.2.4).
Next, we experiment with different ways of inverting into the latent space of pretrained GAN networks: (1) our method of using a trained encoder network, (2)
[Figure]
using LBFGS optimization into a pretrained generator, (3) or a hybrid of the encoder and LBFGS.
We also evaluate the compositional properties of two additional iterative optimization methods: Gu et al.
(2019a) which optimizes for multiple latent codes at an intermediate layer of the GAN, and
a perceptual-loss based projection method demonstrated in Karras et al.
(2019b).
Qualitatively, the different methods have varying degrees of realism when trying to reconstruct unrealistic input collages (we show examples in Supp.
Fig. 17); optimization-based methods such as DIP tend to overﬁt and lead to unrealistic reconstructions with lower L1 error but also low density (unrealistic), whereas segmentation-based methods such as SPADE are not trained to reconstruct the input, leading to high L1.
Fig 5 illustrates this compositional tradeoff, where the encoder based methods perform slightly worse in L1 reconstruction compared to optimization approaches, but maintain more realistic output and can reconstruct with much lower computational time.
On the face domain, we compare composite outputs generated from ProGAN and StyleGAN using the regressor network to the ALAE architecture (Pidhorskyi et al., 2020), which jointly learns the encoder and decoder, and the Im2StyleGAN (Abdal et al., 2019) algorithm which does not use an encoder but directly optimizes the latent code.
We also compare to In-domain inversion (Zhu et al., 2020), which uses an unmasked encoder to initialize and regularize a secondary masked optimization step.
We show qualitative examples in Supplementary Fig.
18 and a comparison of reconstruction L and density in Fig.
5: the our ProGAN and StyleGAN masked encoders can maintain closer proximity to the real image manifold (higher density) compared to the alternative methods, with much faster inference time compared to Im2StyleGAN.
While the In-domain encoder is also a feed-forward method, it cannot handle incomplete inputs, so both the encoder and subsequent optimization step have low density.
On these same inputs, ALAE exhibits interesting compositional properties and is qualitatively able to correct misalignments in face patches, but the density of generated images lower than that of the pretrained GANs.
4. H OW DOES COMPOSITION DIFFER FROM INTERPOLATION ?
Combining images in pixel space and using the encoder bottleneck to rectify the input is only one way that a generator can mix different images.
Another commonly used method is to simply use a linear interpolation between two latent codes to obtain an output that has properties of both input images.
Here, we investigate how these two approaches differ.
When collaging parts of two images, we desire that part of a context image stays the same while the remaining portion changes to match a chosen target: to achieve this composition , we replace the pixels of our context image with the desired modiﬁed target, and pass the result through the encoder bottleneck to obtain the blended image.
This can be described using: G ( E ( m ⊗ x 1 + (1 − m ) ⊗ x 2 )) .
Does directly interpolating in latent space give us the same result?
To compare, we compute the latent α -blending by performing a weighted average of the context and target latent codes: G ( α ∗ E ( x 1 )
+ (1 − α ) ∗ E ( x 2 )) .
Third, we also compare to pixel α -blending the inputs in pixel space and using the encoder bottleneck to make the output more realistic:
G ( E ( α ∗ x 1 + (1 − α ) ∗ x 2 )) .
We select the weight α to be proportional to the area of the target modiﬁcation.
[Figure]
Qualitatively, the disentangled interpolation is better able to change the target region while keeping the context region ﬁxed, e.g., add white windows while reducing changes in the ﬁreplace or couch in Fig.
6, whereas the other two interpolation methods are less faithful in preserving image content.
To quantify this effect, we compute the masked L1 distance (Eqn. 5) of the interpolated images to (1) the original masked context region, (2) to the original masked target region, and (3) to the input collage over 500 random samples.
The disentangled interpolation has lower distance from the target and context, and is also closer to the desired collage.
Unlike interpolations using attribute vectors, these distentangled directions do not need to be learned from data - they are based on a single context and target image, and also allow multiple possible directions of manipulation.
We show additional interpolations and comparison to learned attribute vectors in Supp.
Sec.
A.2.2, and additional applications such as dataset rebalancing and one-shot image editing in Supp.
Sec.
A.2.1.
4. U SING REGRESSION TO INVESTIGATE INDEPENDENCE OF IMAGE COMPONENTS
One property of the regressor and generator pair is that it enforces global coherence: When we modify part of the input, the generator will create an output that is still globally consistent.
While global consistency is desirable, we also want to minimize leakage of the edits.
For example, if we change a facial expression from a frown to a smile, the hair style should not change.
Here, we seek to quantify the independence of parts under the global coherence constraint imposed by the regressor / generator pair.
We parse a given face image x into separate semantic components (such as eyes, nose, etc.) represented as masks m c .
For each component, we generate N new faces x n , and replace the component region m c in x by the corresponding region in x n , yielding (after regression and generation) for each component c a set of N images x c,n = G ( E ( m c ⊗
[Figure]
x n + (1 − m c ) ⊗ x )) .
We can now measure how much changing component c changes each pixel location by computing the normalized pixel-wise standard deviation of x c,n across the n different replacements as v c = σ c / (cid:80) c σ c , where σ c = (cid:112) E n [( x n,c − E n [ x n,c ]) 2 ] .
For a given component c , we measure independence as the average variation outside of c that results from modifying c as s c =
E
[(1 − m c ) ⊗ v c ] (a lower s c means higher independence).
We repeat this experimenttimes and use N = 20 samples.
[Table]
Table 1 shows the variations of ProGAN and StyleGAN.
With the exception of the nose, StyleGAN achieves better independence, indicating increased part localization and less leakage of edits.
Figure 7(a) and (b) qualitatively show examples for the variation maps v c for different parts for ProGAN (a) and StyleGAN (b); the replaced part is marked in red.
It can be seen that in StyleGAN the variation is more concentrated in the region that is being replaced.
Lastly, this method can be utilized for unsupervised part discovery, as shown in Fig. 7(c).
Here, changing the color of the rear door (top left) changes the appearance of the whole car body; changing the windshield appearance (top right) also changes the windows, and the foreground (bottom left) and background (bottom right) are large parts varying together, but distinct from the car.
C ONCLUSION
Using a simple latent space regression model, we investigate the compositional properties of pretrained
GAN generators.
We train a regressor model to predict the latent code given an input image with the additional objective of learning to complete a scene with missing pixels.
With this regressor, we can probe various properties and biases that the GAN learns from data.
In creating scenes, the
GAN allows for local degrees of freedom but maintains an overall degree of global consistency.
This allows us to input approximate templates to the regressor model, such as partially occluded scenes or collages extracted from parts of images, and use the regressor and generator as image prior to regenerate a realistic output.
Inverting the collaged images yields disentangled interpolation directions where part of the input changes while the remaining context remains ﬁxed.
The latent regression approach allows us to investigate how the GAN enforces independence of image parts, and is learned without knowledge of objects or explicit attribute labels.
It only requires a single forward pass on the models, which enables real time image editing based on single image examples.
A A PPENDIX A. S UPPLEMENTARY M ETHODS A.1. A DDITIONAL T RAINING D ETAILS
The loss function of the encoder contains image loss terms to ensure that the output of the generator approximates the target image, and a latent recovery loss term to ensure that the predicted latent code matches the original latent code.
On the image side, we use mean square error loss in conjunction with LPIPS perceptual loss (Zhang et al., 2018).
The latent recovery loss depends on the type of GAN.
Due to pixel normalization in ProGAN, we use a latent recovery loss based on cosine similarity, as the exact magnitude of the recovered latent code does not matter after normalization:
[Equation]
For StyleGAN, we invert to the intermediate latent space, as it is known that in this space semantic properties are better disentangled than in Z .
Furthermore, allowing the latents to differ on different scales has been shown to better capture the variability of real images (Abdal et al., 2019).
During training, we therefore generate different latents for different scales, and train the encoder to estimate different styles, i.e. estimate w ∈ W + .
Unlike the latent space of ProGAN, however, w ∈ W + is not normalized to the hypersphere.
Instead of a cosine loss, we therefore use a mean square error loss as the latent recovery loss:
[Equation]
We train the encoders using a ResNet backbone (ResNet-18 for ProGAN, and ResNet-34 for Stylegan), modifying the output dimensionality to match the number of latent dimensions for each GAN.
The encoders are trained with the Adam optimizer with default parameters, learning rate lr = 0 .
We use batch size 16 for the 256 resolution generators, and train for 500K batches (training takes around two days on a single GPU).
For the 1024 resolution generator, we use batch size 4 and 400K batches.
When training with masks, we take a small 6x6 patch of random uniform noise u , upsample to the generator’s resolution, and sample a threshold t from the uniform distribution in range [0 . 3 , 1 . 0] to create the binary mask.
[Equation]
At inference time, the exact shape of the mask does not matter: we can use hard coded rectangles, hand-drawn masks, or masks based on other pretrained networks.
Note that the mask does not distinguish between input image parts – it is a binary mask with value 1 where the generator should try to reconstruct, and 0 where the generator should ﬁll in the missing region.
A.1. A DDITIONAL D ETAILS ON C OMPOSITION
When creating automated collages from image parts, we use a pretrained segmentation network (Xiao et al., 2018) to extract parts from randomly sampled individual images.
We manually deﬁne a set of segmentation class for a given image class, and, to handle overlap between parts, specify an order to these segmentation classes.
To generate a collage, we then sample one random image per class.
For church scenes, we use an ordering of (from back to front) sky, building, tree, and foreground layers – this ensures that a randomly sampled building patch will appear in front of the randomly sampled sky patch.
For living rooms, the ordering we use is ﬂoor, ceiling, wall, painting, window, ﬁreplace, sofa, and coffee table – again ensuring that the more foreground elements are layered on top.
For cars, we use sky, building, tree, foreground, and car.
For faces we order by background, skin, eye, mouth, nose, and hair.
In Sec.
A.2.4 we investigate using other methods to extract patches from images, rather than image parts derived from segmentation classes.
A. S UPPLEMENTARY R ESULTS A.2. A DDITIONAL A PPLICATIONS
In the main text, we primarily focus on using the regression network as a tool to understand how the generator composes scenes from missing regions and disparate image parts.
However, because the regressor allows for fast inference, it enables a number of other real-time image synthesis applications.
Image Completion From Masked Inputs.
Using the masked latent space regressor in Eqn. 4, we investigate the GAN’s ability to automatically ﬁll in unknown portions of a scene given incomplete context.
We show qualitative examples of this property in Fig.
These reconstructions are done on parts of real images to ensure that the regressor is not simply memorizing the input, as could be the case with a generated input (the regressor is never trained on real images).
For example, when a headless horse is shown to the regressor, the rest of the horse can be ﬁlled in by the GAN.
In contrast, a vanilla latent regressor (Eqn. 2; RGB) overﬁts on the missing pixels and is unable to produce a realistic output.
The image completion also depends on the exposed context; in the living room example, the white couch is completed in both cases but the window is only added based on known context, whereas the background is different when the window is not shown.
This suggests that the networks inherently learn an unsupervised object representation, allowing them to complete objects from only partial hints, even though the generator and regressor are never provided with structured labels or concepts during training.
[Figure]
Multimodal Editing.
Because the regressor only requires a single example of the property we want to reconstruct, it is possible to achieve multimodal editings simply by overlaying different desired properties on a given context image.
Here, we demonstrate an example of adding different styles of trees to a church scene.
In each of the context images, there is originally no tree on the right-hand side.
We can add different styles of trees to each context image simply by overlaying a tree from a different image, and performing image inversion to create the composite.
Here, we use a rectangular region as a mask, rather than following the boundary of the tree precisely.
However, note that, after inversion, the color of the sky remains consistent in the composite image, and so does the building color in second tree example.
This image editing approach does not required learning of clusters or having labelled attributes, and therefore can be done with a single pair of images without known cluster deﬁnitions, unlike the methods in Bau et al.
; Collins et al.
Furthermore, unlike methods based on segmentation maps (Park et al., 2019; Gu et al., 2019b), styles within each individual semantic category, e.g. the color of the sky, is also changeable based on the provided input to the encoder.
Attribute Editing With A Single Unlabeled Example.
Typically in attribute editing using generative models, one requires a collection of labeled examples of the attribute to modify; for example, we take a collection of generated images with and without the attribute based on an attribute classiﬁer, and ﬁnd the average difference in their latent codes (Jahanian et al., 2019; Radford et al., 2015;
Tree Tree
Tree Tree
Tree Tree
Tree
C o n t e x t C o n t e x t
C o n t e x t
Goetschalckx et al., 2019).
Here, we demonstrate an approach towards attribute editing without using attribute labels (Fig. 10).
We overlay a single example of our target attribute (e.g. a smiling mouth or a round church tower), on top of the context image we want to modify, and encode it to get the corresponding latent code.
We then subtract the difference between the original and modiﬁed latent codes, and add it to the latent code of the secondary context image:
z 1 , modiﬁed − z 1 + z 2 .
This bypasses the need for an attribute classiﬁer, such as one that identiﬁes round or square church towers
[Figure]
Dataset Rebalancing.
Because the latent regression inverter only requires a forward pass through the network, we can quickly generate a large number of reconstructions.
Here, we use this property to investigate rebalancing a dataset (Fig. 11).
Using pretrained attribute detectors from Karras et al.
(2019a)
, we ﬁnd that there is a smaller fraction of smiling males than smiling females in CelebAHQ (ﬁrst panel), although the detector itself is biased and tends to overestimate smiling in general compared to ground truth labels (second panel).
The detections in the GAN generated images mimic this imbalance (third panel).
Next, using a ﬁxed set of generated images, we randomly swap the mouth regions among them in accordance to our desired proportion of smiling rates – we give each face a batch of 16 possible swaps and taking the one that yields the strongest detection as smiling/not smiling based on our desired characteristic (if no swaps are successful, the face is just used as is).
We use a hardcoded rectangular box region around the mouth, and encode the image through the generator to blend the modiﬁed mouth to the face.
After performing swapping on generated images, this allows us rebalance the smiling rates, better equalizing smiling among males and females (fourth panel).
Finally, we use the latent regression to perform mouth swapping on the dataset, and train a secondary GAN on this modiﬁed dataset – this also improves the smiling imbalance, although the effect is stronger in males than females (ﬁfth panel).
We note that a limitation of this method is that the rebalanced proportion uses the attribute detector as supervision, and therefore a biased or inaccurate attribute detector will still result in biases in the rebalanced dataset.
[Figure]
A.2.
C OMPARING
COMPOSITION WITH LATENT SPACE INTERPOLATION
In the main text, we compare different interpolation examples – disentangled, latent, and pixel interpolations – on a living room generator.
Here, we show equivalent examples in church scenes.
In
Fig.
12, we demonstrate a “tree” interpolation, in which we want the background church to remain the same but trees to be added in the foreground.
Compared to the latent and pixel interpolations, the disentangled interpolation better preserves the church while adding trees, which we quantify using masked perceptual distance.
Similarly, we can change the sky of context scene - e.g. by turning a clear sky into a cloudy one in Fig.
13, where again, doing the disentangled interpolation better preserves the details of the details of the church while the sky changes.
In Fig 14, we show the result on changing the smile on a generated face image.
For the smile attribute, we can compare to a learned smile edit vector using labelled images from a pretrained smile classiﬁer.
We also measure the the facial embedding distance using a pretrained face-identiﬁcation network 1 – the goal of the interpolation is to change the smile of the image while reducing changes to the rest of the face identify, and thus minimize embedding distance.
Because the mouth region is small, choosing the interpolation weight α by the target area minimally changes the interpolated image ( α > 0 . 99 ), so instead we use an α = 0 .
7 weight so that all methods have similar distance to the target.
While the disentangled approach and the learned attribute vector approach perform similarly, learning the attribute vector requires labelled examples, and cannot perform multimodal modiﬁcations such as applying different smiles to a given face.
[Figure]
[Figure] [Figure]
pixel interpolations using the same encoder-based
Composition, Latent α -blend, and Pixel α -blend methods as above when adding a smile to a face.
We also compare these approaches to the modiﬁcations produced by edit vector learning from attribute labels.
A.2. L OSS A BLATIONS
When training the regression network, we use a combination of three losses: L2 pixel loss, perceptual loss, and a latent recovery loss.
In this section, we investigate the reconstruction result using different combinations of losses.
In the ﬁrst case, we do not enforce latent recovery, so the encoded latent code does not have to be close to the ground truth latent code but the encoder and generator just need to reconstruct the masked input image.
In the second case, we investigate omitting the perceptual loss, and simple use an L2 loss in image space.
Since the encoders are trained with L2 loss and the VGG variant of perceptual loss, we evaluate with L1 reconstruction and the AlexNet variant of perceptual loss.
Training with all losses leads to reconstructions that are more perceptually similar compared to the two other variants, while per-pixel L1 reconstruction is not greatly affected (Tab. 2).
We show qualitative examples of the reconstructions on masked input in Fig. 15.
[Table]
I n p u t
M a s k e d
A ll L o ss e s
N o Z
N o L P
I P
S
[Figure]
A DDITIONAL C OMPOSITION R ESULTS
Comparing different composition approaches.
In Fig 16, we show examples of extracted image parts, the collage formed by overlaying the image parts, and the result of poisson blending the images according to the outline of each extracted part.
We further compare to variations of the encoder setup, where (1) RGB: the encoder is not aware of missing pixels in the collaged input, (2) RGB Fill: we ﬁll the missing pixels with the average color of the background layer, and (3) RGBM: we allow the encoder and generator to inpaint on missing regions.
Table 3 shows image quality metrics of FID (lower is better) and density and coverage (higher is better) and L1 reconstruction (lower is better) for each generator and domain.
To get feature representations for the density and coverage metrics, we use pretrained VGG features.
While the composite input collages are highly unrealistic with high
FID and low density/coverage, the inverted images are closer to the image manifold.
Of the three inversion methods, the RGBM inversion tends to yield lower FID and higher density and coverage, while also minimizing L1 reconstruction error.
Comparing different image reconstruction generators.
How do image priors different across different image reconstruction pipelines?
Our encoder method relies on a pretrained generator, and trains an encoder to the latent space of that generator.
Therefore, it can take advantage of the image priors learned by the generator, where the encoder keeps the result close to the image manifold.
Here, we compare to different image reconstruction approaches, such as autoencoder architectures or optimization methods rather than feed-forward inference.
We construct the same set of input collages using parts of real images, where inference times of each method is listed in Table 4.
For the church domain, we compare to three autoencoding methods, where the generator and encoder is learned jointly.
(1) DIP (Ulyanov et al., 2018) learns an autoencoder to reconstruct a single image,
P r
o G A
N
C e l e b A - H Q
Part
Part Part
Collage
Poisson
RGB
RGB Fill
RGBM
P r
o G A
N
C hu r c h
P r
o G A
N
L i v i n g r oo m
S t y l e G
A N
FF H Q
S t y l e G
A N
C hu r c h
S t y l e G
A N c a r where a masked loss allows it to inpaint missing regions.
(2) We train a CycleGAN (Zhu et al., 2017) between image collage and real image domain, creating a network that is explicitly trained for image composition.
(3) We use a pretrained SPADE (Park et al., 2019) network which creates images from segmentation maps, but information about object style is lost in the segmentation input.
Next, we use the ProGAN and StyleGAN2 pretrained generators, and experiment with different ways of inverting into the latent code, using LBFGS optimization, a trained encoder network, or both.
We also investigate two additional iterative optimization approaches: Multi-Code Prior (Gu et al., 2019a) for image inpainting, and the StyleGAN2 projection method using perceptual loss (Karras et al., 2019a), which we modify to use a masked perceptual loss.
Fig 17 shows qualitative results, where in particular the third example is a challenging input where the lower part of the tower is missing.
The two encoders demonstrate an image prior in which the bottom of the tower is reconstructed on output.
While DIP can inpaint missing regions, it cannot leverage learned image prior.
CycleGAN can ﬁll in missing patterns, but with visible artifacts, whereas SPADE changes the style of each input component.
Iteratively optimizing on a pretrained generator may fail to reconstruct the the input collage if an appropriate initialization is not found, and it can also lose semantic priors as it optimizes towards an unrealistic input.
On the face domain we ﬁnd that the optimization-based Im2StyleGAN Abdal et al.
(2019) algorithm is not able to realistically inpaint missing regions in the input collage.
While the ALAE autoencoder (Pidhorskyi et al., 2020) exhibits characteristics of blending the collage into a cohesive output image, the reconstruction error is higher than that of the GAN-based approaches.
The In-domain encoder method (Zhu et al., 2020) is negatively impacted by the missing pixels in the input, resulting in low density, although the subsequent optimization step is able to reduce L1 distance (Fig. 18).
Composing images using alternative deﬁnitions of image parts.
In the main text, we focus on creating composite images using a pretrained segmentation network.
However, we note that the exact
[Table] [Table]
process of extracting image parts does not matter, as the encoder and generator form an image prior that will remove inconsistencies in the input.
In Fig.
19 we show composites generated using the output of a pretrained segmentation network Liu et al.
(2018), and in Fig.
20 we show compositions using hand-selected compositional parts, where the parts extracted from each image do not have to correpond precisely with object boundaries.
Random composition samples.
We show additional random samples of the generated composites across the ProGAN and StyleGAN2 generators for a variety of image domains in Fig.
21 and Fig.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]