O N THE “ STEERABILITY ” OF GENERATIVE ADVERSARIAL NETWORKS
A BSTRACT
An open secret in contemporary machine learning is that many models work beautifully on standard benchmarks but fail to generalize outside the lab.
This has been attributed to biased training data, which provide poor coverage over real world events.
Generative models are no exception, but recent advances in generative adversarial networks (GANs) suggest otherwise – these models can now synthesize strikingly realistic and diverse images.
Is generative modeling of photos a solved problem?
We show that although current GANs can ﬁt standard datasets very well, they still fall short of being comprehensive models of the visual manifold.
In particular, we study their ability to ﬁt simple transformations such as camera movements and color changes.
We ﬁnd that the models reﬂect the biases of the datasets on which they are trained (e g., centered objects) but that they also exhibit some capacity for generalization: by “steering” in latent space, we can shift the distribution while still creating realistic images.
We hypothesize that the degree of distributional shift is related to the breadth of the training data distribution.
Thus we conduct experiments to quantify the limits of GAN transformations and introduce techniques to mitigate the problem
I NTRODUCTION
The quality of deep generative models has increased dramatically over the past few years.
When introduced in 2014, Generative Adversarial Networks (GANs) could only synthesize MNIST digits and low-resolution grayscale faces (Goodfellow et al., 2014).
The most recent models, however, produce diverse high-resolution images that are often indistinguishable from natural photos (Brock et al., 2018;
Karras et al., 2018).
Science ﬁction has long dreamed of virtual realities ﬁlled of synthetic content as rich as, or richer, than the real world (e.g., The Matrix , Ready Player One ).
How close are we to this dream?
Traditional computer graphics can render photorealistic 3D scenes, but cannot automatically generate detailed content.
Generative models like GANs, in contrast, can create content from scratch, but we do not currently have tools for navigating the generated scenes in the same kind of way as you can walk through and interact with a 3D game engine.
In this paper, we explore the degree to which you can navigate the visual world of a GAN.
Figureillustrates the kinds of transformations we explore.
Consider the dog at the top-left.
By moving in some direction of GAN latent space, can we hallucinate walking toward this dog?
As the ﬁgure indicates, and as we will show in this paper, the answer is yes.
However, as we continue to zoom in, we quickly reach limits.
Once the dog face ﬁlls the full frame, continuing to walk in this direction fails to increase the zoom.
A similar effect occurs in the daisy example (row 2 of Fig. 1), where a direction in latent space moves the daisy up and down, but cannot move it out of frame.
We hypothesize that these limits are due to biases in the distribution of images on which the GAN is trained.
For example, if the training dataset consists of centered dogs and daises, the same may be the case in GAN-generated images.
Nonetheless, we ﬁnd that some degree of transformation is possible.
When and why can we achieve certain transformations but not others?
Code and project page at: https://ali-design.github.io/gan_steerability/
- Brightness + - Rotate2D + - Rotate3D + - Zoom + - Shift Y +
[Figure]
+
This paper seeks to quantify the degree to which we can achieve basic visual transformations by navigating in GAN latent space.
In other words, are GANs “steerable” in latent space?
1 We analyze the relationship between the data distribution on which the model is trained and the success in achieving these transformations.
From our experiments, it is possible to shift the distribution of generated images to some degree, but we cannot extrapolate entirely out of the dataset’s support.
In particular, attributes can be shifted in proportion to the variability of that attribute in the training data.
We further demonstrate an approach to increase model steerability by jointly optimizing the generator and latent direction, together with data augmentation on training images.
One of the current criticisms of generative models is that they simply interpolate between datapoints, and fail to generate anything truly new, but our results add nuance to this story.
It is possible to achieve distributional shift, but the ability to create realistic images from a modiﬁed distributions relies on sufﬁcient diversity in the dataset along the dimension that we vary.
Our main ﬁndings are:
• A simple walk in the latent space of GANs achieves camera motion and color transformations in the output image space.
These walks are learned in self-supervised manner without labeled attributes or distinct source and target images.
• The linear walk is as effective as more complex non-linear walks, suggesting that the models learn to roughly linearize these operations without being explicitly trained to do so.
• The extent of each transformation is limited, and we quantify a relationship between dataset variability and how much we can shift the model distribution.
• The transformations are a general-purpose framework that work with different model architectures, e.g. BigGAN, StyleGAN, and DCGAN, and illustrate different disentanglement properties in their respective latent spaces.
• Data augmentation improves steerability, as does jointly training the walk trajectory and the generator weights, which allows us to achieve larger transformation effects.
R ELATED WORK
Latent space manipulations can be seen from several perspectives – how we achieve it, what limits it, and what it enables us to do.
Our work addresses these three aspects together, and we brieﬂy refer to each one in related work.
Interpolations in latent space Traditional approaches to image editing with GAN latent spaces ﬁnd linear directions that correspond to changes in labeled attributes, such as smile-vectors and gender-vectors for faces (Radford et al., 2015; Karras et al., 2018).
However these manipulations are not exclusive to GANs; in ﬂow-based generative models, linearly interpolating between two encoded images allow one to edit a source image toward attributes of the target (Kingma & Dhariwal, 2018).
M¨ollenhoff & Cremers (2019) proposes a modiﬁed GAN formulation by treating data as directional k -currents, where moving along tangent planes naturally corresponds to interpretable manipulations.
Upchurch et al.
(2017) removes the generative model entirely and instead interpolates in the intermediate feature space of a pretrained classiﬁer, again using feature mappings of source and target sets to determine an edit direction.
Unlike these approaches, we learn our latentspace trajectories in a self-supervised manner without labeled attributes or distinct source and target images.
Instead, we learn to approximate editing operations on individual source images.
We ﬁnd that linear trajectories in latent space can capture simple image manipulations, e.g., zoom-vectors and shift-vectors, although we also obtain similar results using nonlinear trajectories.
Dataset bias Biases from training data and network architecture both impact the generalization capacity of learned models (Torralba & Efros, 2011;
Geirhos et al., 2018;
Amini et al.).
Dataset biases partly comes from human preferences in taking photos: we tend to take pictures in speciﬁc “canonical” views that are not fully representative of the entire visual world (Mezuman & Weiss, 2012;
Jahanian et al., 2015).
Consequently, models trained with these datasets inherit their biases.
This may result in models that misrepresent the given task – such as tendencies towards texture bias rather than shape bias on ImageNet classiﬁers (Geirhos et al., 2018) – and in turn limits their generalization performance on similar objectives (Azulay & Weiss, 2018).
Our latent space trajectories transform the output corresponding to various image editing operations, but ultimately we are constrained by biases in the data and cannot extrapolate arbitrarily far beyond the data’s support.
Generative models for content creation
The recent progress in generative models has opened interesting avenues for content creation (Brock et al., 2018;
Karras et al., 2018), including applications that enable users to ﬁne-tune the generated output (Simon; Zhu et al., 2016;
Bau et al., 2018).
A by-product the current work is enable users to modify image properties by turning a single knob – the magnitude of the learned transformation in latent space.
We further demonstrate that these image manipulations are not just a simple creativity tool; they also provide us with a window into biases and generalization capacity of these models.
Applications of latent space editing Image manipulations using generative models suggest several interesting downstream applications.
For example, Denton et al.
(2019) learns linear walks corresponding to various facial characteristics – they use these to measure biases in facial attribute detectors, whereas we study biases in the generative model that originate from training data.
Shen et al.
(2019) also assumes linear latent space trajectories and learns paths for face attribute editing according to semantic concepts such as age and expression, thus demonstrating disentanglement of the latent space.
White (2016) suggests approaches to improve the learned manipulations, such as using spherical linear interpolations, resampling images to remove biases in attribute vectors, and using data augmentation as a synthetic attribute for variational autoencoders.
Goetschalckx et al.
(2019) applies a linear walk to achieve transformations corresponding to cognitive properties of an image such as memorability, aesthetics, and emotional valence.
Unlike these works, we do not require an attribute detector or assessor function to learn the latent space trajectory, and therefore our loss function is based on image similarity between source and target images.
In addition to linear walks, we explore using non-linear walks parametrized by neural networks for editing operations.
M ETHOD
Generative models such as GANs (Goodfellow et al., 2014) learn a mapping function G such that
G : z → x .
Here, z is the latent code drawn from a Gaussian density and x is an output, e.g., an image.
Our goal is to achieve transformations in the output space by moving in latent space, as shown in Fig.
In general, this goal also captures the idea in equivariance, in which transformations in the input space result in equivalent transformations in the output space (c.f. Hinton et al.
Cohen et al.
; Lenc & Vedaldi (2015)).
Objective We want to learn an N -dimensional vector representing the optimal path in latent space for a given transformation.
The vector is multiplied with continuous parameter α which signiﬁes the step size: large α values correspond to a greater degree of transformation, while small α values correspond to a lesser degree.
Formally, we learn the walk w by minimizing the objective function:
[Equation]
[Figure]
The previous setup assumes linear latent space walks, but we can also learn non-linear trajectories in which the walk direction depends on the current latent space position.
For the non-linear walk, we learn a function, f ∗ ( z ) , which corresponds to a small (cid:15) -step transformation edit ( G ( z ) , (cid:15) ) .
To achieve bigger transformations, we apply f recursively, mimicking discrete Euler ODE approximations.
Formally, for a ﬁxed (cid:15) , we minimize
[Equation]
where f n ( · ) is an n th-order function composition
f ( f ( f ( ... ))) , and f ( z ) is parametrized with a neural network.
We discuss further implementation details in Appendix A.4.
We use this function composition approach rather than the simpler setup of G ( z + α NN ( z )) because the latter learns to ignore the input z when α takes on continuous values, and is thus equivalent to the previous linear trajectory (see Appendix A.3 for further details).
Quantifying Steerability We further seek to quantify how well we can achieve desired image manipulations under each transformation.
To this end, we compare the distribution of a given attribute, e.g., “luminance”, in the dataset versus in images generated after walking in latent space.
For color transformations, we consider the effect of increasing or decreasing the α coefﬁcient corresponding to each color channel.
To estimate the color distribution of model-generated images, we randomly sample N = 100 pixels per image both before and after taking a step in latent space.
Then, we compute the pixel value for each channel, or the mean RGB value for luminance, and normalize the range between 0 and 1.
For zoom and shift transformations, we rely on an object detector which captures the central object in the image class.
We use a MobileNet-SSD v1 (Liu et al., 2016) detector to estimate object bounding boxes, and average over image classes recognizable by the detector.
For each successful detection, we take the highest probability bounding box corresponding to the desired class and use that to quantify the amount of transformation.
For the zoom operation, we use the area of the bounding box normalized by the area of the total image.
For shift in the X and Y directions, we take the center X and Y coordinates of the bounding box, and normalize by image width or height.
Truncation parameters in GANs (as used in Brock et al.
; Karras et al.
(2018)) trade off between the diversity of the generated images and sample quality.
When comparing generated images to the dataset distribution, we use the largest possible truncation for the model and perform similar cropping and resizing of the dataset as done during model training (see Brock et al.
When comparing the attributes of generated distributions under different α magnitudes to each other but not to the dataset, we reduce truncation to 0.5 to ensure better performance of the object detector.
Reducing Transformation Limits Equations 1 and 2 learn a latent space walk assuming a pretrained generative model, thus keeping the model weights ﬁxed.
The previous approach allows us to understand the latent space organization and limitations in the model’s transformation capacity.
To overcome these limits, we explore adding data augmentation by editing the training images with each corresponding transformation, and train the generative model with this augmented dataset.
We also introduce a modiﬁed objective function that jointly optimizes the generator weights and a linear walk vector:
[Equation]
where the edit loss encourages low L 2 error between learned transformation and target image:
[Equation]
The GAN loss optimizes for discriminator error:
[Equation]
where we draw images x from the training dataset and perform data augmentation by applying the edit operation on them.
This optimization approach encourages the generator to organize its latent space so that the transformations lie along linear paths, and when combined with data augmentation, results in larger transformation ranges which we demonstrate in Sec. 4.
E XPERIMENTS
We demonstrate our approach using BigGAN (Brock et al., 2018), a class-conditional GAN trained on 1000 ImageNet categories.
We learn a shared latent space walk by averaging across the image categories, and further quantify how this walk affects each class differently.
We focus on linear walks in latent space for the main text, and show additional results on nonlinear walks in Sec. 4. and Appendix B.4.2.
We also conduct experiments on StyleGAN (Karras et al., 2018), which uses an unconditional style-based generator architecture in Sec.
4.3 and Appendix B.5.
[Figure]
of w ∗ , the operation either does not transform the image any further, or the image becomes unrealisitic. Below each ﬁgure we also indicate the average LPIPS perceptual distance between 200 sampled image pairs of that category.
Perceptual distance decreases as we move farther from the source (center image), which indicates that the images are converging.
We show qualitative results of the learned transformations in Fig.
By steering in the generator latent space, we learn a variety of transformations on a given source image (shown in the center panel of each transformation).
Interestingly, several priors come into play when learning these image transformations.
When we shift a daisy downwards in the Y direction, the model hallucinates that the sky exists on the top of the image.
However, when we shift the daisy up, the model inpaints the remainder of the image with grass.
When we alter the brightness of a image, the model transitions between nighttime and daytime.
This suggests that the model can extrapolate from the original source image, and still remain consistent with the image context. [Figure]
However, when we increase the step size of α , we observe that the degree to which we can achieve each transformation is limited.
In Fig.
3 we observe two potential failure cases: one in which the the image becomes unrealistic, and the other in which the image fails to transform any further.
When we try to zoom in on a Persian cat, we observe that the cat no longer increases in size beyond some point, and in fact consistently undershoots the target zoom.
On the other hand, when we try to zoom out on the cat, we observe that it begins to fall off the image manifold, and does not become any smaller after some point.
Indeed, the perceptual distance (using LPIPS) between images decreases as we push α towards the transformation limits.
Similar trends hold with other transformations: we are able to shift a lorikeet up and down to some degree until the transformation yields unrealistic output, and despite adjusting α on the rotation vector, we are unable to rotate a pizza.
Are the limitations to these transformations governed by the training dataset?
In other words, are our latent space walks limited because in ImageNet photos the cats are mostly centered and taken within a certain size?
We seek to investigate and quantify these biases in the next sections.
An intriguing characteristic of the learned trajectory is that the amount it affects the output depends on the image class.
In Fig.
4, we investigate the impact of the walk for different image categories under color transformations.
By moving in the direction of a redness vector, we are able to successfully recolor a jellyﬁsh, but we are unable to change the color of a goldﬁnch, which remains yellow which slight changes in background textures.
Likewise, increasing brightness changes an erupting volcano to a dormant one, but does not have much effect on Alps, which only transitions between night and day.
In the third example, we use our latent walk to turn red sports cars to blue, but it cannot recolor ﬁretrucks.
Again, perceptual distance over image samples conﬁrms these qualitative observations: a 2-sample t -test yields t = 20 .
77 , p < 0 .
001 for jellyﬁsh/goldﬁnch, t = 8 .
14 , p < 0 .
001 for volcano/alp, and t = 6 .
84 , p < 0 .
001 for sports car/ﬁre engine.
We hypothesize that the different impact of the shared transformation on separate image classes relates to the variability in the underlying dataset.
The overwhelming majority of ﬁretrucks are red 2 , but sports cars appear in a variety of colors.
Therefore, our color transformation is constrained by the dataset biases of individual classes.
With shift, we can move the distribution of the center object by varying α .
In the underlying model, the center coordinate of the object is most concentrated at half of the image width and height, but after applying the shift in X and shift in Y transformation, the mode of the transformed distribution varies between 0.3 and 0.7 of the image width/height.
To quantify the distribution changes, we compute the area of intersection between the original model distribution and the distribution after applying each transformation and observe that the intersection decreases as we increase or decrease the magnitude of α . However, our transformations are limited to a certain extent – if we increase α [Figure] Shift X
[Figure] beyond 150 pixels for vertical shifts, we start to generate unrealistic images, as evidenced by a sharp rise in FID and converging modes in the transformed distributions (Fig. 5 columns 2 & 3).
We perform a similar procedure for zoom, by measuring the area of the bounding box for the detected object under different magnitudes of α .
Like shift, we observe that subsequent increases in α magnitude start to have smaller and smaller effects on the mode of the resulting distribution (Fig.last column).
Past an 8x zoom in or out, we observe an increase in the FID signifying decreasing image quality.
Interestingly for zoom, the FID under zooming in and zooming out is anti-symmetric, indicating that how well we can zoom-in and retain realisitic images differs from that of zooming out.
These trends are consistent with the plateau in transformation behavior that we qualitatively observe in Fig.
Although we can arbitrarily increase the α step size, after some point we are unable to achieve further transformation and risk deviating from the natural image manifold.
4. H OW DOES THE DATA AFFECT THE TRANSFORMATIONS ?
Is the extent to which we can transform each class, as we observed in Fig.
4, due to limited variability in the underlying dataset for each class?
One way of quantifying this is to measure the difference in transformed model means, model(+ α ) and model(- α ) , and compare it to the spread of the dataset distribution.
For each class, we compute standard deviation of the dataset with respect to our statistic of interest (pixel RGB value for color, and bounding box area and center value for zoom and shift transformations respectively).
We hypothesize that if the amount of transformation is biased depending on the image class, we will observe a correlation between the distance of the mean shifts and the standard deviation of the data distribution.
More concretely, we deﬁne the change in model means under a given transformation as:
[Equation]
6) for a given class k and we set α ∗ to be largest and smallest α values used in training.
The degree to which we achieve each transformation is a function of α , so we use the same α value for all classes – one that is large enough to separate the means of µ k, model( α ∗ ) and µ k, model(- α ∗ ) under [Figure] [Figure] transformation, but also for which the FID of the generated distribution remains below a threshold
T of generating reasonably realistic images (for our experiments we use T = 22 ).
In Fig.
6 we plot the standard deviation σ of the dataset on the x-axis, and the model ∆ µ under a + α ∗ and − α ∗ transformation on the y-axis, as deﬁned in Eq.
We sample randomly fromclasses for the color, zoom and shift transformations
[Figure]
and generate 200 samples of each class under the positive and negative transformations.
We use the same setup of drawing samples from the model and dataset and computing the statistics for each transformation as described in Sec. 4.1.
Indeed, we ﬁnd that the width of the dataset distribution, captured by the standard deviation of random samples drawn from the dataset for each class, relates to how much we can transform.
There is a positive correlation between the spread of the dataset and the magnitude of ∆ µ observed in the transformed model distributions, and the slope of all observed trends differs signiﬁcantly from zero ( p < 0 .
001 for all transformations).
For the zoom transformation, we show examples of two extremes along the trend.
For the “robin” class the spread σ in the dataset is low, and subsequently, the separation ∆ µ that we are able to achieve by applying + α ∗ and − α ∗ transformations is limited.
On the other hand, for “laptops”, the dataset spread is broad; ImageNet contains images of laptops of various sizes, and we are able to attain wider shifts in the model distribution.
From these results, we conclude that the amount of transformation we can achieve relates to the dataset variability.
Consistent with our qualitative observations in Fig.
4, we ﬁnd that if the images for a particular class have adequate coverage over the entire range of a given transformation, then we are better able to move the model distribution to both extremes.
On the other hand, if the images for a given class are less diverse, the transformation is limited by this dataset bias.
4. A LTERNATIVE ARCHITECTURES AND WALKS
We ran an identical set of experiments using the nonlinear walk in the BigGAN latent space (Eq 2) and obtained similar quantitative results.
To summarize, the Pearson’s correlation coefﬁcient between dataset σ and model ∆ µ for linear walks and nonlinear walks is shown in Table 1, and full results in Appendix B.4.2.
Qualitatively, we observe that while the linear trajectory undershoots the targeted level of transformation, it is able to preserve more realistic-looking results (Fig. 7). The [Figure]
transformations involve a trade-off between minimizing the loss and maintaining realistic output, and we hypothesize that the linear walk functions as an implicit regularizer that corresponds well with the inherent organization of the latent space. [Figure] X Shift Y
Zoom
[Figure] [Figure] [Figure] To test the generality of our ﬁndings across model architecture, we ran similar experiments on StyleGAN, in which the latent space is divided into two spaces, z and W .
As Karras et al.
(2018) notes that the W space is less entangled than z , we apply the linear walk to W and show results in Fig.and Appendix B.5.
One interesting aspect of StyleGAN is that we can change color while leaving other structure in the image unchanged.
In other words, while green faces do not naturally exist in the dataset, the StyleGAN model is still able to generate them.
This differs from the behavior of
BigGAN, where changing color results in different semantics in the image, e.g., turning a dormant volcano to an active one.
StyleGAN, however, does not preserve the exact geometry of objects under other transformations, e.g., zoom and shift (see Appendix B.5).
4. T OWARDS S TEERABLE GAN S
So far, we have frozen the parameters of the generative model when learning a latent space walk for image editing, and observe that the transformations are limited by dataset bias.
Here we investigate approaches to overcome these limitations and increase model steerability.
For these experiments, we use a class-conditional DCGAN model (Radford et al., 2015) trained on MNIST digits (LeCun, 1998).
To study the effect of dataset biases, we train (1) a vanilla DCGAN and (2) a DCGAN with data augmentation, and then learn the optimal walk in Eq.
1 after the model has been trained – we refer to these two approaches in Fig.
9 as argmin W and argmin W + aug , respectively.
We observe that adding data augmentation yields transformations that better approximate the target image and attain lower L 2 error than the vanilla DCGAN (blue and orange curves in Fig. 9).
Qualitatively, we observe that transformations using the vanilla GAN ( argmin W ) become patchy and unrealistic as we increase the magnitude of α , but when the model is trained with data augmentation ( argmin W + aug ), the digits retain their structural integrity.
Rather than learning the walk vector w assuming a frozen generator, we may also jointly optimize the model and linear walk parameter together, as we formalized in Eq. 3.
This allows the model to learn an equivariance between linear directions in the latent space and the corresponding image transformations.
We refer to this model as argmin G,W in Fig.
Compared to the frozen generator (in argmin W and argmin W + aug ), the joint objective further decreases L 2 error (green curve in Fig. 9).
We show additional qualitative examples in Appendix B.8.
The steerable range of the generator increases with joint optimization and data augmentation, which provides additional evidence that training data bias impacts the models’ steerability and generalization capacity.
We tried DCGAN on CIFAR10 as a more complicated dataset, however were unable to get steering to be effective – all three methods failed to produce realistic transformations and joint training in fact performed the worst.
Finding the right steering implementation per GAN and dataset, especially for joint training, may be a difﬁcult problem and an interesting direction for future work.
[Figure]
2D
[Figure]
C ONCLUSION
GANs are powerful generative models, but are they simply replicating the existing training datapoints, or can they to generalize beyond the training distribution?
We investigate this question by exploring walks in the latent space of GANs.
We optimize trajectories in latent space to reﬂect simple image transformations in the generated output, learned in a self-supervised manner.
We ﬁnd that the model is able to exhibit characteristics of extrapolation – we are able to “steer” the generated output to simulate camera zoom, horizontal and vertical movement, camera rotations, and recolorization.
However, our ability to naively move the distribution is ﬁnite: we can transform images to some degree but cannot extrapolate entirely outside the support of the training data.
To increase model steerability, we add data augmentation during training and jointly optimize the model and walk trajectory.
Our experiments illustrate the connection between training data bias and the resulting distribution of generated images, and suggest methods for extending the range of images that the models are able to create. A CKNOWLEDGEMENTS
We would like to thank Quang H Le, Lore Goetschalckx, Alex Andonian, David Bau, and Jonas
Wulff for helpful discussions.
This work was supported by a Google Faculty Research Award
to P.I., and a U.S. National Science Foundation Graduate Research Fellowship to L.C. A M ETHOD DETAILS A. O PTIMIZATION FOR THE LINEAR WALK
We learn the walk vector using mini-batch stochastic gradient descent with the Adam opti mizer (Kingma & Ba 2014) in tensorﬂow, trained on 20000 unique samples from the latent space z .
We share the vector w across all ImageNet categories for the BigGAN model.
A. I MPLEMENTATION D ETAILS FOR L INEAR W ALK
We experiment with a number of different transformations learned in the latent space, each corresponding to a different walk vector.
Each of these transformations can be learned without any direct supervision, simply by applying our desired edit to the source image.
Furthermore, the parameter α allows us to vary the extent of the transformation.
We found that a slight modiﬁcation to each transformation improved the degree to which we were able to steer the output space: we scale α differently for the learned transformation G ( z + α g w ) , and the target edit edit ( G ( z ) , α t ) .
We detail each transformation below:
Shift.
We learn transformations corresponding to shifting an image in the horizontal X direction and the vertical Y direction.
We train on source images that are shifted − α t pixels to the left and α t pixels to the right, where we set α t to be between zero and one-half of the source image width or height D .
When training the walk, we enforce that the α g parameter ranges between -1 and 1; thus for a random shift by t pixels, we use the value α g = α t /D .
We apply a mask to the shifted image, so that we only apply the loss function on the visible portion of the source image.
This forces the generator to extrapolate on the obscured region of the target image.
Zoom.
We learn a walk which is optimized to zoom in and out up to four times the original image.
For zooming in, we crop the central portion of the source image by some α t amount, where 0 .
25 <
α t < 1 and resize it back to its original size
To zoom out, we downsample the image by α t where 1 < α t <
To allow for both a positive and negative walk direction, we set α g = log( α t )
Similar to shift, a mask applied during training allows the generator to inpaint the background scene
Color.
We implement color as a continuous RGB slider, e.g., a 3-tuple α t =
( α R , α G , α B ) , where each α R , α G , α B can take values between [ − 0 . 5 , 0 . 5] in training.
To edit the source image, we simply add the corresponding α t values to each of the image channels.
Our latent space walk is parameterized as z + α g w =
z + α R w R
+ α G w G
+ α B w B
where we jointly learn the three walk directions w R , w G , and w B .
Rotate in 2D. Rotation in 2D is trained in a similar manner as the shift operations, where we train with − 45 ≤ α t ≤ 45 degree rotation.
Using R = 45 , scale α g = α t /R .
We use a mask to enforce the loss only on visible regions of the target.
Rotate in 3D. We simulate a 3D rotation using a perspective transformation along the Z-axis, essentially treating the image as a rotating billboard.
Similar to the 2D rotation, we train with − 45 ≤ α t ≤ 45 degree rotation, we scale α g = α t /R
where R = 45 , and apply a mask during training.
A. L INEAR NN ( z ) WALK
Rather than deﬁning w as a vector in z space (Eq. 1), one could deﬁne it as a function that takes a z as input and maps it to the desired z (cid:48) after taking a variable-sized step α in latent space.
In this case, we may parametrize the walk with a neural network w = NN ( z ) , and transform the image using
G ( z + α NN ( z )) .
However, as we show in the following proof, this idea will not learn to let w be a function of z .
Proof.
For simplicity, let w =
F ( z ) .
We optimize for
J ( w, α ) =
E z
[ L ( G ( z + αw ) , edit ( G ( z ) , α ))] where α is an arbitrary scalar value.
Note that for the target image, two equal edit operations is equivalent to performing a single edit of twice the size (e.g., shifting by 10px the same as shifting by 5px twice; zooming by 4x is the same as zooming by 2x twice).
That is,
[Equation]
To achieve this target, starting from an initial z , we can take two steps of size α in latent space as follows:
[Equation]
However, because we let α take on any scalar value during optimization, our objective function enforces that starting from z and taking a step of size 2 α equals taking two steps of size α :
[Equation]
Therefore:
[Equation]
Thus F ( · ) simply becomes a linear trajectory that is independent of the input z .
A. O PTIMIZATION FOR THE NON - LINEAR WALK
Given the limitations of the previous walk, we deﬁne our nonlinear walk F ( z ) using discrete step sizes (cid:15) .
We deﬁne F ( z ) as z + NN ( z ) , where the neural network NN learns a ﬁxed (cid:15) step transformation, rather than a variable α step.
We then renormalize the magnitude z .
This approach mimics the
Euler method for solving ODEs with a discrete step size, where we assume that the gradient of the transformation in latent space is of the form (cid:15) dzdt = NN ( z ) and we approximate z
i +
=
z i + (cid:15) dzdt | z i .
The key difference from A.3 is the ﬁxed step size, which avoids optimizing for the equality in (7).
We use a two-layer neural network to parametrize the walk, and optimize over 20000 samples using the Adam optimizer as before.
Positive and negative transformation directions are handled with two neural networks having identical architecture but independent weights.
We set (cid:15) to achieve the same transformation ranges as the linear trajectory within 4-5 steps.
B A DDITIONAL E XPERIMENTS B. M ODEL AND D ATA D ISTRIBUTIONS
How well does the model distribution of each property match the dataset distribution?
If the generated images do not form a good approximation of the dataset variability, we expect that this would also impact our ability to transform generated images.
In Fig.
10 we show the attribute distributions of the BigGAN model G ( z ) compared to samples from the ImageNet dataset.
We show corresponding results for StyleGAN and its respective datasets in Appendix B.5.
While there is some bias in how well model-generated images approximate the dataset distribution, we hypothesize that additional biases in our transformations come from variability in the training data.
B. Q UANTIFYING T RANSFORMATION L IMITS
We observe that when we increase the transformation magnitude α in latent space, the generated images become unrealistic and the transformation ceases to have further effect.
We show this qualitatively in Fig.
To quantitatively verify this trends, we can compute the LPIPS perceptual distance of images generated using consecutive pairs of α i and α i +1 .
For shift and zoom transformations, perceptual distance is larger when α (or log( α ) for zoom) is near zero, and decreases as the the magnitude of α increases, which indicates that large α magnitudes have a smaller transformation effect, and the transformed images appear more similar.
On the other hand, color and rotate in 2D/3D exhibit a steady transformation rate as the magnitude of α increases.
Note that this analysis does not tell us how well we achieve the speciﬁc transformation, nor whether the latent trajectory deviates from natural-looking images.
Rather, it tells us how much we manage to change the image, regardless of the transformation target.
To quantify how well each transformation is achieved, we rely on attribute detectors such as object bounding boxes (see B.3).
B. D ETECTED B OUNDING B OXES
To quantify the degree to which we are able to achieve the zoom and shift transformations, we rely on a pre-trained MobileNet-SSD v1 3 object detection model.
In Fig.
12 and 13 we show the results of applying the object detection model to images from the dataset, and images generated by the model under the zoom, horizontal shift, and vertical shift transformations for randomly selected values of α , to qualitatively verify that the object detection boundaries are reasonable.
Not all ImageNet images contain recognizable objects, so we only use ImageNet classes containing objects recognizable by the detector for this analysis.
B. A LTERNATIVE W ALKS IN B IG GAN B.4. LPIPS OBJECTIVE
In the main text, we learn the latent space walk w by minimizing the objective function:
[Equation]
(8) using a Euclidean loss for L .
In Fig.
14 we show qualitative results using the LPIPS perceptual similarity metric (Zhang et al., 2018) instead of Euclidean loss.
Walks were trained using the same parameters as those in the linear-L2 walk shown in the main text: we use 20k samples for training, with Adam optimizer and learning rate 0.001 for zoom and color, 0.0001 for the remaining edit operations (due to scaling of α ).
B.4. N ON - LINEAR W ALKS
Following B.4.2, we modify our objective to use discrete step sizes (cid:15) rather than continuous steps.
We learn a function F ( z ) to perform this (cid:15) -step transformation on given latent code z , where F ( z ) is parametrized with a neural network.
We show qualitative results in Fig.
We perform the same set of experiments shown in the main text using this nonlinear walk in Fig.
These experiments exhibit similar trends as we observed in the main text – we are able to modify the generated distribution of images using latent space walks, and the amount to which we can transform is related to the variability in the dataset.
However, there are greater increases in FID when we apply the non-linear transformation, suggesting that these generated images deviate more from natural images and look less realistic.
B.4. A DDITIONAL Q UALITATIVE E XAMPLES
We show qualitative examples for randomly generated categories for BigGAN linear-L2, linear
LPIPS, and nonlinear trajectories in Figs.
17, 18, 19 respectively.
B. W ALKS IN S TYLE GAN
We perform similar experiments for linear latent space walks using StyleGAN models trained on the LSUN cat, LSUN car, and FFHQ face datasets.
As suggested by Karras et al.
(2018), we learn the walk vector in the intermediate W latent space due to improved attribute disentanglement in W .
We show qualitative results for color, shift, and zoom transformations in Figs.
20, 22, 24 and corresponding quantitative analyses in Figs.
We show qualitative examples for the comparison of optimizing in the W and z latent spaces in Stylegan in 28.
B. W ALKS IN P ROGRESSIVE GAN
We also experiment with the linear walk objective in the latent space of Progressive GAN Karras et al.
One interesting property of the Progressive GAN interpolations is that they take much longer to train to have a visual effect – for example for color, we could obtain drastic color changes in
Stylegan W latent space using as few as 2k samples, but with progressive gan, we used 60k samples and still did not obtain as strong of an effect.
This points to the Stylegan w latent space being more “ﬂexible” and generalizable for transformation, compared to the latent space of progressive GAN.
Moreover, we qualitatively observe some entanglement in the progressive gan transformations – for example, changing the level of zoom also changes the lighting.
We did not observe big effects in the horizontal and vertical shift transformations.
Qualitative examples and quantitative results are shown in Figs.
B. Q UALITATIVE EXAMPLES FOR ADDITIONAL TRANSFORMATIONS
Since the color transformation operates on individual pixels, we can optimize the walk using a segmented target – for example when learning a walk for cars, we only modify pixels in segmented car region when generating edit ( G ( z ) , α ) .
StyleGAN is able to roughly localize the color transformation to this region, suggesting disentanglement of different objects within the W latent space (Fig.left) as also noted in Karras et al.
; Shen et al.
We also show qualitative results for adjust image contrast (Fig. 29 right), and for combining zoom, shift X, and shift Y transformations (Fig. 30).
B. A DDITIONAL RESULTS FOR IMPROVING MODEL STEERABILITY
We further test the hypothesis that dataset variability impacts the amount we are able to transform by comparing DCGAN models trained with and without data augmentation.
Namely, with data augmentation, the discriminator is able to see edited versions of the real images.
We also jointly train the model and the walk trajectory which encourages the model to learn linear walks.
For zoom, horizontal shift, and 2D rotate transformations, additional samples for three training approaches – without data augmentation, with data augmentation, and joint optimization – appear in Fig.
Qualitatively, transformations using the model trained without data augmentation degrade the digit structure as α magnitude increases, and may even change one digit to another.
Training with data augmentation and joint optimization better preserves digit structure and identity.
[Figure]
[Figure]
P e r ce p t u a l D
i s t a n ce
Rotate
[Figure]
P e r
[Figure]
p t u a l D
i s t a n ce
[Figure]
p=
[Figure]
p=0.
[Figure] [Figure]
[Figure]
- Rotate 2D + - Shift X + - Zoom + - Rotate 3D + - Shift Y + - Color +
[Figure]
[Figure] Rotate 2D
[Figure]
3D
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
n e r ec t i o n
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]