GAN “S TEERABILITY ” W ITHOUT O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent research has shown remarkable success in revealing “steering” directions in the latent spaces of pre-trained GANs.
These directions correspond to semantically meaningful image transformations ( e.g. , shift, zoom, color manipulations), and have similar interpretable effects across all categories that the GAN can generate.
Some methods focus on user-speciﬁed transformations, while others discover transformations in an unsupervised manner.
However, all existing techniques rely on an optimization procedure to expose those directions, and offer no control over the degree of allowed interaction between different transformations.
In this paper, we show that “steering” trajectories can be computed in closed form directly from the generator’s weights without any form of training or optimization.
This applies to user-prescribed geometric transformations, as well as to unsupervised discovery of more complex effects.
Our approach allows determining both linear and nonlinear trajectories, and has many advantages over previous methods.
In particular, we can control whether one transformation is allowed to come on the expense of another ( e.g. , zoom-in with or without allowing translation to keep the object centered).
Moreover, we can determine the natural end-point of the trajectory, which corresponds to the largest extent to which a transformation can be applied without incurring degradation.
Finally, we show how transferring attributes between images can be achieved without optimization, even across different categories.
I NTRODUCTION
Since their introduction by Goodfellow et al.
(2014), generative adversarial networks (GANs) have seen remarkable progress, with current models capable of generating samples of very high quality (Brock et al., 2018; Karras et al., 2019a; 2018; 2019b).
In recent years, particular effort has been invested in constructing controllable models, which allow manipulating attributes of the generated images.
These range from disentangled models for controlling e.g. , the hair color or gender of facial images (Karras et al., 2019a;b; Choi et al., 2018), to models that even allow specifying object relations (Ashual & Wolf, 2019).
Most recently, it has been demonstrated that GANs trained without explicitly enforcing disentanglement, can also be easily “steered” (Jahanian et al., 2020; Plumerault et al., 2020).
These methods can determine semantically meaningful linear directions in the latent space of a pre-trained GAN, which correspond to various different image transformations, such as zoom, horizontal/vertical shift, in-plane rotation, brightness, redness, blueness, etc.
Interestingly, a walk in the revealed directions typically has a similar effect across all object categories that
the
GAN can generate, from animals to man-made objects.
To detect such latent-space directions, the methods of Jahanian et al.
(2020) and Plumerault et al.
(2020) require a training procedure that limits them to transformations for which synthetic images can be produced for supervision ( e.g. , shift or zoom).
Other works have recently presented unsupervised techniques for exposing meaningful directions (Voynov & Babenko, 2020; H¨ark¨onen et al., 2020; Peebles et al., 2020).
These methods can go beyond simple user-speciﬁed transformations, but also require optimization or training of some sort ( e.g. , drawing random samples in latent space).
In this paper, we show that for most popular generator architectures, it is possible to determine meaningful latent space trajectories directly from the generator’s weights without performing any kind of training or optimization.
As illustrated in Fig.
1, our approach supports both simple userdeﬁned geometric transformations , such as shift and zoom, and unsupervised exploration of directions that typically reveals more complex controls, like the 3D pose of the camera or the blur of the background.
We also discuss how to achieve attribute transfer between images, even across object
User-prescribed geometric transformations
Z oo m
Sh i ft X
Sh i ft Y
B a c k g r o und
R e m ova l
Target
T r a n s f erre d
Original
Color Transfer
Pose Transfer
Target
T r a n s f erre d
Original
Automatically detected transformations
C a m er a
R o t a t e
D e p t h
o f
F i e l d
T e m p er a t u re
L i g h t categories (see Fig. 1), again without any training.
We illustrate results mainly on BigGAN, which is class-conditional, but our trajectories are class-agnostic.
Our approach is advantageous over existing methods in several respects.
First, it is 10 4 × - 10 5 × faster.
Second, it seems to detect more semantic directions than other methods.
And third, it allows explicitly accounting for dataset biases.
First order dataset biases
As pointed out by Jahanian et al.
(2020), dataset biases affect the extent to which a pre-trained generator can accommodate different transformations.
For example, if all objects in the training set are centered, then no walk in latent space typically allows shifting an object too much without incurring degradation.
This implies that a “steering” latent-space trajectory should have an end-point.
Our nonlinear trajectories indeed possess such convergence points, which correspond to the maximally-transformed versions of the images at the beginning of the trajectories.
Conveniently, the end-point can be computed in closed form, so that we can directly jump to the maximally-transformed image without performing a gradual walk.
Second order dataset biases
Dataset biases can also lead to coupling between transformations.
For example, in many datasets zoomed-out objects can appear anywhere within the image, while zoomed-in objects are always centered.
In this case, trying to apply a zoom transformation may also result in an undesired shift so as to center the enlarged object.
Our unsupervised method allows controlling the extent to which transformation A comes on the expense of transformation B.
1. R ELATED WORK
Walks in latent space Many works use walks in a GAN’s latent space to achieve various effects ( e.g. , (Shen et al., 2020;
Radford et al., 2015; Karras et al., 2018; 2019b; Denton et al., 2019; Xiao et al., 2018; Goetschalckx et al., 2019)).
The recent works of Jahanian et al.
(2020) and Plumerault et al.
(2020) speciﬁcally focus on determining trajectories which lead to simple user-speciﬁed transformations, by employing optimization through the (pre-trained) generator.
Voynov & Babenko (2020) proposed an unsupervised approach for revealing dominant directions in latent space.
This
[Figure]
technique reveals more complex transformations, such as background blur and background removal, yet it also relies on optimization.
Most recently, the work of H ¨ ark ¨ onen et al.
(2020) studied unsupervised discovery of meaningful directions by using PCA on deep features of the generator.
The method seeks linear directions in latent space that best map to those deep PCA vectors, and results in a set of non-orthogonal directions.
Similarly to the other methods, it also requires a very demanding training procedure (drawing random latent codes and regressing the latent directions), which can take a day for models like BigGAN.
Nonlinear walks in latent space Linear latent-space trajectories may arrive at regions where the probability density is low.
To avoid this, some methods proposed to replace the popular Gaussian latent space distribution by other priors (Kilcher et al., 2018), or to optimize the generator together with the latent space (Bojanowski et al., 2018).
Others suggested to use nonlinear walks in latent space that avoid low-probability regions.
For example, Jahanian et al.
(2020) explored nonlinear trajectories parametrized by two-layer neural networks, while White (2016) proposed spherical paths for interpolating between two latent codes.
Hierarchical GAN architectures Recently there is tendency towards hierarchical GAN architectures (Karras et al., 2018; 2019a;
Brock et al., 2018; Choi et al., 2018), which are capable of producing high resolution images at very high quality.
It is known that the earlier scales in such models are responsible for generating the global composition of the image, while the deeper scales are responsible for more local attributes (Karras et al., 2019a; Yang et al., 2019;
H ¨ ark ¨ onen et al., 2020).
Here, we distil this common knowledge and show how meaningful directions can be detected in each level, and how these architectures allow transferring attributes between images.
U SER - SPECIFIED GEOMETRIC TRANSFORMATIONS
Most modern generator architectures map a latent code vector z ∈ R d having no notion of spatial coordinates, into a two-dimensional output image.
In some cases ( e.g. , BigGAN), different parts of z are processed differently.
In others ( e.g. , BigGAN-deep), z is processed as a whole.
However, in all cases, the ﬁrst layer maps z (or part of it) into a tensor with low spatial resolution ( e.g. , 4 × 4 ×in BigGAN 128).
This tensor is then processed by a sequence of convolutional layers that gradually increase its spatial resolution (using fractional strides), until reaching the ﬁnal image dimensions.
Our key observation is that since the output of the ﬁrst layer already has spatial coordinates, this layer has an important role in determining the coarse structure of the generated image.
This suggests that if we were to apply a geometric transformation, like zoom or shift, on the output of the ﬁrst layer, then we would obtain a similar effect to applying it directly on the generated image (Fig. 2).
In fact, it may even allow slight semantic changes to take place due to the deeper layers that follow, which can compensate for the inability of the generator to generate the precise desired transformed image.
As we now show, this observation can be used to ﬁnd latent space directions corresponding to simple geometric transformations.
2. L INEAR TRAJECTORIES
Let us start with linear trajectories.
Given a pre-trained generator G and some transformation T , our goal is to ﬁnd a direction q in latent space such that G ( z + q ) ≈ T { G ( z ) } for every z .
To this end, we deﬁne P to be the matrix corresponding to T in the resolution of the ﬁrst layer’s output.
Denoting the weights and biases of the ﬁrst layer by W and b , respectively, our goal is therefore to bring 1 W ( z + q ) + b as close as possible to P ( W z + b ) .
To guarantee that this holds on average over random draws of z , we formulate our problem as
[Equation]
where p z is the probability density function of z , and D is a diagonal matrix that can be used to assign different weights to different elements of the tensors.
For example, if P corresponds to a horizontal shift of one element to the right, then we would not like to penalize for differences in the leftmost column of the shifted feature maps (see Fig. 2).
In this case, we set the corresponding diagonal elements of D to 0 and the rest to 1 .
Assuming E
[ z ] = 0 , as is the case in most frameworks, the objective in (1) simpliﬁes to
[Equation]
where I is the identity matrix.
The ﬁrst term in (2) is independent of q , and the second term is quadratic in q and is minimized by
[Equation]
We have thus obtained a closed form expression for the optimal linear direction corresponding to transformation P in terms of only the weights W and b of the ﬁrst layer.
Figure 2 illustrates this framework in the context of the BigGAN model, in which the feature maps at the output of the ﬁrst layer are 4 × 4 .
For translation, we use a matrix P that shifts the tensor by one element (aiming at translating the output image by one fourth its size).
For zoom-in, we use a matrix P that performs nearest-neighbor 2 × up-sampling, and for zoom-out we use sub-sampling by 2 × .
For each such transformation, we can control the extent of the effect by multiplying the steering vector q by some α > 0 .
Figure 1 (top-left) and Fig. 3(a) show example results for zoom and shift with the BigGAN generator.
As can be seen, this simple approach manages to produce pronounced effects, although not using optimization through the generator, as in (Jahanian et al., 2020).
Following (Jahanian et al., 2020), we use an object detector to quantify our zoom and shift transformations.
Figure 4 shows the distributions of areas and centers of object bounding boxes in the transformed images.
As can be seen, our trajectories lead to similar effects to those of Jahanian et al.
(2020), despite being 10 4 × faster to compute (see Tab. 1).
Please refer to App. A.1 for details about the evaluation, and see additional results with BigGAN and with the DCGAN architecture of (Miyato et al., 2018) in App. A.
A CCOUNTING FOR FIRST -O RDER DATASET BIASES VIA N EUMANN TRAJECTORIES
With linear trajectories, the generated image inevitably becomes improbable after many steps, as p z ( z + α q ) is necessarily small for large α .
This causes the generated image to distort until eventually becoming meaningless after many steps.
One way to remedy this, is by using nonlinear trajectories that have endpoints.
Here, we focus on walks in latent space, having the form
[Equation]
for some matrix M and vector q .
We coin these Neumann trajectories, since unfolding the iterations leads to a Neumann series.
An important feature of such walks is that if the spectral norm of M is strictly smaller than 1 (a condition we ﬁnd to be satisﬁed in practice for the optimal M ), then they have a convergence point.
We use a diagonal M , which we ﬁnd gives the best results.
To determine the optimal M and q for a transformation P , we modify Problem (1) into
[Equation]
[Figure]
+ -
[Figure]
+ -
Zoom +
[Figure]
O u r O u r
O u r O u r
N e u m a nn
S t ee r .
O u r G r ea tZoom + -
Shift Y + -
Shift X
+
O u r
N e u m a nn
S t ee r .
O u r G r ea t
O u r
N e u m a nn
S t ee r .
O u r G r ea t
S t ee r .
S t ee r . S t ee r .
[Figure]
We assume again that E [ z ] = 0 , and make the additional assumption that E [ zz T ] =
σ 2 z I , which is the case in all current GAN frameworks.
In this setting, the objective in (5) reduces to
[Equation]
[Figure]
where (cid:107)·(cid:107) F denotes the Frobenius norm.
Here, q appears only in the second term, which is identical to the second term of (2).
Therefore, the optimal q is as in (3).
The matrix M appears only in the ﬁrst term, which is easily shown to be minimized when setting the diagonal entries of M to
[Equation]
where w i is the i th column of W .
Controlling the step size As opposed to linear trajectories, reﬁning the step size along our curved trajectories necessitates modifying both M and q .
To do so, we can search for a matrix ˜ M and vector ˜ q with which N steps of the form z n +1 =
˜
Mz n + ˜ q are equivalent to a single step of the walk (4).
Noting that the N th step of the reﬁned walk can be explicitly written as z N = ˜ M N z 0 + (
[Equation]
N
[Equation]
k =0 ˜ M k )˜ q , we conclude that the parameters of this N -times ﬁner walk are
[Equation]
Convergence point If the spectral norm of M is smaller than 1 , then we have that
[Equation]
where we used the fact that the ﬁrst term tends to zero and the second term is a Newmann series.
Superﬁcially, this may seem to imply that the endpoint of the trajectory is not a function of the initial point z 0 .
However, recall that in hierarchical architectures, like BigGAN, z refers to the part of the latent vector that enters the ﬁrst layer.
The rest of the latent vector is not modiﬁed throughout the walk.
Therefore, the latent vector at the endpoint equals the latent vector of the initial point, except for its subset of entries corresponding to the ﬁrst hierarchy level, which are replaced by ( I − M ) − 1 q .
2. A CCOUNTING FOR FIRST - ORDER DATASET BIASES VIA GREAT CIRCLE TRAJECTORIES
In the Neumann walk, the step size decreases along the path (as (cid:107) z n +
− z n (cid:107) → 0 ).
We now discuss an alternative nonlinear trajectory that has a natural endpoint yet permits a constant step size.
Here we avoid low density regions by explicitly requiring that the likelihood of all images along the path is constant.
For z ∼ N (0 , I ) , this translates to the requirement that the whole trajectory lie on the sphere whose radius equals the norm of the original latent code z 0 .
We stress that the method we discuss here can be applied to any direction q , whether determined in a supervised manner or not.
Speciﬁcally, suppose we want to steer our latent code towards a normalized direction v = q / (cid:107) q (cid:107) .
Then we can walk along the great circle on the sphere that passes through our initial point z 0 , and the point (cid:107) z 0 (cid:107) v (blue circle in Fig. 6).
Mathematically, let V denote the (one-dimensional) subspace spanned by v and let P V = vv T and P V ⊥ =
I − P V denote the orthogonal projections onto V and
V ⊥ , respectively.
Then the great circle trajectory can be expressed as
[Equation]
where u = P V ⊥ z 0 / (cid:107) P V ⊥ z 0 (cid:107) and θ = arccos( P V ⊥ z 0 / (cid:107) z 0 (cid:107) )
× sign ( (cid:104) z 0 , v (cid:105) ) .
The effect of this trajectory for a zoom-in direction is shown in Fig.
6 (third row).
The natural endpoint of the greatcircle path is (cid:107) z 0 (cid:107) v (blue point), beyond which the contribution of v starts to decrease.
As seen in
Fig.
6, this endpoint indeed corresponds to a plausible zoomed-in version of the original image.
2. C OMPARISON
Figure 3(b) compares our nonlinear walks (Neumann and great-circle) with those of the GAN steerabilty work of Jahanian et al.
As can be seen, the latter tend to involve undesired brightness changes.
The advantage of our nonlinear trajectories over the linear ones becomes apparent when performing long walks, as exempliﬁed in Fig.
In such settings, the linear trajectories deteriorate, whereas our nonlinear paths have meaningful endpoints.
This can also be seen in Fig.
4, which reports the Fre ´ chet
Inception distances (FID) achieved by the two approaches.
Interestingly, the nonlinear trajectories of the GAN steerability method also have endpoints, but these endpoints are the same for all images of a certain class (and distorted).
U NSUPERVISED EXPLORATION OF TRANSFORMATIONS
To go beyond simple user-prescribed geometric transformations, we now discuss exploration of additional manipulations in an unsupervised manner.
The key feature of our approach is that by revealing a large set of directions, we can now also account for second-order dataset biases.
3. P RINCIPAL LATENT SPACE DIRECTIONS
We start by seeking a set of orthonormal directions (possibly a different set for each generator hierarchy) that lead to the maximal change at the output of the layer to which z is injected.
These directions are precisely the right singular vectors of the corresponding weight matrix W , i.e., the k th most signiﬁcant direction is the k th column of the matrix V in the singular value decomposition
W = USV T (assuming the diagonal entries of S are arranged in decreasing order).
This reveals directions corresponding to many geometric, texture, color, and background effects (see Fig. 1).
[Table]
Our approach is seemingly similar to
GANspace (H ¨ ark ¨ onen et al., 2020), which computes PCA of activations within the network.
However, they optimize over latent space directions that best map to this deep PCA basis.
Concretely, they feed-forward random latent codes { z ( j ) } to obtain deep-feature representations { y ( j ) } , compute the PCA basis A and mean vector µ of these features, and then solve for a steering basis V = arg min (cid:80) j (cid:107) V A T ( y ( j ) − µ ) − z ( j ) (cid:107) .
Thus, besides computational inefﬁciency (see Tab. 1), they obtain a set of non-orthogonal latent-space directions (see App.
Fig.
48) that correspond to repeated effects (see App.
Figs.
In contrast, our directions are orthogonal by construction, and therefore capture a more diverse set of effects (App.
Figs.
For example, the semantic dissimilarity between G ( z ) and G ( z + 3 v ) is 64% larger with our method, as measured by the average LPIPS distance (Zhang et al., 2018) over the ﬁrst 50 directions ( 33 · 10 − 3 for GANSpace, 54 · 10 − 3 for us).
Having determined a set of semantic directions, we now want to construct trajectories that exhibit the corresponding effects, but also account for dataset biases.
As discussed in Sec. 2 and illustrated in the ﬁrst two rows of Fig. 6, performing linear walks along these directions eventually leads to distorted images.
A more appropriate choice is thus to use the great-circle walk described in Sec. 2.
This is illustrated in the third row of Fig.
While leading to meaningful endpoints, a limitation of
G ANS p a ce
O u r G re a t
O u r S m a ll
O u r L i n e a r
Steps
Nose Y coord.
Particularly, the great circle trajectory is that when walking on the sphere towards v , we actually also modify the projections onto other principal directions.
This causes other properties to change besides the desired attribute.
For example, in Fig. 6, the great circle causes a shift, centering the dog in addition to the principal zoom effect (see the nose position graphs on the right).
This stems from a secondorder dataset bias.
Indeed, as shown in Fig. 7, BigGAN generates small (zoomed-out) dogs at almost any location within the image, but its generated large (zoomed-in) dogs tend to be centered.
3. A CCOUNTING FOR SECOND - ORDER DATASET BIASES VIA SMALL CIRCLE TRAJECTORIES
Using our set of directions to battle second-order biases is non-trivial, as walking on the sphere towards v while keeping the projections onto all other principal directions ﬁxed is impossible (it induces too many constraints).
However, we note that if we allow the projection onto only one of the other directions, say v ref , to change, then it becomes possible to keep the projections onto all other axes ﬁxed.
Such a trajectory is in fact a small circle on the sphere, that lies in the afﬁne subspace that contains z 0 and is parallel to V = span { v , v ref } .
Speciﬁcally, the small circle walk is given by
[Equation]
where θ = arccos( P V ref z 0 / (cid:107) P V z 0 (cid:107) )
× sign ( (cid:104) P V z 0 , v (cid:105) ) with P V ref = v ref v T ref .
One natural choice for v ref is the principal direction having the smallest singular value, which corresponds to the weakest effect.
As can be seen in the bottom row of Fig.
6, the small circle trajectory with this choice leads to a zoom effect without shift or any other dominant transformation.
This is also illustrated in Fig.
7, which shows the distribution of the horizontal translation between the initial point and the endpoint of the trajectory.
As can be seen, the small circle walk incurs the smallest shift and keeps the FID highest, albeit leading to a slightly smaller zoom effect.
In App.
A.2 we show additional examples, including with different choices of v ref .
A TTRIBUTE TRANSFER
In the previous sections we explicitly computed directions in latent space.
An alternative way of achieving a desired effect, is to transfer attributes from a different image.
As we now show, this can also be achieved without optimization.
Speciﬁcally, in App.
A.2 we show that for BigGAN, principal directions corresponding to different hierarchies control distinctively different attributes.
Now, our key observation is that this allows transferring attributes between images, simply by copying from a target image the part of z corresponding to a particular hierarchy (see Fig.
For example,
[Figure]
C e n t e r
[Figure]
R e l a t i v e H
o r i z o n t a l
S h
i f t
R e l a t i v e H
o r i z o n t a l
S h
i f t
R e l a t i v e H
o r i z o n t a l
S h
i f t
[Figure]
s f e rr e d to transfer pose , we replace the part corresponding to the ﬁrst level.
As seen in Figs.
1 and 8, this allows transferring pose even across classes.
Within the same class, we can transfer color by copying the elements of hierarchies 4,5 and 6 and texture by copying hierarchies 3,4 and 5 (see Appendix for more examples).
Note that unlike other works discussing semantic style hierarchies ( e.g. , (Karras et al., 2019a; Yang et al., 2019)), our pre-trained BigGAN was not trained to disentangle attributes.
C ONCLUSION
We presented methods for determining paths in the latent spaces of pre-trained GANs, which correspond to semantically meaningful transformations.
Our approach extracts those trajectories directly from the generator’s weights, without requiring optimization or training of any sort.
Our methods are signiﬁcantly more efﬁcient than existing techniques, they determine a larger set of distinctive semantic directions, and are the ﬁrst to allow explicitly accounting for dataset biases.
A A PPENDIX A. Q UANTITATIVE EVALUATION
We adopt the method proposed in Jahanian et al.
and utilize the MobileNet-SSD-V1 detectorto estimate object bounding boxes.
To quantify shifts, we extract the centers of the bounding boxes along the corresponding axis.
To quantify zoom we use the area of the bonding boxes.
In the following paragraphs we elaborate on all quantitative evaluations reported in the main text.
Figure 4 Here, we show the probability densities of object areas and locations after 2 (top) and 5 (bottom) steps.
Since we use unit-norm direction vectors, the length of the linear paths we walk through are 2 and 5 as well.
As for the nonlinear path, we choose the ﬁrst step to have the same length.
However, the overall length of the path is different.
For example, on average, ﬁve steps of the nonlinear trajectory have a total length of 5 . 95 , but reach at a point of distance only 4 .
3 from the initial point.
We used 100 randomly chosen classes from the ImageNet dataset, and 30k images from each class.
The same images are used for both the FID measurement and for generating the
PDF plots.
Figure
In order to ensure that each step of the linear walk and the great and small circle walks has the same geodesic distance, we set
[Equation]
where ∆ L , ∆ G and ∆ S are the step sizes of the linear, great circle and small circle walks, respectively.
This ensures that the arc-length of a step on the circles is the same as the length of a step of the linear walk.
Figure
Here, we aim to demonstrate a particular second order dataset bias.
We chose 10 classes which we found to exhibit strong coupling between the size and location of the object.
For example, dogs, cats and in general, animals.
We plotted 80 levels-sets of 2D KDEs computed using the seaborne package.
In Fig.
7 we show results for a Labrador retriever dog, we observed similar results for the classes: golden retriever (207), Welsh springer spaniel (218), Great grey Owl (24),
Persian cat (283), plane (726), tiger (292), Old English sheepdog (229), passenger car (705), goose (99), husky (248).
See Figs.
40 and 41 for additional results.
Table
In Tab. 1, we compare the running time and memory usage of all methods.
For 3 (Jahanian et al., 2020), we measure the time it takes to learn one direction, which includes the training process.
For 4 (H¨ark¨onen et al., 2020), we measure the total time it takes to extract the directions, including the sample collection, the PCA, and the regression.
We noticed that the regression stage was the heaviest.
As for our method, we measure the time it takes the CPU to perform SVD.
The column “Memory” speciﬁes the required memory for collecting samples.
Only GANSpace (H¨ark¨onen et al., 2020) requires that stage.
A. U NSUPERVISED EXPLORATION OF PRINCIPAL DIRECTIONS A.2. C OMPARISONS WITH RANDOM DIRECTIONS
In Fig.
10 - 9 we explore principal directions via linear walks, using the same initial image (in the middle).
In Figs.
14-16 we explore the transformations that arise in each hierarchy of BigGAN-128.
Speciﬁcally, we compare our linear directions which are based on SVD, with random directions.
We draw 5 different directions from an isotropic Gaussian distribution and normalize them to have unit-norms, similarly to our directions.
Then, we linearly add them to the initial latent code with ﬁxed number of steps.
We can observe that each random direction induces a different complex effect, which cannot be described by a single semantic property.
For examples, in the ﬁrst random direction (R1) we can see rotation, zoom and background changes, while in the third (R3), there is a kind of vertical shift.
On the other hand, our principal directions show one prominent transformation for each scale.
We focus on directions that have the same effect for all classes and do not show directions that lead to different effects for different classes, like changes of day-night in one class and background in another class.
A.2. A LTERNATIVE SMALL CIRCLE WALKS
In Figs.
29-33, we show more examples, this time with small circle walks towards principal directions.
In all those examples, the reference direction v ref for the small circle, is the least dominant direction (namely, the singular vector with the smallest singular value).
This ensures that when walking towards the principal direction v , we modify no other dominant property.
That is, we modify the property associated with v without modifying the properties associated with any other principal direction, besides v ref (which is the least dominant one).
In Figs.
29-33 we show some cases in which the initial generated image is not in the middle of the small circle path and therefore in these cases, we need to take a different number of steps to each side.
The endpoints are deﬁned as the points where the cosine in Eq.
11 becomes 0 and 1.
We do not have to choose the reference direction v ref to be the least dominant one.
If we choose it to be a dominant direction, then we may obtain various interesting phenomena, depending on the interaction between the directions v and v ref .
This is illustrated in Figs.
Speciﬁcally, in Fig.and 38, we perform a walk in the direction corresponding to zoom, while allowing only the vertical shift to change.
In this case, the walk manages to center the object so as to achieve a signiﬁcant zoom effect.
In Fig. 39, on the other hand, we perform a walk in the direction corresponding to zoom while allowing only the rotation to change.
Here, the zoom effect is less dominant, but we do see a strong rotation effect.
A.2. S ECOND ORDER DATASET BIASES
In Figs.
40 and 41 we show more examples for second order dataset biases.
Speciﬁcally, those ﬁgures depict the joint distributions of area and horizontal center shift (top) and area and vertical center shift (bottom) at the end of walks that are supposed to induce only zoom-in.
Our small circle walks exhibit the smallest undesired shifts.
A.2. C OMPARISONS WITH GANS PACE
In Fig.
42-46, we show visual comparisons with GANSpace (H¨ark¨onen et al., 2020).
We speciﬁcally focus on the ﬁrst 50 directions founded by each method and show that our linear directions lead to stronger effects for most of the directions.
All directions were scaled to have a unit norm and are linearly added or subtracted from the initial latent code with the same step size.
In Fig.
48, we show that our direction are orthogonal to each other much more then the directions found by (H¨ark¨onen et al., 2020).
A. U SER PRESCRIBED SPATIAL MANIPULATIONS
We provide additional examples for walks corresponding to user prescribed geometric transformations.
We focus on zoom, vertical shift and horizontal shift, and show both linear and our nonlinear trajectories.
A.3. C OMPARISONS WITH J AHANIAN ET AL .
In Figs.
49 we show additional comparisons with Jahanian et al.
A.3. A DDITIONAL RESULTS
In Figs.
50-53 we show additional zoom trajectories and and in Fig.
54, 55 additional shift trajectories.
As can be seen, the linear trajectories often remain more loyal to the original image (at the center) after a small number of steps.
However, for a large number of steps, the nonlinear trajectories lead to more plausible images.
A.3. R ESULTS ON DCGAN
In Figs.
56 - 58, we show results with ResNet based GAN presented in Miyato et al.
That
GAN has a FC layer as the ﬁrst stage, which is all we need in order to perform our spatial manipulations and to extract principal components.
Since that architecture is not an hierarchical one, we can manipulate the ﬁrst layer only.
[Figure]
[Figure] [Figure]
We illustrate the effects of ﬁve random directions
R 1 , . . .
, R 5 (normally distributed and scaled to have unit norms) in the ﬁrst and second scales of
BigGAN, in comparison with our principal directions.
We can see that each random direction leads to different changes, but it is impossible to associate a single dominant property with each direction.
For example, in R5 we can see changes in size, location, and pose.
This is while our directions separate those effects into unique paths.
[Figure] [Figure]
[Figure]
We illustrate the effects of ﬁve random directions
R 1 , . . .
, R 5 (normally distributed and scaled to have unit norms) in the ﬁrst and second scales of
BigGAN, in comparison with our principal directions.
We can see that each random direction leads to different changes, but it is impossible to associate a single dominant property with each direction.
For example, in R5 we can see changes in size, location, and pose.
This is while our directions separate those effects into unique paths.
[Figure] [Figure]
[Figure]
We illustrate the effects of ﬁve random directions
R 1 , . . .
, R 5 (normally distributed and scaled to have unit norms) in the ﬁrst and second scales of
BigGAN, in comparison with our principal directions.
We can see that each random direction leads to different changes, but it is impossible to associate a single dominant property with each direction.
For example, in R5 we can see changes in size, location, and pose.
This is while our directions separate those effects into unique paths.
[Figure] [Figure]
[Figure]
S c a l e 1 , D i rec t i o n
Sh i ft up
[Figure] [Figure]
rec
[Figure]
i
[Figure]
B a c k g r o und m a n i pu l a t i o n ( “ B a c k g r
o und c l u tt er ” )
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
B a c k g r o und m a n i pu l a t i o n s
[Figure]
[Figure]
S c a l e 3 , D i rec t i o n
B l a c k a nd
W h
i t e
S c a l e 3 , D i rec t i o n
B r
i g h t n e ss ( “ s h a d o w ” )
[Figure]
c
[Figure]
l
[Figure]
, D
[Figure]
rec t i o n
z oo m ) a t e x p e n s e o f
d
i rec t i o n 2 ( s h
i ft
Y )
S c a l e 1 , D i rec t i o n 1 ( z oo m ) a t e x p e n s e o f d i rec t i o n 2 ( s h i ft
Y )
S c a l e 1 , D i rec t i o n 1 ( z oo m ) a t e x p e n s e o f d i rec t i o n 3 ( r o t a t e )
H o r i z
o n t a l S h
i f t
H o r i z
o n t a l S h
i f t
H o r i z
o n t a l S h
i f t
H o r i z
o n t a l S h
i f t
Jahanian et al.
GANSpace
Our great circle
Our small circle
V e r t i c a l S h
i f t
V e r t i c a l S h
i f t
V e r t i c a l S h
i f t
V e r t i c a l S h
i f t
H o r i z
o n t a l S h
i f t
H o r i z
o n t a l S h
i f t
Jahanian et al.
GANSpace
Our great circle
Our small circle
H o r i z
o n t a l S h
i f t
H o r i z
o n t a l S h
i f t
V e r t i c a l S h
i f t
V e r t i c a l S h
i f t
V e r t i c a l S h
i f t
V e r t i c a l S h
i f t
[Figure] [Figure]
GANSpace
Our
GANSpace
Our
GANSpace
Our
[Figure]
“ Show
[Figure] [Figure]
AN
Sp a ce
O u r ,
S m a l l
O u r ,
L i n e a r
[Figure]
Shift Y + - Shift Y + -
Shift X + - Shift X + -
Zoom + - Zoom +
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Equation] [Equation]
A. A TTRIBUTE TRANSFER
We next provide more attribute transfer examples.
Figures 59,60 show pose transfer examples, which are obtained by swapping the part of the latent vector corresponding to scale 1.
Figuredepict texture transfer examples, which correspond to swapping the parts of the latent vector and the class, corresponding to scales 3,4,5.
[Equation]
[Figure]
erre d
[Figure]
r a n s f erre d
[Figure]
[Figure] [Figure]