D O N OT E SCAPE F ROM THE M ANIFOLD : D ISCOVERING THE L OCAL C OORDINATES ON THE L ATENT S PACE OF GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
The discovery of the disentanglement properties of the latent space in GANs motivated a lot of research to ﬁnd the semantically meaningful directions on it.
In this paper, we suggest that the disentanglement property is closely related to the geometry of the latent space.
In this regard, we propose an unsupervised method for ﬁnding the semantic-factorizing directions on the intermediate latent space of
GANs based on the local geometry.
Intuitively, our proposed method, called Local Basis , ﬁnds the principal variation of the latent space in the neighborhood of the base latent variable.
Experimental results show that the local principal variation corresponds to the semantic factorization and traversing along it provides strong robustness to image traversal.
Moreover, we suggest an explanation for the limited success in ﬁnding the global traversal directions in the latent space, especially W -space of StyleGAN2.
We show that W -space is warped globally by comparing the local geometry, discovered from Local Basis, through the metric on Grassmannian Manifold.
The global warpage implies that the latent space is not well-aligned globally and therefore the global traversal directions are bound to show limited success on it.
I NTRODUCTION
Generative Adversarial Networks (GANs, Goodfellow et al.
(2014)), such as ProGAN (Karras et al., 2018), BigGAN (Brock et al., 2018), and StyleGANs (Karras et al., 2019; 2020b;a), have shown tremendous performance in generating high-resolution photo-realistic images that are often indistinguishable from natural images.
However, despite several recent efforts (Goetschalckx et al., 2019;
Jahanian et al., 2019; Plumerault et al., 2020;
Shen et al., 2020) to investigate the disentanglement properties (Bengio et al., 2013) of the latent space in GANs, it is still challenging to ﬁnd meaningful traversal directions in the latent space corresponding to the semantic variation of an image.
The previous approaches to ﬁnd the semantic-factorizing directions are categorized into local and global methods.
The local methods (e.g. Ramesh et al.
, Latent Mapper in StyleCLIP (Patashnik et al., 2021), and attribute-conditioned normalizing ﬂow in StyleFlow (Abdal et al., 2021)) suggest a sample-wise traversal direction.
By contrast, the global methods, such as GANSpace (H¨ark¨onen et al., 2020) and SeFa (Shen & Zhou, 2021), propose a global direction for the particular semantics (e.g. glasses, age, and gender) that works on the entire latent space.
Throughout this paper, we refer to these global methods as the global basis .
These global methods showed promising results.
However, these methods are successful on the limited area, and the image quality is sensitive to the perturbation intensity.
In fact, if a latent space does not satisfy the global disentanglement property itself, all global methods are bound to show a limited performance on it.
Nevertheless, to the best of our knowledge, the global disentanglement property of a latent space has not been investigated except for the empirical observation of generated samples.
In this regard, we need a local method that describes the local disentanglement property and an evaluation scheme for the global disentanglement property from the collected local information.
In this paper, we suggest that the semantic property of the latent space in GANs (i.e. disentanglement of semantics and image collapse) is closely related to its geometry, because of the sample-wise optimization nature of GANs.
In this respect, we propose an unsupervised method to ﬁnd a traver- sal direction based on the local structure of the intermediate latent space W , called Local Basis (Fig 1a).
We approximate W with its submanifold representing its local principal variation, discovered in terms of the tangent space T w W .
Local Basis is deﬁned as an ordered basis of T w W corresponding to the approximating submanifold.
Moreover, we show that Local Basis is obtained from the simple closed-form algorithm, that is the singular vectors of the Jacobian matrix of the subnetwork.
The geometric interpretation of Local Basis provides an evaluation scheme for the global disentanglement property through the global warpage of the latent manifold.
Our contributions are as follows:
We propose Local Basis, a set of traversal directions that can reliably traverse without escaping from the latent space to prevent image collapse.
The latent traversal along Local Basis corresponds to the local coordinate mesh of local-geometry-describing submanifold.
We show that Local Basis leads to stable variation and better semantic factorization than global approaches.
This result veriﬁes our hypothesis on the close relationship between the semantic and geometric properties of the latent space in GANs.
We propose Iterative Curve-Traversal method, which is a way to trace the latent space in the curved trajectory.
The trajectory of the images with this method shows a more stable variation compared to the linear traversal.
We introduce the metrics on the Grassmannian manifold to analyze the global geometry of the latent space through Local Basis.
Quantitative analysis demonstrates that the W -space of StyleGAN2 is still globally warped.
This result provides an explanation for the limited success of the global basis and proves the importance of local approaches.
R ELATED W ORK
Style-based Generators.
In recent years, GANs equipped with style-based generators (Karras et al., 2019; 2020b) have shown state-of-the-art performance in high-ﬁdelity image synthesis.
The style-based generator consists of two parts: a mapping network and a synthesis network.
The mapping network encodes the isotropic Gaussian noise z ∈ Z to an intermediate latent vector w ∈ W .
The synthesis network takes w and generates an image while controlling the style of the image through w .
Here, W -space is well known for providing a better disentanglement property compared to Z (Karras et al., 2019).
However, there is still a lack of understanding about the effect of latent perturbation in a speciﬁc direction on the output image.
Latent Traversal for Image Manipulation.
The impressive success of GANs in producing highquality images has led to various attempts to understand their latent space.
Early approaches (Radford et al., 2016; Upchurch et al., 2017) show that vector arithmetic on the latent space for the semantics holds, and StyleGAN (Karras et al., 2019) shows that mixing two latent codes can achieve style transfer.
Some studies have investigated the supervised learning of latent directions while assuming access to the semantic attributes of images (Goetschalckx et al., 2019; Jahanian et al., 2019;
Shen et al., 2020; Yang et al., 2021; Abdal et al., 2021).
In contrast to these supervised methods, some recent studies have suggested novel approaches that do not use the prior knowledge of training dataset, such as the labels of human facial attributes.
In Voynov & Babenko (2020), an unsupervised optimization method is proposed to jointly learn a candidate matrix and a corresponding reconstructor, which identiﬁes the semantic direction in the matrix.
GANSpace (H¨ark¨onen et al., 2020) ﬁnds a global basis for W in StyleGAN using a PCA, enabling a fast image manipulation.
SeFa (Shen &
Zhou, 2021) focuses on the ﬁrst weight parameter right after the latent code, suggesting that it contains essential knowledge of an image variation.
SeFa proposes singular vectors of the ﬁrst weight parameter as meaningful global latent directions.
StyleCLIP (Patashnik et al., 2021) achieves a stateof-the-art performance in the text-driven image manipulation of StyleGAN.
StyleCLIP introduces an additional training to minimize the CLIP loss (Radford et al., 2021).
Jacobian Decomposition.
Some works use the Jacobian matrix to analyze the latent space of GAN (Zhu et al., 2021; Wang & Ponce, 2021; Chiu et al., 2020; Ramesh et al., 2018). However, these methods focus on the Jacobian of the entire model, from the input noise z to the output image.
Ramesh et al.
(2018) suggested the right singular vectors of the Jacobian as local disentangled directions in the Z space.
Zhu et al.
(2021) proposed a latent perturbation vector that can change only a particular area of the image.
The perturbation vector is discovered by taking the principal vector of
[Figure]
the Jacobian to the target area and projecting it into the null space of the Jacobian to the complementary region. On the other hand, our Local Basis utilizes the Jacobian matrix of the partial network, from the input noise to the intermediate latent code w , and investigates the black- intermediate latent space from it.
The top- Local Basis corresponds to the best local geometry submanifolds.
This intuition leads to exploiting Local Basis to assess the global geometry of the space.
T RAVERSING A CURVED LATENT SPACE
In this section, we introduce a method for ﬁnding a local-geometry-aware traversal direction in the intermediate latent space W .
The traversal direction is refered to as the Local Basis at w ∈ W .
In addition, we evaluate the proposed Local Basis by observing how the generated image changes as we traverse the intermediate latent variable.
Throughout this paper, we assess Local Basis of the Wspace in StyleGAN2 (Karras et al., 2020b).
However, our methodology is not limited to StyleGAN2.
See appendix for the results on StyleGAN (Karras et al., 2019) and BigGAN (Brock et al., 2018).
3. F INDING A L OCAL B ASIS
Given a pretrained GAN model M : Z → X , from the input noise space Z to the image space
X , we choose the intermediate layer ˜ W to discover Local Basis.
We refer to the former part of the
GAN model as the mapping network f : Z → ˜ W .
The image of the mapping network is denoted as
W = f ( Z ) ⊂ ˜ W .
The latter part, a non-linear mapping from ˜ W to the image space X , is denoted by G : ˜ W → X .
Local Basis at w ∈ W is deﬁned as the basis of the tangent space T w W .
This basis can be interpreted as a local-geometry-aware linear traversal direction starting from the given w .
To deﬁne the tangent space of the intermediate latent space W properly, we assume that W is a differentiable manifold .
Note that the support of the isotropic Gaussian prior Z = R d Z and the ambient space ˜ W = R d ˜ W are already differentiable manifolds.
The tangent space at w , denoted by
T w W , is a vector space consisting of tangent vectors of curves passing through point w .
Explicitly,
[Equation]
Then, the differentiable mapping network f gives a linear map df z between the two tangent spaces
T z Z and T w ˜ W where w = f ( z ) .
[Equation]
We utilize the linear map df z , called the differential of f at z , to ﬁnd the basis of T w W .
Based on the manifold hypothesis in representation learning, we posit that the latent space of the image space X in ˜ W is a lower-dimensional manifold embedded in W .
In this approach, we estimate the latent manifold as a lower-dimensional approximation of W describing its principal variations.
The approximation manifold can be obtained by solving the low-rank approximation problem of df z .
The manifold hypothesis is supported by the empirical distribution of singular values σ z
i .
The analysis is provided in Fig 9 in the appendix.
The low-rank approximation problem has an analytic solution deﬁned by Singular Value Decomposition (SVD).
Because the matrix representation of df z is a Jacobian matrix ( ∇ z f )( z )
∈ R d ˜ W ×
d Z ,
Local Basis is obtained as the following: For the i -th right singular vector u z
i ∈
R
d Z , i -th left singular vector v w i ∈ R d ˜ W , and i -th singular value σ z i ∈ R of ( ∇ z f )( z ) with σ z 1 ≥ · · · ≥ σ z n ,
[Equation]
Then, the k -dimensional approximation of W around w is described as the following because Z =
R d Z (if σ z k > 0 ).
Note that W k w is a submanifold 1 of W corresponding to the k components of
Local Basis, i.e. T w W k w = span { v w i : 1 ≤ i ≤ k } .
[Equation]
Locally afﬁne mapping network In this paragraph, we focus on the locally afﬁne mapping network f , which is one of the most widely adopted GAN structures, such as MLP or CNN layers with
ReLU or leaky-ReLU activation functions.
This type of mapping network has several well-suited properties for Local Basis.
[Equation]
where Ω denotes a partition of Z , and A p and b p are the parameters of the local afﬁne map.
With this type of mapping network f , it is clear that the intermediate latent space W satisﬁes a differentiable manifold property at least locally on the interior of each p ∈ Ω .
The region where the property may not hold, the intersection of several closure of p ’s in Ω , has measure zero in Z .
Moreover, the Jacobian matrix ( ∇ z f )( z ) becomes a locally constant matrix.
Then, the approximating manifold W k w (Eq 5) satisﬁes the submanifold condition, and is consistent locally for each p , avoiding being deﬁned for each w .
In addition, the linear traversal of the latent variable w along v w i can be described as the curve on W (Eq 7).
Most importantly, these curves on W (Eq 7), starting from w in the direction of Local Basis, corresponds to the local coordinate mesh of W k w .
[Equation]
Equivalence to Local PCA To provide additional intuition about Local Basis, we prove the following proposition.
The proposition shows that Local Basis is equivalent to applying a PCA on the samples on W around w .
Proposition 1 (Equivalence to Local PCA) .
Consider the Local PCA problem around the base latent variable w b = f ( z b ) on W , i.e. PCA of the latent variable samples w (cid:48) around w b . w (cid:48)
=
T 1 f ( z b + c · (cid:15) ) with (cid:15) ∼ N (0 , I ) and for some small c > 0 .
[Equation]
where T 1 f ( z ) = w b
+ ( ∇ z b f )( z − z b ) is the linear approximation of f around z b .
Then, the principal components discovered in the Local PCA problem are equivalent to Local Basis at w b .
3. I TERATIVE C URVE -T RAVERSAL
We suggest a natural curve-traversal that can keep track of the W -manifold and an iterative method to implement it.
We divide the long curved trajectory into small pieces and approximate each piece by the local curves using Local Basis.
We call this curve-traversal Iterative Curve-Traversal method.
Consistent with the linear traversal method, we consider Iterative Curve-Traversal γ departing in the direction of a Local Basis.
Explicitly, for a sufﬁciently large c > 0 ,
[Equation]
where { v w i } i = Local Basis ( w ) .
We split the curve-traversal γ into N pieces γ n and denote each n -th iterate in W and Z as w n and z n for 1 ≤ n ≤ N .
The starting point of the traversal is denoted as the 0 -th iterate w = w 0 , z
= z 0 , and w 0 = f
( z 0 ) .
(Fig 2) Note that to ﬁnd Local Basis at w n , we need a corresponding z n ∈ Z such that w n = f ( z n ) .
Below, we describe the positive part γ + = γ | [0 ,c ) of Iterative Curve-Traversal.
For the negative part, we repeat the same procedure using the reversed tangent vector − v w k .
The ﬁrst step γ +1 of Iterative
Curve-Traversal method with perturbation intensity I is as follows:
[Equation]
[Equation]
Note that w 1 is the endpoint of the curve γ +1 and ˙ γ +1 (0)
=
v w k .
We scale the step size in Z by 1 /σ
z
k to ensure each piece of curve has a similar length of ( I/N ) .
To preserve the variation in semantics during the traversal, the departure direction of γ +2 is determined by comparing the similarity between the previous departure direction v w 0 k and Local Basis at w 1 .
The above process is repeated N -times.
(The algorithm for Iterative Curve-Traversal can be found in the appendix.)
[Equation]
where j = argmax 1 ≤ i ≤ d W | [Equation] w 0 k , v w [Equation]
|
[Equation]
R ESULTS OF L OCAL B ASIS TRAVERSAL
We evaluate Local Basis by observing how the generated image changes as we traverse W -space in
StyleGAN2 and by measuring FID score for each perturbation intensity.
The evaluation is based on two criteria:
Robustness and Semantic Factorization.
Robustness Fig 3 and 4 present the Robustness Test results 2 .
In Fig 3, the traversal image of Local
Basis is compared with those of the global methods (GANSpace (H¨ark¨onen et al., 2020) and SeFa (Shen & Zhou, 2021)) under the strong perturbation intensity of 12 along the 1 st and 2 nd direction of each method.
The perturbation intensity is deﬁned as the traversal path length in W .
The two global methods show severe degradation of the image compared to Local Basis.
Moreover, we perform a quantitative assessment of robustness.
We measure the FID score for 10,000 traversed images for each perturbation intensity.
In Fig 4, the global methods show the relatively small FID under the small perturbation.
But, as we impose the stronger perturbation, the FID scores on the global methods increase sharply, implying the image collapse in Fig 3.
By contrast, Local Basis achieves much smaller FID scores with and without Iterative Curve-Traversal. I = I = I =
I =
I = I = I = [Figure]
I =
I =
I =
I =
I =
I =
I = [Figure] [Figure] We interpret the degradation of image as due to the deviation of trajectory from W .
The theoretical interpretation shows that the linear traversal along Local Basis corresponds to a local coordinate axis on W , at least locally.
Therefore, the traversal along Local Basis is guaranteed to stay close to W even under the longer traversal.
However, we cannot expect the same property on the global basis because it is based on the global geometry.
Iterative Curve-Traversal shows more stable traversal because of its stronger tracing to the latent manifold.
This further supports our interpretation.
Semantic Factorization Local Basis is discovered in terms of singular vectors of df z .
The disentangled correspondence, between Local Basis and the corresponding singular vectors in the prior space, induces a semantic-factorization in Local Basis.
Fig 5 and 6 presents the semantics of the image discovered by Local Basis.
In Fig 5, we compare the semantic factorizations of Local Basis and GANSpace (H¨ark¨onen et al., 2020) for the particular semantics discovered by GANSpace.
For each interpretable traversal direction of GANSpace provided by the authors, the corresponding
Local Basis is chosen by the one with the highest cosine similarity.
For a fair comparison, each traversal is applied to the speciﬁc subset of layers in the synthesis network (Karras et al., 2020b) provided by the authors of GANSpace with the same perturbation intensity.
In particular, as we impose the stronger perturbation (from left to right), GANSpace shows the image collapse in Fig 5a and entanglement of semantics ( Glasses + Head Raising ) in Fig 5d. However, Local Basis does not show any of those problems.
Fig 6 provides additional examples of semantic factorization where the latent traversal is applied to a subset of layers predeﬁned in StyleGAN.
The subset of the layers is selected as one of four, i.e. coarse , middle , ﬁne , or all styles.
Local Basis shows decent factorization of semantics such as Body Length of car and Age of cat in LSUN (Yu et al., 2015) in Fig 6.
[Figure] [Figure]
appendix additional examples
Semantic Factorization layer restriction.)
3. E XPLORATION INSIDE A BSTRACT S EMANTICS
Abstract semantics of image often consists of several lower-level semantics.
For instance, Old can be represented as the correlated distribution of Hair color , Wrinkle , Face length , etc.
In this section, we show that the adaptation of Iterative Curve-Traversal can explore the variation of abstract semantics, which is represented by the cone-shaped region of the generative factors (Tr¨auble et al., 2021).
Because of its text-driven nature, we utilize the global basis 3 from StyleCLIP (Patashnik et al., 2021) corresponding to the abstract semantics of Old .
Then, we consider the modiﬁcation of Iterative
Curve-Traversal following the given global basis v global .
To be more speciﬁc, the departure direction of each piece of curve γ i in Eq 12 is chosen by the similarity to v global , not by the similarity to previous departure direction.
The results for old are provided in Fig 7.
(See the appendix for other examples.)
Step size denotes the length of each piece of curve, i.e. ( I/N ) in Sec 3.2.
For a fair comparison, the overall perturbation intensity I is ﬁxed to 4 by adjusting the number of steps N .
The linear traversal along the global basis adds only wrinkles to the image and the image collapses shortly.
On the contrary, both Iterative Curve-Traversal methods obtain the diverse and high-ﬁdelity image manipulation for the target semantics old .
In particular, the diversity is greatly increased as we add stochasticity to the step size.
We interpret this diversity as a result of the increased exploration area from the stochastic step size while exploiting the high-ﬁdelity of Iterative Curve-Traversal.
E VALUATING WARPAGE OF W -M ANIFOLD
In this section, we provide an explanation for the limited success of the global basis in W -space of StyleGAN2.
In Sec 3, we showed that Local Basis corresponds to the generative factors of data. Hence, the linear subspace spanned by Local Basis, which is the tangent space T w W k w in Eq 5, describes the local principal variation of image. In this regard, we assess the global disentanglement property by evaluating the consistency of the tangent space at each w ∈ W . We refer to the inconsistency of the tangent space as the warpage of the latent manifold.
Our evaluation proves that W -manifold is warped globally.
In this section, we present the quantitative evaluation of the global warpage by introducing the Grassmannian Metric.
The qualitative evaluation by observing the subspace traversal is provided in the appendix.
The subspace traversal denotes a simultaneous traversal in multiple directions.
Grassmannian Manifold Let V be the vector space.
The Grassmannian manifold Gr ( k, V ) (Boothby, 1986) is deﬁned as the set of all k -dimensional linear subspaces of V .
We evaluate the global warpage of W -manifold by measuring the Grassmannian distance between the linear subspaces spanned by top- k Local Basis of each w ∈ W .
The reason for measuring the distance for top- k Local Basis is the manifold hypothesis.
The linear subspace spanned by top- k Local Basis corresponds to the tangent space of the k -dimensional approximation of W (Eq 5).
From this perspective, a large Grassmannian distance means that the k -dimensional local approximation of W severely changes.
Likewise, we consider the subspace spanned by the top- k components for the global basis.
In this study, two types of metrics (i.e. Projection metric and Geodesic metric) are adopted as metrics of the Grassmannian manifold.
Grassmannian Metric First, for two subspaces W, W (cid:48)
∈
Gr ( k, V ) , let the projection into each subspace be P W and P W (cid:48) , respectively.
Then the Projection Metric (Karrasch, 2017) on Gr ( k, V ) is deﬁned as follows. [Equation]
[Equation]
where || · || denotes the operator norm.
Second, let M W , M W (cid:48)
∈
R d V × k be the column-wise orthonormal matrix of which columns span
W, W (cid:48)
∈
Gr ( k, V ) , respectively.
Then, the Geodesic Metric (Ye & Lim, 2016) on Gr ( k, V ) , which is induced by canonical Riemannian structure, is formulated as follows. [Equation]
[Equation]
) where θ i = cos − 1 ( σ i ( M (cid:62) W M W (cid:48) )) denotes the i -th principal angle between W and W (cid:48) .
[Figure] Evaluation We evaluate the global warpage of the W -manifold by comparing the ﬁve distances as we vary the subspace dimension.
Random O ( d ) :
Between two random basis of R d W uniformly sampled from O ( d W ) 2.
Random w : Between two Local Basis from two random w ∈ W 3.
Close w : Between two Local Basis from two close w (cid:48) , w ∈ W (See appendix for the Grassmannian metric with various (cid:15) = | z (cid:48) − z | .)
[Equation]
To global GANSpace : Between Local Basis and the global basis from GANSpace 5.
To global SeFa : Between Local Basis and the global basis from SeFa
Fig 8 shows the above ﬁve Grassmannian metrics.
We report the metric results from 100 samples for the Random O ( d W ) and 1,000 samples for the others.
The Projection metric increases in order of
Close w , To global GANSpace, Random w , To global SeFa, and Random O ( d W ) .
For the Geodesic metric, the order is reversed for Random w and To global SeFa .
Most importantly, the Random w metric is much larger than Close w .
This shows that there is a large variation of Local Basis on W , which proves that W -space is globally warped.
In addition, Close w metric is always signiﬁcantly smaller than the others, which implies the local consistency of Local Basis on W .
Finally, the metric results prove the existence of limited global disentanglement on W .
Random w is smaller than Random O ( d ) .
This order shows that Local Basis on W is not completely random, which implies the existence of a global alignment.
In this regard, both To global results prove that the global basis ﬁnds the global alignment to a certain degree.
To global GANSpace lies in between
Close w and Random w .
To global SeFa does so on the Geodesic metric and is similar to Random w on the Projection metric.
However, the large gap between Close w and both To global implies that the discovered global alignment is limited.
C ONCLUSION
In this work, we proposed a method for ﬁnding a meaningful traversal direction based on the localgeometry of the intermediate latent space of GANs, called Local Basis.
Motivated by the theoretical explanation of Local Basis, we suggest experiments to evaluate the global geometry of the latent space and an iterative traversal method that can trace the latent space.
The experimental results demonstrate that Local Basis factorizes the semantics of images and provides a more stable transformation of images with and without the proposed iterative traversal.
Moreover, the suggested evaluation of the W -space in StyleGAN2 proves that the W -space is globally distorted.
Therefore, the global method can ﬁnd a limited global consistency from W -space.
E THICS S TATEMENT
The limitations and the potential negative societal impacts of our work are that Local Basis would reﬂect the bias of data.
The GANs learn the probability distribution of data through samples from it.
Thus, unlike the likelihood-based method such as Variational Autoencoder (Kingma & Welling, 2014) and Flow-based models (Kingma & Dhariwal, 2018) the GANs are more likely to amplify the dependence between the semantics of data, even the bias of it.
Because Local Basis ﬁnds a meaningful traversal direction based on the local-geometry of latent space, Local Basis would show the bias of data as it is.
Moreover, if Local Basis is applied to real-world problems like editing images, Local Basis may amplify the bias of society.
However, in order to ﬁx a problem, we have to ﬁnd a method to analyze it.
In this respect, Local Basis can serve as a tool to analyze the bias.
R EPRODUCIBILITY S TATEMENT
To ensure the reproducibility of this study we attached the entire source code in the supplementary material.
Every ﬁgure can be reproduced by running the jupyter notebooks in notebooks/*.
In addition, the proof of Proposition 1 is included in the appendix.
A P ROPOSITION PROOF
Denote ( ∇ z b f ) by J .
Then, from w (cid:48)
=
T 1 f ( z
[Equation]
+ c
[Equation] [Equation]
The ﬁrst principal component v 1 is the vector such that v (cid:124) 1 ( c · J(cid:15) ) has the maximum variance.
[Equation]
Therefore,
[Equation]
Clearly, v 1 corresponds to the ﬁrst right singular vector of J (cid:124) , i.e. the ﬁrst left singular vector of J , from the linear operator norm maximizing property of singular vectors.
Inductively, k -th principal component v k is the vector such that
[Equation]
where
[Equation]
Thus, v k becomes the k -th left singular vector of J .
Therefore, the principal components from the
Local PCA problem are equivalent to Local Basis at w b .
B A LGORITHM
Input: 1: z
R is the input code.
is the mapping network.
Output: L OCAL B ASIS ( z, f ) 3: w ← f ( z ) 4: J ∈ R d W × d
← J ACOBIAN (
, w ) 5: U, S, V ← SVD ( J ) 6: return { U
V } c
S C M ODEL AND C OMPUTATION R ESOURCE D ETAILS
Model We evaluate GANSpace (H¨ark¨onen et al., 2020), SeFa (Shen & Zhou, 2021), and Local
Basis on StyleGAN2 models for FFHQ (Karras et al., 2019) and LSUN (Yu et al., 2015) provided by the authors (Karras et al., 2020b).
Computation Resource We generated Latent traversal results on the environment of TITAN RTX with Intel(R)
Xeon(R)
Gold 5220 CPU @
2.20GHz.
However, it requires a low computational cost to get a Local Basis.
For example, on the environment of GTX 1660 with Ryzen 5 2600, computing a Local Basis takes about 0.05 seconds.
D C ODE L ICENSE
The ﬁles models/wrappers.py, notebooks/ganspace utils.py and notebooks/notebook utils.py are a derivative of the GANSpace, and are provided under the Apache 2.0 license.
The directory netdissect is a derivative of the GAN Dissection project, and is provided under the MIT license.
The directories models/biggan and models/stylegan2 are provided under the MIT license.
E D ISTRIBUTION OF S IGULAR V ALUES OF J ACOBIAN
[Figure]
The random matrix is sampled from the Gaussian distribution, then transformed to have the mean and standard deviation of the 100 Jacobian matrix.
The sharp peak around zero demonstrates that most of the linear perturbation from z collapses.
This observation proves our manifold hypothesis.
To better represent the sparsity of singular values, we provide the histogram of sigular values σ z i with 0 ≤ σ z i ≤ 2 separately.
[Figure]
[Figure] [Figure]
[Figure]
H I MPLEMENTATION D ETAILS FOR S EC 3.
In Sec 3.4, we utilize the global basis from StyleCLIP (Patashnik et al., 2021) deﬁned on W + , the layer-wise extension of W introduced in (Abdal et al., 2019; Patashnik et al., 2021).
To be more speciﬁc, since the synthesis network in StyleGAN has 18 layers, we obtain an extended latent code w + ∈ W + deﬁned by the concatenation of latent codes w i ∈ W of dimension 512 for each
i -th layer and it can be described as follows:
[Equation]
Note that our Iterative Curve-Traversal originally deﬁned on W has a canonical extension to W + without additional changes in structures or methodologies.
For implementing the stochastic Iterative Curve-Traversal introduced in Sec 3.4, we ﬁrstly ﬁnd a global basis on W + using StyleCLIP (Patashnik et al., 2021) which implies a given semantic attribute in the form of text (e.g. old).
Now denote the global basis by v + global , which can be represented as follows:
[Equation]
[Equation]
Then we perform the (extended) Iterative Curve-Traversal following v + global , equipped with a stochastic movement for each step.
In practice, we consider two options to choose a traversal direction for each step; First, follow the direction most similar to the previously selected basis (as
Algorithm 2), except for the ﬁrst iteration.
Note that we compute the similarity between the local basis and the global basis only at once when choosing the ﬁrst traversal direction.
Second, follow the direction most similar to the given global basis.
This is slightly different from our Algorithm 2, however, we empirically verify that setting the exploration in that way leads to a more desirable image change.
Fig 14 shows that the ﬁrst method still preserves the image quality well, but it does not guarantee that the desired direction of image change, namely ‘old’.
We speculate the reason why such phenomenon occurs is that most of the information contained in the meaningful global basis disappears after the ﬁrst step (a unique, direct comparison to the global basis), although our methodology guarantees that the latent code does not escape from the manifold and achieve a high image quality.
Nevertheless,
Fig 15 shows that the second method for the stochastic Iterative Curve-Traversal can change a given facial image in a very high quality and various ways.
[Figure]
I S UBSPACE T RAVERSAL
In Section 4, we proved that the W -space in StyleGAN2 is warped globally.
Speciﬁcally, the subspace of traversal direction generating principal variation in the image changes severely as we vary the starting latent variable w .
To verify the claim further, we visualize the subspace traversal on the latent space W .
The subspace traversal denotes a simultaneous traversal in multiple directions.
In this paper, we visualize the two-dimensional traversal,
[Equation]
where w = f ( z ) and G denotes a subnetwork of the given GAN model from W to the images space
X .
Since the disentanglement into the linear subspace implies the commutativity of transformation (Pfau et al., 2020), the subspace traversal can be a more challenging version of linear traversal experiments.
Fig 16 and Fig 17 show results of the subspace traversal for the global basis and Local Basis.
Starting from the center, the horizontal and vertical traversals correspond to the 1 st and 2 nd directions of each method.
The same perturbation intensity per step is applied for both directions.
When restricted to the linear traversal (red and green box), the GANSpace shows relatively stable traversals.
However, the traversal image deteriorates at the corner of the subspace traversal.
By contrast, Local Basis shows a stable variation during the entire subspace traversal.
This result proves that the global basis is not well-aligned with the local-geometry of the W manifold.
[Figure]
J L OCAL B ASIS ON O THER MODELS
[Figure] [Figure]