Nonlinear and Commutative Editing in Pretrained GAN Latent Space
Abstract
Semantic editing of images is a fundamental goal of computer vision.
While generative adversarial networks (GANs) are gaining attention for their ability to produce high-quality images, they do not provide an inherent way to edit images semantically.
Recent studies have investigated how to manipulate the latent variable to determine the images to be generated.
However, methods that assume linear semantic arithmetic have limitations in the quality of image editing.
Also, methods that discover nonlinear semantic pathways provide editing that is non-commutative, in other words, inconsistent when applied in diﬀerent orders.
This paper proposes a method for discovering semantic commutative vector ﬁelds.
We theoretically demonstrate that thanks to commutativity, multiple editing along the vector ﬁelds depend only on the quantities of editing, not on the order of the editing.
We also experimentally demonstrated that the nonlinear and commutative nature of editing provides higher quality editing than previous methods.
1. Introduction
The generation and editing of realistic images are one of the fundamental goals in the ﬁeld of computer vision.
Generative adversarial networks (GANs) (Goodfellow et al., 2014) have emerged as a major image generation approach because of the quality of their generated images (Karras et al., 2019, 2020, 2021).
However, GANs do not inherently have a way of semantic image editing.
Several studies aimed to discover a vector corresponding to an attribute of images and to edit images by adding the attribute vector to the latent variables (Voynov and Babenko, 2020; H¨ark¨onen et al., 2020; Shen and Zhou, 2021).
These studies introduce the strong assumption of a linear semantic arithmetic on the latent space (see the ﬁrst column of Table 1), limiting in the quality of image editing.
Other studies have proposed to ﬁnd an attribute vector ﬁeld in the latent space (Tzelepis et al., 2021;
Choi et al., 2022; Ramesh et al., 2018).
These methods edit images by integrating a latent variable along the vector ﬁeld.
This approach seems elegant, but edits of diﬀerent attributes are non-commutative in general.
That is, what we get is diﬀerent when we edit one attribute (denoted by e 1 ) and then edit another (denoted by e 2 ) or when we edit in the reverse order (see the second column of Table 1).
This property becomes problematic when one wants to edit various attributes of the same image repeatedly.
In contrast, linear arithmetic on the latent space ensures that edits of diﬀerent attributes are commutative.
To overcome this dilemma, we propose CurvilinearGANSpace , which discovers a set of commutative and nonlinear attribute vector ﬁelds in pretrained GANs’ latent spaces.
[Table]
2. Methods 2.1. Background
Let X and Z denote an image space and a GAN latent space, respectively.
The generator G of GANs is a mapping from the latent space Z to the image space X ; given a latent variable z ∈ Z , the generator produces an image x ∈ X as x = G ( z ).
We assume the latent space Z to be an N -dimensional space.
Let { z i } Ni =1 denote the coordinate system (i.e., the basis) on a neighborhood of the point z ∈ Z .
Let X denote the set of all vector ﬁelds on the latent space Z .
Let X k ∈ X denote a vector ﬁeld on the latent space Z indexed by k , that is,
X k : Z → T z Z , where T z Z is the tangent space of the latent space Z at the point z .
Then, at the point z , the coordinate system on tangent space T z Z is denoted by { ∂∂z i } Ni =1 , and a vector ﬁeld X k is expressed as X k ( z )
= (cid:80) Ni =
1 X ik ( z ) ∂∂z i for smooth functions
X ik : Z → R .
When considering a method that assumes attribute vector ﬁelds (e.g., Tzelepis et al.
; Choi et al.
; Ramesh et al.
(2018)), an edit of an attribute k of an image x is done by integrating a latent variable z along the corresponding vector ﬁeld X k ; the edited image is given by x (cid:48)
= G ( z (cid:48) ) for z (cid:48)
=
z + (cid:82) t 0 X k
( z ( τ ))
d τ = φ tk ( z ), where φ tk denotes the ﬂow that arises from the vector ﬁeld X k .
Edits of two attributes k and l are commutative if and only if two ﬂows are commutative, that is, φ sl ◦ φ tk
= φ
tk ◦ φ sl for any s, t ∈ R at any point z ∈ Z .
In general, two vector ﬁelds are non-commutative, and hence two edits are noncommutative (Lee, 2012).
A method that assumes linear attribute arithmetic (e.g., Voynov and Babenko (2020); H¨ark¨onen et al.
Shen and Zhou (2021)) can be regarded as a special case.
Using an attribute vector a k independent of the position z , a vector ﬁeld can be deﬁned as X k ( z ) ≡ a k , and then its ﬂow is φ tk ( z ) = (cid:82) t 0 a k d τ = t a k .
Edits are commutative, but the quality of image editing is limited due to the linearity.
2.2. CurvilinearGANSpace
We introduce the following theorem of diﬀerential geometry (see Lee (2012) for example).
Theorem
Let vector ﬁelds X 1 , X 2 , . . .
, X N on an N -dimensional space Z be linearly independent and commutative on an open set U ⊂ Z .
At each z ∈ U , there exists a smooth coordinate chart { ∂ ∂s i } Ni =1 centered at z such that ∂∂s
i
=
X i .
Roughly speaking, a set of linearly independent and commutative vector ﬁelds is compatible with a set of vector ﬁelds along the axes of a coordinate system up to geometric transfor- mation.
Hence, we consider the case where the open set U in Theorem 1 is not a proper subset but equal to the latent space Z .
We prepare an N -dimensional Euclidean space V and name it the Cartesianized latent space.
Its coordinate system { v i } Ni =1 is a global Cartesian coordinate system.
Let e k denote the k -th element of the standard basis, and the vector ﬁled ˜ X
k corresponding to the attribute k is deﬁned as ˜ X k :=
e k for e k := ∂∂v k .
The ﬂow ψ k : R × V → V the arises from the vector ﬁled ˜ X k is given by ψ tk ( v ) :
= v + (cid:82) τ 0 e k d τ = v + t e k .
Obviously, the ﬂows ψ k are commutative because ψ sl ◦ ψ tk ( v ) =
v + t e k
+ s e l = ψ tk ◦ ψ sl ( v ).
We introduce a smooth bijective mapping f : Z → V , z (cid:55)→ v , corresponding to the coordinate chart in Theorem 1.
We deﬁne a ﬂow φ tk that edits the attribute k on the latent space Z as φ tk :=
f − 1 ◦ ψ tk ◦ f .
A vector ﬁeld X k on the latent space Z is implicitly deﬁned by pushforwarding the vector ﬁeld ˜ X k on the Cartesianized latent space V ; in particular, X k ( z ) =
( f − 1 ) ∗ ( ˜ X k ) =
∂f − 1 ( v ) ∂v e k at the point z for v
= f ( z ).
Then, one can generate an edited image x (cid:48) =
G ( z (cid:48) ) using the generator G .
A coordinate system deﬁned by a bijective transformation of a Cartesian coordinate is called a curvilinear coordinate (Arfken et al., 2012).
Hence, we name this method CurvilinearGANSpace.
CurvilinearGANSpace is a commutative special case of method that assumes attribute vector ﬁelds (e.g., Choi et al.
; Ramesh et al.
; Tzelepis et al.
At the same time, it is a nonlinear generalization of method that assumes attribute arithmetic (e.g., Voynov and Babenko (2020);
H¨ark¨onen et al.
Shen and Zhou (2021))
; CurvilinearGANSpace enjoys the advantages of both methods; the nonlinearity and commutativity.
3. Experiments and Results Experimental Settings
The proposed methodology is available for any framework that manipulates the latent variables.
This paper focuses on the unsupervised learning framework proposed by Voynov and Babenko (2020).
We used CelebA-HQ (Liu et al., 2015) as the dataset, StyleGAN2 (Karras et al., 2020) for the GANs, and ResNet-18 (He et al., 2016) for the reconstructor used in the learning framework.
For a smooth bijective mapping f , we employ a continuous normalizing ﬂow (CNF)
(Chen et al., 2018).
For comparison, we also evaluated a method that assumes a linear arithmetic (Voynov and Babenko, 2020) and a method that assumes vector ﬁelds called WarpedGANSpace (Tzelepis et al., 2021).
To clarify the diﬀerence, we will refer to the former method as LinearGANSpace, hereafter.
We used their pretrained models.
Evaluation Metrics
Even when editing an attribute k of a latent variable z by a change amount t , it is not guaranteed that the same attribute k of the image x is edited by the same amount t .
Hence, we normalized the change amount t for the latent variable z by the change amount for the generated image x , following the measurements by a separate attribute predictor A k ( · ).
We used CelebA-HQ attributes classiﬁer for smiling and bangs (Jiang et al., 2021) and Hopenet for face direction (yaw)
(Doosti et al., 2020).
As well as attributes, we used ArcFace for the identity score I ( · , · ), evaluating whether two images are of the same person (Deng et al., 2019).
We deﬁned commutativity error of attributes k + l to evaluate how commutative image editing.
It is the error when edits of these two attributes k and l are applied in diﬀerent orders; namely, | A k ( G ( φ tk ( φ tl ( z ))))
− A k ( G ( φ tl ( φ tk ( z ))))
| and
[Table]
Error
[Table]
LinearGANSpace
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] | A l ( G ( φ tk ( φ tl ( z ))))
− A l ( G ( φ tl ( φ tk ( z ))))
| .
We also deﬁned identity error of attribute k to evaluate the editing quality.
It measures how much an edit of the attribute k reduces the identity score;
namely, 1 − I ( G ( z ) , G ( φ tk ( z ))).
We set the change amount of every edit to t = 0 .
1 for the smiling and bangs attribute and t = 5 degrees for the yaw attribute.
We divided the commutativity errors by the change amount t and showed them in percentages.
Results
We summarized the numerical results in Table 2.
WarpedGANSpace produced the commutativity errors of at least 2.79 % and often more than 10 %.
Those of LinearGANSpace and the proposed CurvilinearGANSpace were always less than 0.5 %; while not exactly zero due to numerical and rounding errors, they are negligible.
Hence, as expected, edits by WarpedGANSpace are not commutative, and edits by LinearGANSpace and CurvilinearGANSpace are commutative.
CurvilinearGANSpace produced the lowest identity errors for smiling and bangs attributes and the second lowest one for the yaw attribute.
Hence, we can say that it learned disentangled representations better.
We showed an example image with a sequence of edits in Fig.
The amount of change was set to double to make the change easier to ﬁnd; +Smiling indicates that t =
+0 . 2 and k = Smiling.
LinearGANSpace’s image editing quality is clearly inferior.
The image editing qualities of WarpedGANSpace and CurvilinearGANSpace are competitive.
After six edits, the change amounts should cancel out, and the attributes should return to their original values.
LinearGANSpace and CurvilinearGANSpace show the expected results.
However, the edited result of WarpedGANSpace shows that the woman’s mouth opening has not returned to its original state; edits by WarpedGANSpace are not commutative.
Therefore, we conclude that CurvilinearGANSpace enjoys the advantages of both of previous methods; the nonlinearity and commutativity.
This study was partially supported by JST PRESTO (JPMJPR21C7), JST CREST (JPMJCR1914), and JSPS KAKENHI (19H04172, 19K20344), Japan.