D ISENTANGLED GAN S FOR C ONTROLLABLE G ENERATION OF H IGH -R ESOLUTION I MAGES
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative adversarial networks (GANs) have achieved great success at generating realistic samples.
However, achieving disentangled and controllable generation still remains challenging for GANs, especially in the high-resolution image domain.
Motivated by this, we introduce AC-StyleGAN , a combination of
AC-GAN and StyleGAN, for demonstrating that very limited supervision is sufﬁcient to beneﬁt the generic disentanglement learning of high-resolution images.
Speciﬁcally, only using 5% of the labelled data signiﬁcantly improves the disentanglement quality, as compared to the unsupervised alternatives.
Inspired by the observed separation of ﬁne and coarse styles in StyleGAN, we then extend
AC-StyleGAN to a new image-to-image model called FC-StyleGAN for semantic manipulation of ﬁne-grained factors in a high-resolution image.
In experiments, we show that FC-StyleGAN performs well in only controlling ﬁne-grained factors, with the use of instance normalization, and also demonstrate its good generalization ability to unseen images.
Finally, we create two new datasets – Falcor3D and Isaac3D with higher resolution, more photorealism, and richer variation, as compared to existing disentanglement datasets.
I NTRODUCTION
High-resolution controllable generation is an important component in many applications, such as image editing (Yao et al., 2018), 3D scene understanding (Eslami et al., 2018) and inverse graphics (Kulkarni et al., 2015).
Generative adversarial networks (GANs) (Goodfellow et al., 2014) have achieved great success at generating realistic images, of which two representatives are StyleGAN (Karras et al., 2019) for unconditional generation and BigGAN
(Brock et al., 2018) for class conditional generation.
However, the controllable generation is still a challenge for state-of-the-art
GANs, especially with high-resolution images.
For instance, StyleGAN cannot be directly used to synthesize high-ﬁdelity human faces by speciﬁcally controlling skin color or eye size without affecting other face attributes.
Also, BigGAN is not able to control the hair color or length of a dog image without changing other features.
Disentanglement of various factors allows us to independently control the variations across all the factors.
But this is not easy to learn in GANs without further modiﬁcations such as adding regularization to encourage better disentanglement.
For example, InfoGAN (Chen et al., 2016) proposes maximizing mutual information between latent code and its reconstruction for an unsupervised disentanglement.
Most recently, Locatello et al. (2019a) has showed that unsupervised disentanglement learning is impossible without model inductive bias or supervision .
In that light, several recent works attempt to learn disentanglement in GANs by introducing a strong model bias (Nguyen-Phuoc et al., 2019).
However, these works are restricted to a speciﬁc domain (e.g. 3D rotations only) and usually difﬁcult to scale up to higher resolution, mainly due to the intrinsic limitations of the model biases themselves.
This motivates us to study the impact of limited supervision in a generic disentanglement learning with GANs, especially in the more challenging high-resolution image domain.
Main contributions:
By combining the advantages of AC-GAN (Odena et al., 2017) and StyleGAN, we introduce AC-StyleGAN, and demonstrate that, very limited supervision is sufﬁcient: just 5% of the labelled data signiﬁcantly improves the disentanglement quality, as compared to the unsupervised alternatives.
However, if we are only interested in controlling a subset of factors (effectively treating the rest as random unobserved nuisance variables), we ﬁnd that the disentanglement qual- ity of AC-StyleGAN degrades signiﬁcantly.
We believe this is due to the latent nuisance factors strongly confounding the observed factors, a common and difﬁcult problem in high-dimensional partially observed latent variables models (Bishop, 1998).
To address this, we propose FC-StyleGAN, a new image-to-image model that adds controllable ﬁnegrained factors along with a super-resolution process.
This is inspired by the separation of ﬁne and coarse styles, observed in StyleGAN.
FC-StyleGAN enables semantic manipulation of ﬁne styles in high-resolution images, without a commonly-used encoder-decoder structure.
Finally, we create two new high-quality datasets – Falcor3D and Isaac3D – that present a new challenge for controllable generation in terms of image resolution, photorealism, and richness of style factors, as compared to existing disentanglement datasets such as 3D Chairs (Aubry et al., 2014), dSprite (Matthey et al., 2017) and MPI3D (Gondal et al., 2019).
Thus, we propose new semi-supervised GAN architectures that enable a generic disentanglement learning of high-resolution images as well as new high-quality disentanglement datasets.
R ELATED W ORK Disentanglement learning
Learning generic disentangled representations in an unsupervised way has attracted a lot of attention.
Two representative models are InfoGAN (Chen et al., 2016) and β -VAE (Higgins et al., 2017).
The nature of unsupervised disentanglement learning does not guarantee that the learned disentangled factors are semantically meaningful without an additional inductive bias or supervision (Locatello et al., 2019a; Nguyen-Phuoc et al., 2019; Locatello et al., 2019b).
Another line of work aims to learn disentangled representations via supervised learning wherein factors are observed variables (Kulkarni et al., 2015;
Reed et al., 2014; Xiao et al., 2018;
Locatello et al., 2019b).
However, little work has systematically investigated the necessity of very limited supervision to generic disentanglement learning.
The analysis of Locatello et al.
(2019b) only applies to disentangled VAEs on low-resolution datasets.
Instead, our work tries to answer a more practical question: with very little supervision on more complex and higher-resolution datasets, can we still well disentangle the factors of variation while maintaining the high generation quality?
Deep image manipulation.
Deep neural networks have enabled various image editing tasks, such as style transfer (Gatys et al., 2016), image-to-image translation (Zhu et al., 2017), automatic colorization (Zhang et al., 2016) and 3D-aware attribute editing (Yao et al., 2018).
None of the above methods except the 3D-SDN (Yao et al., 2018) has dealt with the semantic manipulation of multiple attributes for scene images.
Different from the 3D-SDN and other previous works on attribute editing that have been mostly focused on the 3D geometry manipulation, our proposed FC-StyleGAN is designed to semantically manipulate the ﬁne-grained factors instead, such as lighting conditions and object colors.
Furthermore, FC-StyleGAN does not apply an encoder-decoder structure as commonly used in previous works for semantic manipulation (Yao et al., 2018), instead, it adds controllable ﬁne-grained factors along with a super-resolution process, which turns out to perform well in editing ﬁne styles of high-resolution images with a good generalization ability.
M ODELS 3. B ACKGROUND ON AC-GAN AND S TYLE GAN
AC-GAN.
AC-GAN (Odena et al., 2017) is a variant of class conditional GANs.
In the AC-GAN, the generator inputs are a latent z (or called random noise) and the class label.
The input to the discriminator is the real or fake image, whilst its output is the probability that the image is real and the prediction of the class label.
Odena et al.
(2017) has shown that this modiﬁcation to the standard conditional GAN formulation produces more realistic images and appears to stabilize training.
StyleGAN StyleGAN (Karras et al., 2019) is a state-of-the-art GAN architecture for unsupervised image generation, particularly for high-ﬁdelity human faces.
Basically, StyleGAN comprises a mapping network whose role is to map the latent z to an intermediate space, which then controls the styles at each convolutional layer in the synthesis network via adaptive instance normalization (AdaIN) (Ulyanov et al., 2016; Huang & Belongie, 2017).
StyleGAN also enables the separation of ﬁne-grained and coarse-grained features.
For example, modifying the styles of low-resolution blocks affects only coarse-grained features (e.g. pose and eyeglasses), while modifying the styles of high-resolution blocks affects only ﬁne-grained features (e.g. color scheme and microstructure).
[Figure]
3. AC-S TYLE GAN FOR C ONTROLLABLE I MAGE S YNTHESIS
We introduce AC-StyleGAN, a combination of AC-GAN and StyleGAN, that enables conditional generation of high-resolution images.
As shown in Figure 1a, the generator in AC-StyleGAN conditions on a meta code by simply concatenating the meta code, a vector representing all the factors of variation, with the latent z .
The discriminator in AC-StyleGAN now provides two outputs, the classiﬁcation of real/fake and the prediction of meta code.
And ﬁnally, the outputs of the mapping network – the conditioned styles – will modulate each block in the synthesis network via AdaIN.
We choose AC-GAN, instead of other conditional GAN methods, such as cGANs (with Projection Discriminator) (Miyato & Koyama, 2018), mainly because only AC-GAN has the property of reconstructing meta code in the discriminator, which can easily be extended to a semi-supervised disentanglement learning framework.
More formally, let x ( n ) r denote the n -th image from the set of N real images and let c ( n ) r be the corresponding meta code, randomly sampled from the dataset.
Let c ( n ) f denote a random meta code sampled from the label distribution, and let D ( · ) and G ( · ) denote the discriminator and generator neural networks.
We also assume s ( n ) r and s ( n ) f represent the real and fake classiﬁcation logits in the discriminator, respectively, and ˆ c ( n ) r and ˆ c ( n ) f denote the predictions of c ( n ) r and c ( n ) f , respectively.
From Figure 1a, we have
[Equation]
Since we are aiming at achieving both high image quality and good controllability, we deﬁne a semi-supervised loss function for the generator and discriminator in AC-StyleGAN as follows,
[Equation]
2) which consists of three terms: the GAN loss term, unsupervised disentanglement term and supervised disentanglement term.
For the GAN loss, we apply the non-saturating loss plus gradient penalty in the discriminator, as in StyleGAN.
For both the unsupervised and supervised disentanglement terms, we simply use l 2 norm of the reconstruction losses.
Furthermore, we introduce two coefﬁcients γ and η ( n ) in the above loss function: (i) the disentanglement coefﬁcient γ ∈ (0 , ∞ ) balances the trade-off between the generation quality and disentanglement performance; (ii) the coefﬁcient η ( n ) is the label mask denoted by η ( n )
=
1 { α ( n ) <α } where α ( n ) i.i.d.
∼ Uniform [0 , 1] and α ∈ [0 , 1] is a hyperparameter.
Accordingly, the hyperparameter α controls the fraction of labelled data that will be used for supervision ( fully unsupervised in the disentanglement learning if α = 0 , and fully supervised if α = 1 ).
Note that we sample all the α ( n ) ’s before training so that the label mask η ( n ) remains unchanged during training to reﬂect the real semi-supervised case.
Finally, note that AC-StyleGAN reduces to an InfoGAN variant in the special case of α = 0 (fully unsupervised), when an appropriate reconstruction loss is used.
3. FC-S TYLE GAN FOR S EMANTIC M ANIPULATION OF F INE S TYLES
While we apply AC-StyleGAN with supervision over all observed factors for demonstrating a controllable generation of high-resolution images in GANs, in a potentially more realistic case where we are only interested in controlling a subset of factors (effectively treating the rest as random unobserved nuisance variables), the disentanglement quality of AC-StyelGAN drops signiﬁcantly, as shown in Section 4.2.
Furthermore, AC-StyleGAN cannot be directly applied to manipulate an existing high-resolution image.
Therefore, we propose a new image-to-image model called FCStyleAGN (i.e., Fine-grained Controlled StyleGAN) for only controlling ﬁne-grained factors.
Inspired by the observations that the lower resolution blocks in the StyleGAN generator learn the coarse-grained features while its high-resolution blocks accounts for ﬁne styles, the generator in
FC-StyleGAN does not contain the lower-resolution blocks.
As shown in Figure 1b, the generator instead takes the real image as one of its inputs by downscaling it to a lower resolution φ (e.g., φ = 32 in Figure 1b).
After that, it generates the high-resolution image by only modulating the ﬁne-grained code into higher resolution blocks.
Also, the discriminator in FC-StyleGAN predicts the value of ﬁne-grained code from the block with resolution φ , instead of the last output block.
The intuition is that similar to the observations in the generator, lower resolution blocks in the discriminator have less relationship with ﬁne-grained styles.
So it makes more sense to directly predict the meta code from higher resolution blocks.
Note that the value of φ varies from 4 to image size, determining what factors are identiﬁed as ﬁne-grained.
Finally, the loss function in
FC-StyleGAN is the same with Eq.
(2) in AC-StyleGAN, indicating a semi-supervised approach.
During training of FC-StyleGAN, the generator takes the downscaled image as is, and just embellishes it by increasing the resolution and generating the missing pixel-level detail.
As it does not need to learn the coarse-grained features any more, presumably it will have much easier time handling complex high-resolution images.
Furthermore, since the generator is still style-based in higher-resolution blocks with ﬁne-grained code as its input, we will retain the control over ﬁne styles.
However, there are two caveats in FC-StyleGAN.
The ﬁrst caveat is that as an image-toimage model, the original ﬁne-grained factors in the input image may interference the control of ﬁne code over the ﬁne styles in the output image.
In such sense, we emphasize the role of inductive biases on the model – the use of instance normalization in the generator.
Because the output of instance normalization preserves the spatial structure of image content by only normalizing ﬁne styles (Ulyanov et al., 2016), we argue that the original ﬁne styles in the input image could be washed away by instance normalization, and thus the ﬁne code will fully control the ﬁne-grained factors.
The second caveat is how to identify ﬁne-grained factors in a dataset before training FC-StyleGAN.
To this end, we use the meta code of all factors as the input of FC-StyleGAN with different downscaled resolutions φ .
For each φ , we introduce a new term – interpolation variance of each factor c
i , denoted by β
i ( φ ) , based on which the factor c
i is deﬁned as ﬁne-grained at φ if its interpolation variance satisﬁes
that β i ( φ ) > β 0 , where β 0 is a pre-deﬁned threshold.
The interpolation variance β
i ( φ ) is calculated as follows:
Given N real images, we do the latent traversal over each factor c ( n ) i of an image I ( n ) to get S interpolated (fake) images { ˆ I ( n ) i, 0 , · · · , ˆ I ( N ) i,S − 1 } .
Each interpolated image ˆ I ( n ) i,s is then fed into the discriminator to get a predicted factor ˆ c ( n ) i,s .
Thus, the interpolation variance is the average variance of ˆ c ( n ) i,s over the interpolation s -dimension, which is
[Equation]
Intuitively, low interpolation variance β
i ( φ ) means that changing the factor c i does not affect much the generated images, which further implies the factor
c i should not be considered as ﬁne-grained, since FC-StyleGAN with downscaled resolution φ cannot control it any more.
[Table]
E XPERIMENTS
In this section
[Table]
we ﬁrst introduce two new datasets – Isaac3D and Falcor3D. We then evaluate the performance of AC-StyleGAN and FC-StyleGAN on both datasets, focusing on three aspects: disentanglement quality, semantic correctness and image quality.
Quantitatively, we use Frechet Inception Distance (FID) (Heusel et al., 2017) to measure image quality, reconstruction error (i
[Table]
e., l rec =
(cid:107) ˆ c ( r ) − c
( r ) (cid:107) ) to measure semantic correctness, and MIG (Chen et al., 2018) to measure disentanglement quality.
Qualitatively, we apply the latent traversals to visually evaluate all three aspects as well.
Due to space limitations, we only show results on Isaac3D in this section; results on
Falcor3D are quite similar to those on Isaac3D, and are available in the appendix.
4. C REATION OF H IGH -Q UALITY D ISENTANGLEMENT D ATASETS
Current disentanglement datasets, such as 3D Chairs (Aubry et al., 2014),
dSprites (Matthey et al., 2017) and MPI3D (Gondal et al., 2019), are of low resolution and mostly lack of photorealism.
It makes them not suitable as disentanglement benchmarks in the high-resolution image domain.
To this end, we propose two new datasets – Falcor3D and Isaac3D , which possess much higher resolution, better photorealism and richer factors of variations, as shown in Table 1.
Falcor3D
In the Falcor dataset, there are in total 233,280 images and each has a resolution of 1024x1024.
This dataset is based on the 3D scene of a living room, where we can move the camera positions and change the lighting conditions.
Each image is paired with a ground-truth meta code, consisting of 7 factors of variation: lighting intensity (5), lighting x -dir (6), lighting y -dir (6), lighting z -dir (6), camera x -pos (6), camera y -pos (6), and camera z -pos (6).
Note that the number m behind each factor represents that the factor has m possible values, uniformly sampled in the normalized range of variations [0 , 1] .
To interpret this, for example, “lighting x -dir (6)” represents the lighting direction moving along the x
-axis and “camera z -pos (6)” represents the camera position moving along the z -axis.
Also, both factors have 6 values uniformly sampled from [0 , 1] .
Isaac3D In the Isaac3D dataset, there are in total 737,280 images and each has a resolution of 512x512.
This dataset is based on the 3D scene of a kitchen, where we can also move the camera positions and vary the lighting conditions.
To further increase the number of variations, we put a robotic arm inside, attached with an object.
The robotic arm has two degrees of freedom: xmovement (or horizontal rotation) and y -movement (or vertical rotation).
The attached object could change its shape, scale and color.
To make the rendered images more photorealistic, each object in the 3D scene has been provided with proper textures.
Finally, each image is paired with a groundtruth meta code, consisting of 9 factors of variation: lighting intensity (4), lighting y -dir (6), object color (4), wall color (4), object shape (3), object scale (4), camera height (4), robot x -movement (8), and robot y -movement (5).
Similarly, the number m behind each factor represents that the factor has m possible values, uniformly sampled in the normalized range of variations [0 , 1] .
Please see the appendix for more detailed descriptions of the two datasets.
[Figure]
4. E VALUATION OF AC-S TYLE GAN
Latent traversal of AC-StyleGAN.
We ﬁrst show the latent traversal results of AC-StyleGAN on
Isaac3D in Figure 2, where we apply full supervision from meta code of all the factors.
For illustration, here we only show three factors of variation: (robot x -movement, camera height, lighting y -dir).
Please see the appendix for latent traversal of all the factors.
Images in the ﬁrst column (marked by red box) are randomly sampled real images and the rest images in each row are their interpolations, respectively, by uniformly varying each factor from 0 to 1.
Unless otherwise stated, this setting applies to all the latent traversal results below.
As we can see, each factor in the interpolated images changes smoothly without affecting other factors.
Taking “lighting y -dir” as an example, only the direction of the point light gradually moves up from left to right in the bottom row, which can be evidenced by the fact that the light spot moves up from the refrigerator to the wall and the shadow of the cabinet on the wall moves down simultaneously.
More importantly, all the interpolated images visually look the same with the corresponding real images in the ﬁrst column except for the interpolated factors, and the image quality does not degrade over interpolation.
Therefore, the latent traversal results demonstrate three good properties of AC-StyleGAN with supervision: high disentanglement quality and good semantic correctness and high generation quality.
Semi-supervised learning.
How does AC-StyleGAN behave when given fewer labels?
To this end, we vary the supervision coefﬁcient α in Eq.
(2) to show the impact of supervision on ACStyleGAN and the results with γ ∈ { 1 , 10 } on Isaac3D are in Figure 3.
First, we can see with sufﬁcient supervision ( α ≥ 0 . 5 ), the difference between γ = 1 and 10 is quite small.
Also, both values of MIG and l rec almost reach their own optimal ones, along with a lower FID.
This quantitatively supports the above latent traversal results in Figure 2.
As expected, the disentanglement quality and semantic correctness gradually get worse when less labelled data is used.
Interestingly, the gap between γ = 1 and 10 ﬁrst increases and then decreases as α decreases from 0.5 to 0, with the largest one at α = 0 .
It means that properly increasing γ is the most beneﬁcial for disentanglement in the case where only a very limited number of labelled data is provided.
As we can see, by only using 5% of the labelled data ( α = 0 . 05 ) and setting γ = 10 , the disentanglement quality
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
[Figure]
(a) Supervision from a subset of factors 0 1 2 3 4 5 6 7 8 0.
I n t e r p o l a t i o n v a r i a n c e = 0 1 2 3 4 5 6 7 8 0.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
I n
[Figure]
e r p o l a t
i
[Figure]
n v a r
[Figure]
a n
[Figure]
Methods
[Figure]
[Table]
and semantic correctness are still close to those with full supervision ( α = 1 ), and improve over the unsupervised baseline ( α = 0 ) by a signiﬁcant margin.
Please see the appendix for the latent traversal results of AC-StyleGAN with α = 0 .
05 and γ = 10 .
It means adding a very small amount of labelled data into the training dataset could beneﬁt much the disentanglement learning.
Finally, in terms of image quality, we observe that larger α tends to result in a lower FID score, which means better disentanglement is not always obtained by sacriﬁcing the generation quality.
Comparisons with different baselines.
Current start-of-the-art approaches for generic unsupervised disentanglement learning are mostly VAE-based models, such as β -VAE (Higgins et al., 2017),
FactorVAE (Kim & Mnih, 2018) and β -TCVAE (Chen et al., 2018).
To this end, we use the unsupervised version of AC-StyleGAN ( α = 0 ) to compare with VAE-based models on (i) the dSprites dataset and (ii) the Isaac3D dsataset, respectively.
Note that for evaluations on Isaac3D, we ﬁrst downscale its resolution to 128x128 because VAEs have difﬁculties in generating higher-resolution images.
We can see from Table 2 that the unsupervised AC-StyleGAN consistently outperforms all the start-of-the-art disentangled VAEs on both datasets.
Particularly on the downscaled Isaac3D dataset, disentangled VAEs have much higher FID scores (FID > 100 ), indicating their poor image generation quality.
Please see Appendix A.5 for their randomly sampled images.
Interestingly, the disentangled VAEs also fail to capture all the variations in the Isaac3D dataset, supporting the claim that our datasets provide larger variation of factors for disentanglement learning.
Latent traversal on 256x256 CelebA. To show the performance of AC-StyleGAN on the realworld data, we also provide the qualitative results on CelebA with resolution 256x
(Liu et al., 2015).
In this experiment, the meta code in AC-StyleGAN has a length of 40 to capture all the attributes in the dataset.
Hence, this is more challenging than the previous works on CelebA in terms of controlling all the attributes in a higher resolution (Chen et al., 2016; Higgins et al., 2017;
Nguyen-Phuoc et al., 2019).
Note that since CelebA only has binary attributes, which commonly-
[Figure]
used disentanglement metrics including MIG and FactorVAE score are not applicable to, we focus on the qualitative results of latent traversal.
The results are shown in Figure 5.
We can see that
AC-styleGAN can smoothly interpolate in the latent space of face attributes and also maintain high image quality.
More interesting interpolations can be seen in Appendix A.6, where some attributes are more difﬁcult to change during latent traversal, presumably due to the fact that these attributes have low frequency in the CelebA dataset, such as Mustache on female faces.
Only controlling a subset of style factors.
Instead of trying to control all the factors in a dataset, practically speaking we may only be interested in a subset of factors in which case the remaining factors will be considered as random nuisances captured by the latent z .
On a ﬁrst look, it might seem intuitive that the fewer factors of variation we try to control, the better the disentanglement quality we should achieve.
However, the latent traversal results of AC-StyleGAN( Figure 4a) show instead that both disentanglement quality and semantic correctness degrade.
For example, the camera height changes when interpolating the object scale, and the wall color also varies when interpolating the object scale.
Furthermore, the latent z , which is expected to capture the remaining factors: (object shape, camera height, lighting intensity, wall color), can barely change the camera height and wall color (see the appendix for results of other factors).
To explain this, we argue that since factors are highly correlated with each other, learning to control one may help a lot in learning to control the others.
Without observations on other nuisance factors, it may be very difﬁcult to disentangle the impact of the observed factor with unobserved covariates, especially in the high-dimensional space.
4. E VALUATION OF FC-S TYLE GAN
Identifying ﬁne-grained factors.
As we have discussed, it is a crucial for FC-StyleGAN to quantitatively identify ﬁne-grained factors.
Here we calculate the interpolation variance of each factor according to Eq.
(3) by setting N = 100 , S = 10 .
Figure 4b shows the interpolation variance of each factor in FC-StyleGAN with downscaled resolution φ varying from 4 to 128 on Isaac3D. To decide the threshold β 0 , we visually check the latent traversal results and ﬁnd that the interpolations over a factor c i almost stay the same if β i ( φ ) < 0 .
055 , which means β 0 = 0 .
Please see the appendix for the latent traversal results.
We can see that as φ gets larger, more factors have an interpolation variance below the threshold β 0 , meaning more coarse-grained factors appear.
For instance, there are only two coarse-grained factors at φ = 8 : camera height and lighting direction, and two more factors become coarse-grained factors at φ
= 16 : robot x -movement and robot y -movement, and so on.
Interestingly, wall color is always ﬁne-grained even at φ = 128 while object color has already become coarse-grained at φ = 64 , although both of them are about colors.
Latent traversal of FC-StyleGAN.
Based on the above observations, we can obtain the latent traversal results of FC-StyleGAN on ﬁne-grained factors, which are shown in Figure 6.
Speciﬁcally, we train FC-StyleGAN with downscaled resolution φ = 16 on Isaac3D. Since Figure 4b indicates that ﬁne-grained factors at φ = 16 are of index (0 , 4 , 5 , 7 , 8) , we use its subset: (object scale, lighting intensity, object color, wall color) as the input while leaving the object shape together with all
[Figure] [Figure]
the coarse-grained factors as random nuisances.
We can see each considered factor in the interpolated images changes smoothly without affecting other factors, which implies good disentanglement quality.
Because FC-StyleGAN is only applicable for ﬁne-grained control, we can observe that, on the one hand, the coarse-grained factors of the interpolated images always stay the same with the corresponding input image, even though they are random nuisances.
On the other hand, the object shape, as the only one ﬁne-grained random nuisance, could be different from the corresponding input image with some probability.
In Figure 6, for example, the object shape stays the same with the input image when interpolating the object scale, while it becomes different from the input image when interpolating the object color.
More importantly, compared with the results of AC-StyleGAN in Figure 4a, FC-StyleGAN has much better disentanglement quality, when it comes to ﬁne-grained control with supervision from a subset of factors.
Impact of instance normalization.
Now we want to show that without instance normalization in
FC-StyleGAN, the input image can interference the control of ﬁne-grained code over the output ﬁne styles.
The results of FC-StyleGAN with and without instance normalization are shown in Figure 7a, where both are trained with downscaled resolution φ = 32 on Isaac3D, and only a subset of ﬁnegrained factors (lighting intensity, object color, wall color) is used as supervision.
For illustration,
Figure 7a only shows the the wall color for comparison.
Please see the appendix for all three factors.
We can see that without instance normalization, latent traversal in FC-StyleGAN cannot change the wall color any more, which instead stays the same with the input image.
This conﬁrms our previous hypothesis that the instance normalization plays an important role in alleviating the impact of the input image in controlling ﬁne-grained features over generation.
Also, the quantitative results in
Figure 7a further conﬁrm the advantages of using instance normalization.
In particular, the fact that the reconstruction error l rec becomes larger without instance normalization implies that the strong interference of the input image also degrades the prediction of ﬁne-grained code in the discriminator.
Generalization to unseen images.
As an image-to-image model, it is natural to ask how FCStyleGAN generalizes to unseen images.
To test its generalization ability, the novel test images are provided as follows: i) we shift the robot position to the right hand side of the camera (instead of standing right in the middle for the training dataset), and ii) we also attach the robot arm with an unseen object.
Figure 7b shows the results of interpolating the wall color of the same test image with different downscaled resolutions φ ∈ { 16 , 32 , 64 } .
Please see the appendix for results of other test images and different ﬁne-grained factors.
As we can see, the wall color keeps changing during its interpolations in each case without affecting other factors, implying good disentanglement quality.
Furthermore, the interpolated images in each case maintain the new robot position and particularly maintain new object shape (i.e., octahedron) in the case of φ = 64 .
The reason the new object shape is only maintained at φ = 64 is because that the object shape, together with the robot position, is not ﬁne-grained at φ = 64 any more.
Therefore, it demonstrates that the disentanglement learning of FC-StyleGAN can generalize well to unseen novel test images.
Finally, Figure 7b shows that the generalization results get better in terms of image quality as we increase the downscaled resolution φ .
As we know, with the larger value of φ , we can control fewer ﬁne-grained factors.
Therefore, there exists a trade-off between the generalization and controllability in FC-StyleGAN.
We leave the investigation into how to improve this trade-off in the future work.
C ONCLUSIONS
In this work, we developed AC-StyleGAN, by combining both advantages of AC-GAN and StyleGAN, to demonstrate that it is possible to perfectly disentangle and control different factors of variation in the high-resolution image domain with sufﬁcient supervision.
In particular, with only 5% of the labelled dataset, the disentanglement quality is very close to the fully supervised case and outperforms the fully unsupervised case by a signiﬁcant margin, emphasizing the importance of (weak) supervision.
To address the performance degradation of AC-StyleGAN in cases where we observe/control only a subset of factors, we proposed FC-StyleGAN, a new image-to-image model for semantically manipulating ﬁne-grained factors of a given high-ﬁdelity image.
We demonstrated the importance of instance normalization in FC-StyleGAN, and also showed its good generalization ability to unseen images.
Finally, we proposed two new datasets – Falcor3D and Isaac3D with higher resolution, better photorealism and richer factors of variation, compared to current disentanglement datasets.
We believe that our proposed models, new datasets, along with the useful insights we have gleaned, can beneﬁt the further development of disentangled GANs for the controllable generation of high-ﬁdelity images.
For the future work, we would like to explore how to control coarse-grained factors only in high-resolution images while leaving ﬁne-grained factors as random nuisances
[Figure]
E XAMPLES IN THE F ALCOR 3D DATASET
[Figure]
A. I MAGE RECONSTRUCTIONS WITH THE SAME META CODE IN AC-S TYLE GAN
[Figure]
A. M ORE RESULTS ON LATENT TRAVERSAL OF AC-S TYLE GAN
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
lighting intensity object shape robot x-move.
robot y-move.
camera height object scale lighting y-dir object color wall color
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
[Figure]
A. L ATENT TRAVERSAL RESULTS AC-S TYLE GAN WITH 5% LABELLED DATA
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
lighting intensity object shape robot
[Figure]
robot y
[Figure]
camera height object scale lighting y-dir object color wall color
[Figure]
A. M ORE LATENT TRAVERSAL RESULTS OF AC-S TYLE GAN ON C ELEB A
[Figure] [Figure]
ORE RESULTS ON ONLY CONTROLLING A SUBSET OF FACTORS IN AC-S TYLE GAN
[Figure]
A. M ORE RESULTS ON IDENTIFYING FINE - GRAINED FACTORS
[Figure]
A.
M ORE RESULTS ON THE LATENT TRAVERSAL OF FC-S TYLE GAN
[Figure]
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
1 NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
NVIDIA CONFIDENTIAL.
DO NOT DISTRIBUTE.
[Figure]
A. M ORE RESULTS ON THE IMPACT OF INSTANCE NORMALIZATION
-S TYLE GAN
[Figure]
A. M ORE RESULTS ON THE GENERALIZATION OF FC-S TYLE GAN
[Figure]