I MAGE
GAN S
MEET
D IFFERENTIABLE
R ENDERING
FOR
I NVERSE
G RAPHICS
AND
I NTERPRETABLE 3D
N EURAL R ENDERING
Anonymous authors
Paper under double-blind review
A BSTRACT
Differentiable rendering has paved the way to training neural networks to perform “inverse graphics” tasks such as predicting 3D geometry from monocular photographs.
To train high performing models, most of the current approaches rely on multi-view imagery which are not readily available in practice.
Recent Generative Adversarial Networks (GANs) that synthesize images, in contrast, seem to acquire 3D knowledge implicitly during training: object viewpoints can be manipulated by simply manipulating the latent codes.
However, these latent codes often lack further physical interpretation and thus GANs cannot easily be inverted to perform explicit 3D reasoning.
In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers.
Key to our approach is to exploit GANs as a multi-view data generator to train an inverse graphics network using an off-the-shelf differentiable renderer, and the trained inverse graphics network as a teacher to disentangle the GAN’s latent code into interpretable 3D properties.
The entire architecture is trained iteratively using cycle consistency losses.
We show that our approach signiﬁcantly outperforms state-of-the-art inverse graphics networks trained on existing datasets, both quantitatively and via user studies.
We further showcase the disentangled GAN as a controllable 3D “neural renderer”, complementing traditional graphics renderers.
I NTRODUCTION
The ability to infer 3D properties such as geometry, texture, material, and light from photographs is key in many domains such as AR/VR, robotics, architecture, and computer vision.
Interest in this problem has been explosive, particularly in the past few years, as evidenced by a large body of published works and several released 3D libraries (TensorﬂowGraphics Valentin et al.
Kaolin J. et al.
, PyTorch3D Ravi et al.
The process of going from images to 3D is often called “inverse graphics”, since the problem is inverse to the process of rendering in graphics in which a 3D scene is projected onto an image by taking into account the geometry and material properties of objects, and light sources present in the scene.
Most work on inverse graphics assumes that 3D labels are available during training (Wang et al., 2018; Mescheder et al., 2019; Groueix et al., 2018;
Wang et al., 2019; Choy et al., 2016), and trains a neural network to predict these labels.
To ensure high quality 3D ground-truth, synthetic datasets such as ShapeNet (Chang et al., 2015) are typically used.
However, models trained on synthetic datasets often struggle on real photographs due to the domain gap with synthetic imagery.
To circumvent these issues, recent work has explored an alternative way to train inverse graphics networks that sidesteps the need for 3D ground-truth during training.
The main idea is to make graphics renderers differentiable which allows one to infer 3D properties directly from images using gradient based optimization, Kato et al.
; Liu et al.
(2019b)
; Li et al.
; Chen et al.
These methods employ a neural network to predict geometry, texture and light from images, by minimizing the difference between the input image with the image rendered from these properties.
While impressive results have been obtained in Liu et al. (2019b); Sitzmann et al.
; Liu et al. (2019a); Henderson & Ferrari (2018); Chen et al.
; Yao et al.
; Kanazawa et al.
(2018), most of these works still require some form of implicit 3D supervision such as multi-view images of the same object with known cameras.
Thus, most results have been reported on the syn-
DIFFGraphics-Renderer input image mesh light texture input image loss rendered images mask training inference loss camera
Inverse Graphics Network
StyleGAN latent code
Multi-view data
Disentangling Mapping Network multi-view loss
Fast Annotation!
thetic ShapeNet dataset, or the large-scale CUB (Welinder et al., 2010) bird dataset annotated with keypoints from which cameras can be accurately computed using structure-from-motion techniques.
On the other hand, generative models of images appear to learn 3D information implicitly, where several works have shown that manipulating the latent code can produce images of the same scene from a different viewpoint (Karras et al., 2019a).
However, the learned latent space typically lacks physical interpretation and is usually not disentangled, where properties such as the 3D shape and color of the object often cannot be manipulated independently.
In this paper, we aim to extract and disentangle 3D knowledge learned by generative models by utilizing differentiable graphics renderers.
We exploit a GAN, speciﬁcally StyleGAN (Karras et al., 2019a), as a generator of multi-view imagery to train an inverse graphics neural network using a differentiable renderer.
In turn, we use the inverse graphics network to inform StyleGAN about the image formation process through the knowledge from graphics, effectively disentangling the GAN’s latent space.
We connect StyleGAN and the inverse graphics network into a single architecture which we iteratively train using cycle-consistency losses.
We demonstrate our approach to significantly outperform inverse graphics networks on existing datasets, and showcase controllable 3D generation and manipulation of imagery using the disentangled generative model.
R ELATED W ORK
3D from 2D: Reconstructing 3D objects from 2D images is one of the mainstream problems in 3D computer vision.
We here focus our review to single-image 3D reconstruction which is the domain of our work.
Most of the existing approaches train neural networks to predict 3D shapes from images by utilizing 3D labels during training, Wang et al.
; Mescheder et al.
Choy et al.
; Park et al.
However, the need for 3D training data limits these methods to the use of synthetic datasets.
When tested on real imagery there is a noticeable performance gap.
Newer works propose to differentiate through the traditional rendering process in the training loop of neural networks, Loper & Black (2014); Kato et al.
; Liu et al.
(2019b)
; Chen et al.
Petersen et al.
Differentiable renderers allow one to infer 3D from 2D images without requiring 3D ground-truth.
However, in order to make these methods work in practice, several additional losses are utilized in learning, such as the multi-view consistency loss whereby the cameras are assumed known.
Impressive reconstruction results have been obtained on the synthetic ShapeNet dataset.
While CMR by Kanazawa et al.
and DIB-R by Chen et al.
(2019) show real-image 3D reconstructions on CUB and Pascal3D (Xiang et al., 2014) datasets, they rely on manually annotated keypoints, while still failing to produce accurate results.
A few recent works, Wu et al.
; Li et al.
; Goel et al.
; Kato & Harada (2019), explore 3D reconstruction from 2D images in a completely unsupervised fashion.
They recover both 3D shapes and camera viewpoints from 2D images by minimizing the difference between original and re-projected images with additional unsupervised constraints, e.g., semantic information (Li et al.
, symmetry (Wu et al.
GAN loss (Kato & Harada (2019)) or viewpoint distribution (Goel et al.
Their reconstruction is typically limited to 2.5D (Wu et al.
and produces lower quality results than when additional supervision is used (Goel et al.
; Li et al.
; Kato & Harada (2019)).
In contrast, we utilize GANs to generate multi-view realistic datasets that can be annotated extremely efﬁciently , which leads to accurate 3D results.
Furthermore, our model achieves disentanglement in GANs and turns them into interpretable 3D neural renderers.
Neural Rendering with GANs: GANs (Goodfellow et al., 2014; Karras et al., 2019a) can be regarded as neural renderers, as they take a latent code as input and “render” an image.
However, the latent code is sampled from a predeﬁned prior and lacks interpretability.
Several works generate images with conditions: a semantic mask (Zhu et al., 2017), scene layout Karacan et al.
(2016), or a caption (Reed et al., 2016),
and manipulate the generated images by modifying the input condition.
Despite tremendous progress in this direction, there is little work on generating images through an interpretable 3D physics process.
Dosovitskiy et al.
(2016) synthesizes images conditioned on object style, viewpoint, and color.
Most relevant work to ours is Zhu et al.
(2018), which utilizes a learnt 3D geometry prior and generates images with a given viewpoint and texture code.
We differ in three important ways.
First, we do not require a 3D dataset to train the 3D prior.
Second, the texture in our model has 3D physical meaning, while Zhu et al.
(2018) still samples from a prior.
We further control background while Zhu et al.
(2018) synthesizes objects onto white background.
Disentangling GANs: Learning disentangled representations has been widely explored, Lee et al.
(2020); Lin et al.
; Perarnau et al.
Representative work is InfoGAN Chen et al.
(2016), which tries to maximize the mutual information between the prior and the generated image distribution.
However, the disentangled code often still lacks physical interpretability.
Tewari et al.
(2020) transfers face rigging information from an existing model to control face attribute disentanglement in the StyleGAN latent space.
Shen et al.
(2020) aims to ﬁnd the latent space vectors that correspond to meaningful edits, while H¨ark¨onen et al.
(2020) exploits PCA to disentangle the latent space.
In our work, we disentangle the latent space with knowledge from graphics.
O UR A PPROACH
We start by providing an overview of our approach (Fig. 1), and describe the individual components in more detail in the following sections.
Our approach marries two types of renderers: a GANbased neural “renderer” and a differentiable graphics renderer.
Speciﬁcally, we leverage the fact that the recent state-of-the-art GAN architecture StyleGAN by Karras et al.
(2019a;b) learns to produce highly realistic images of objects, and allows for a reliable control over the camera.
We manually select a few camera views with a rough viewpoint annotation, and use StyleGAN to generate a large number of examples per view, which we explain in Sec. 3.1.
In Sec. 3.2, we exploit this dataset to train an inverse graphics network utilizing the state-of-the-art differentiable renderer, DIBR by Chen et al.
(2019) in our work, with a small modiﬁcation that allows it to deal with noisy cameras during training.
In Sec. 3.3, we employ the trained inverse graphics network to disentangle
StyleGAN’s latent code and turn StyleGAN into a 3D neural renderer, allowing for control over explicit 3D properties.
We ﬁne-tune the entire architecture, leading to signiﬁcantly improved results.
3. S TYLE GAN AS S YNTHETIC D ATA G ENERATOR
We ﬁrst aim to utilize StyleGAN to generate multi-view imagery.
StyleGAN is a 16 layer convolutional neural network that maps a latent code z ∈ Z drawn from a normal distribution into a realistic image.
The code z is ﬁrst mapped to an intermediate latent code w ∈ W which is transformed to w ∗ =
( w ∗ 1 , w ∗ 2 , ..., w ∗ 16 )
∈ W ∗ through 16 learned afﬁne transformations.
We call W ∗ the transformed latent space to differentiate it from the intermediate latent space W .
Transformed latent codes w ∗ are then injected as the style information to the StyleGAN Synthesis network.
Different layers control different image attributes.
As observed in Karras et al.
(2019a), styles in early layers adjust the camera viewpoint while styles in the intermediate and higher layers inﬂuence shape, texture and background.
We provide a careful analysis of all layers in Appendix.
We empirically ﬁnd that the latent code w ∗ v :=
( w ∗ 1 , w ∗ 2 , w ∗ 3 , w ∗ 4 ) in the ﬁrst 4 layers controls camera viewpoints.
That is, if we sample a new code w ∗ v but keep the remaining dimensions of w ∗ ﬁxed (which we call the conten code), we generate images of the same object depicted in a different viewpoint.
Examples are shown in Fig.
We further observe that a sampled code w ∗ v in fact represents a ﬁxed camera viewpoint.
That is, if we keep w ∗ v ﬁxed but sample the remaining dimensions of w ∗ , StyleGAN produces imagery of different objects in the same camera viewpoint.
This is shown in columns in Fig.
Notice how aligned the objects are in each of the viewpoints.
This makes StyleGAN a multi-view data generator! “StyleGAN” multi-view dataset: We manually select several views, which cover all the common viewpoints of an object ranging from 0-360 in azimuth and roughly 0-30 in elevation.
We pay attention to choosing viewpoints in which the objects look most consistent.
Since inverse graphics works require camera pose information, we annotate the chosen viewpoint codes with a rough absolute camera pose.
To be speciﬁc, we classify each viewpoint code into one of 12 azimuth angles, uniformly sampled along 360 deg .
We assign each code a ﬁxed elevation (0 ◦ ) and camera distance.
These camera poses provide a very coarse annotation of the actual pose – the annotation serves as the initialization of the camera which we will optimize during training.
This allows us to annotate all views (and thus the entire dataset) in only 1 minute – making annotation effort neglible.
For each viewpoint, we sample a large number of content codes to synthesize different objects in these views.
Fig.
2 shows 2 cars, and a horse and a bird.
Appendix provides more examples.
Since DIB-R also utilizes segmentation masks during training, we further apply MaskRCNN by He et al.
(2017) to get instance segmentation in our generated dataset.
As StyleGAN sometimes generates unrealistic images or images with multiple objects, we ﬁlter out “bad” images which have more than one instance, or small masks (less than 10% of the whole image area).
3. T RAINING AN I NVERSE G RAPHICS N EURAL N ETWORK
Following CMR by Kanazawa et al.
(2018), and DIB-R by Chen et al.
(2019), we aim to train a 3D prediction network f , parameterized by θ , to infer 3D shapes (represented as meshes) along with textures from images.
Let I V denote an image in viewpoint V from our StyleGAN dataset, and M its corresponding object mask.
The inverse graphics network makes a prediction as follows: { S, T } = f θ ( I V ) , where S denotes the predicted shape, and T a texture map.
Shape S is deformed from a sphere as in Chen et al.
While DIB-R also supports prediction of lighting, we empirically found its performance is weak for realistic imagery and we thus omit lighting estimation in our work.
To train the network, we adopt DIB-R as the differentiable graphics renderer that takes { S, T } and V as input and produces a rendered image I ′ V
= r ( S, T, V )
along with a rendered mask M ′ .
Following
DIB-R, the loss function then takes the following form:
[Equation]
Here, L col is the standard L 1 image reconstruction loss deﬁned in the RGB color space while L percpt is the perceptual loss that helps the predicted texture look more realistic.
Note that rendered images do not have background, so L col and L percept are calculated by utilizing the mask.
L IOU computes the intersection-over-union between the ground-truth mask and the rendered mask.
Regularization losses such as the Laplacian loss L lap and ﬂatten loss L sm are commonly used to ensure that the shape is well behaved.
Finally, L mov regularizes the shape deformation to be uniform and small.
Since we also have access to multi-view images for each object we also include a multi-view consistency loss.
In particular, our loss per object k is:
[Equation]
While more views provide more constraints, empirically, two views have been proven sufﬁcient.
We randomly sample view pairs ( i, j ) for efﬁciency.
We use the above loss functions to jointly train the neural network f and optimize viewpoint cameras
V (which were ﬁxed in Chen et al.
We assume that different images generated from the same w ∗ v correspond to the same viewpoint V .
Optimizing the camera jointly with the weights of the network allows us to effectively deal with noisy initial camera annotations.
3. D ISENTANGLING S TYLE GAN WITH THE I NVERSE G RAPHICS M ODEL
The inverse graphics model allows us to infer a 3D mesh and texture from a given image.
We now utilize these 3D properties to disentangle StyleGAN’s latent space, and turn StyleGAN into a fully
[Table]
/QnVKNgkk9LvczwlLIRHfCupQmNufEn82un5MwqIYmUtpUgmau/JyY0NmYcB7Yzpjg0y95M/M/rZhhd+xORpBnyhC0WRZkkqMjsdRIKzRnKsSWUaWFvJWxINWVoAyrZELzll1dJq1b1Lqq1+8tK/SaPowgncArn4MEV1OEOGtAEBo/wDK/w5ijnxXl3PhatBSefOYY/cD5/AEQ5juw=</latexit>
[Figure]
controllable 3D neural renderer, which we refer to as StyleGAN-R. Note that StyleGAN in fact synthesizes more than just an object
, it also produces the background, i.e., the entire scene.
Ideally we want control over the background as well, allowing the neural renderer to render 3D objects into desired scenes.
To get the background from a given image, we simply mask out the object.
We propose to learn a mapping network to map the viewpoint, shape (mesh), texture and background into the StyleGAN’s latent code.
Since StyleGAN may not be completely disentangled, we further ﬁne-tune the entire StyleGAN model while keeping the inverse graphics network ﬁxed.
Mapping Network: Our mapping network, visualized in Figure 3, maps the viewpoints to ﬁrstlayers and maps the shape, texture and background to the last 12 layers of W ∗ .
For simplicity, we denote the ﬁrst 4 layers as W ∗ V and the last 12 layers as W ∗ STB , where W ∗ V ∈ R 2048 and
W ∗ STB ∈ R 3008 .
Speciﬁcally, the mapping network g v for viewpoint V and g s for shape S are separate MLPs while g t for texture T and g b for background B are CNN layers:
[Equation]
where z view ∈ R 2048 , z shape , z txt , z bck ∈ R 3008 and θ v , θ s , θ t , θ b are network parameters.
We softly combine the shape, texture and background codes into the ﬁnal latent code as follows:
[Equation]
where ⊙ denotes element-wise product, and s m , s t , s b ∈ R 3008 are shared across all the samples.
To achieve disentanglement, we want each dimension of the ﬁnal code to be explained by only one property (shape, texture or background).
We thus normalize each dimension of s using softmax.
In practice, we found that mapping V to a high dimensional code is challenging since our dataset only contains a limited number of views, and V is limited to azimuth, elevation and scale.
We thus map V to the subset of W ∗ V , where we empirically choose 144 of the 2048 dimensions with the highest correlation with the annotated viewpoints.
Thus, z view ∈ R 144 in our case.
Training Scheme: We train the mapping network and ﬁne-tune StyleGAN in two separate stages.
We ﬁrst freeze StyleGAN’s weights and train the mapping network only.
This warms up the mapping network to output reasonable latent codes for StyleGAN.
We then ﬁne-tune both StyleGAN and the mapping network to better disentangle different attributes.
We provide details next.
In the warm up stage, we sample viewpoint codes w ∗ v among the chosen viewpoints, and sample the remaining dimensions of w ∗ ∈ W ∗ .
We try to minimize the L 2 difference between the mapped code ˜ w and StyleGAN’s code w ∗ .
To encourage the disentanglement in the latent space, we penalize the entropy of each dimension i of s .
Our overall loss function for our mapping network is:
[Equation]
By training the mapping network, we ﬁnd that view, shape and texture can be disentangled in the original StyleGAN model but the background remains entangled.
We thus ﬁne-tune the model to get a better disentanglement.
To ﬁne-tune the StyleGAN network we incorporate a cycle consistency loss.
In particular, by feeding a sampled shape, texture and background to StyleGAN we obtain a synthesized image.
We encourage consistency between the original sampled properties and the shape, texture and background predicted from the StyleGAN-synthesized image via the inverse graphics network.
We further feed the same background B with two different { S, T } pairs to generate two images I
and I 2 .
We then encourage the re-synthesized backgrounds ¯ B 1 and ¯ B 2 to be similar.
This loss tries to disentangle the background from the foreground object.
During training, we ﬁnd that imposing the consistency loss on B in image space results in blurry images, thus we constrain it in the code space.
Our ﬁne-tuning loss takes the following form:
[Equation]
E XPERIMENTS
In this section, we showcase our approach on inverse graphics tasks (3D image reconstruction), as well as on the task of 3D neural rendering and 3D image manipulation.
[Figure]
[Figure]
Our “StyleGAN” Dataset: We ﬁrst randomly sample 6000 cars, 1000 hourse and 1000 birds with diverse shapes, textures, and backgrounds from StyleGAN.
After ﬁltering out images with bad masks as described in Sec. 3, 55429 cars, 16392 horses and 7948 birds images remain in our dataset which is signiﬁcant larger than the Pascal3D car dataset (Xiang et al., 2014) (4175 car images).
Note that nothing prevents us from synthesizing a signiﬁcantly larger amount of data, but in practice, this amount turned out to be sufﬁcient to train good models.
We provide more examples in Appendix.
4. 3D R ECONSTRUCTION R ESULTS
Training Details: Our DIB-R based inverse graphics model was trained with Adam (Kingma &
Ba (2015)), with a learning rate of 1e-4.
We set λ IOU , λ col , λ lap , λ sm and λ mov to 3, 20, 5, and 2.5, respectively.
We ﬁrst train the model with L col loss for 3K iterations, and then ﬁne-tune the model by adding L pecept to make the texture more realistic.
We set λ percept to 0.5.
The model converges in 200K iterations with batch size 16.
Training takes around 120 hours on four V100 GPUs.
Results: We show 3D reconstruction results in Fig.
Notice the quality of the predicted shapes and textures, and the diversity of the 3D car shapes we obtain.
Our method also works well on more challenging (articulated) classes, e.g. horse and bird.
We provide additional examples in Appendix.
Qualitative Comparison: To showcase our approach, we compare our inverse graphics network with exactly the same model but which we train on the Pascal3D car dataset.
Pascal3D dataset has annotated keypoints, which we utilize to train the baseline model, termed as as Pascal3D-model.
We show qualitative comparison on Pascal3D test set in Fig.
Note that the images from Pascal3D dataset are different from those our StyleGAN-model was trained on.
Although the Pascal3Dmodel’s prediction is visually good in the input image view, rendered predictions in other views are of noticeably lower quality than ours, which demonstrates that we recover 3D geometry and texture better than the baseline.
Quantitative Comparison: We evaluate the two networks in Table 1 for the car class.
We report the estimated annotation time in Table.
1 (a) to showcase efﬁciency behind our StyleGAN dataset.
It takes 3-5 minutes to annotate keypoints for one object, which we empirically verify.
Thus, labeling
Pascal3D required around 200-350 hours while ours takes only 1 minute to annotate a 10 times larger dataset.
In Table 1 (b), we evaluate shape prediction quality by the re-projected 2D IOU score.
Our
[Figure] [Figure]
model outperforms the Pascal3D-model on the SyleGAN test set while Pascal3D-model is better on the Pascal test set.
This is not surprising since there is a domain gap between two datasets and thus each one performs best on their own test set.
Note that this metric only evaluates quality of the prediction in input view and thus not reﬂect the actual quality of the predicted 3D shape/texture.
To analyze the quality of 3D prediction, we conduct an AMT user study on the Pascal3D test set which contains 220 images.
We provide users with the input image and predictions rendered in 6 views (shown in Fig.
5, right) for both models.
We ask them to choose the model with a more realistic shape and texture prediction that matches the input object.
We provide details of the study in the Appendix.
We report results in Table.
1 (c).
Users show signiﬁcant preference of our results versus the baseline, which conﬁrms that the quality of our 3D estimation.
4. D UAL R ENDERERS
Training Details: We train StyleGAN-R using Adam with learning rate of 1e-5 and batch size 16.
Warmup stage takes 700 iterations, and we perform joint ﬁne-tuning for another 2500 iterations.
With the provided input image, we ﬁrst predict mesh and texture using the trained inverse graphics model, and then feed these 3D properties into StyleGAN-R to generate a new image.
For comparison, we feed the same 3D properties to the DIB-R graphics renderer (which is the OpenGL renderer).
Results are provided in Fig.
Note that DIB-R can only render the predicted object, while StyleGAN-R also has the ability to render the object into a desired background.
We ﬁnd that
StyleGAN-R produces relatively consistent images compared to the input image.
Shape and texture are well preserved, while only the background has a slight content shift.
4. 3D I MAGE M ANIPULATION WITH S TYLE GAN-R
We test our approach in manipulating StyleGAN-synthesized images from our test set and real images.
Speciﬁcally, given an input image, we predict 3D properties using the inverse graphics network, and extract background by masking out the object with Mask-RCNN.
We then manipulate and feed these properties to StyleGAN-R to synthesize new views.
Controlling Viewpoints:
We ﬁrst freeze shape, texture and background, and change the camera viewpoint.
Example is shown in Fig.
We obtain meaningful results, particularly for shape and texture.
For comparison, an alternative way that has been explored in literature is to directly optimize the GAN’s latent code (in our case the original StyleGAN’s code) via an L2 image reconstruction loss.
Results are shown in the last three columns in Fig.
As also observed in Abdal et al.
(2019), this approach fails to generate plausible images, showcasing the importance of the mapping network and ﬁne-tuning the entire architecture with 3D inverse graphics network in the loop.
Controlling Shape, Texture and Background
: We further aim to manipulate 3D properties, while keeping the camera viewpoint ﬁxed.
In the second column of Fig 9, we replace the shapes of all cars to one chosen shape (red box) and perform neural rendering using StyleGAN-R. We successfully swap the shape of the car while maintaining other properties.
We are able to modify tiny parts of the car, such as trunk and headlights.
We do the same experiment but swapping texture and background in the third and forth column of Fig 9.
We notice that swapping textures also slightly modiﬁes the background, pointing that further improvements are possible in disentangling the two.
[Figure]
S ca l e
A z i m u t h
E l e v a ti on
[Figure]
r
C a r
C a r
[Figure]
Real Image Editing: As shown in Fig.
10, our framework also works well when provided with real images, since StyleGAN’s images, which we use in training, are quite realistic.
4. L IMITATIONS
While recovering faithful 3D gemetry and texture, our model fails to predict correct lighting model, since real images and StyleGAN generated images contain advanced lighting effects such as reﬂection, transparency and shadows and our spherical harmonic lighting model is incapable to deal with it.
And we only partly succeed at disentangling the background, as we could observe slighly change in the background in Fig.
6, Fig. 9 and Fig.
10 We leave further improvements to future work.
C ONCLUSION
In this paper, we introduced a new powerful architecture that links two renderers: a state-of-the-art image synthesis network and a differentiable graphics renderer.
The image synthesis network generates training data for an inverse graphics network.
In turn, the inverse graphics network teaches the synthesis network about the physical 3D controls.
We showcased our approach to obtain signiﬁcantly higher quality 3D reconstruction results while requiring 10000 less annotation effort than standard datasets.
We also provided 3D neural rendering and image manipulation results demonstrating the effectiveness of our approach.
A PPENDIX A O VERVIEW
In the Appendix, we ﬁrst show feature visualization of StyleGAN layers in Sec.
B. We then provide a detailed explanation of our StyleGAN dataset creation in Sec.
C, including examples of the generated images and selected viewpoints.
Next, we do a systematic analysis of our camera initialization method in Sec.
D. Finally, we show more results on 3D inverse graphics task in Sec.
E, more details of user study in Sec.
F, more examples of StyleGAN disentanglement in Sec.
G, with ablation studies and limitaion in Sec.
I and Sec.
J, respectively.
We add rebuttal results in Sec.
K B S TYLE GAN L AYERS V ISUALIZATION
The ofﬁcial StyleGAN code repository provides models of different object categories at different resolution.
Here we take the 512 × 384 car model as the example.
This model contains 16 layers, where every two consecutive layers form a block.
Each block has a different number of channels.
In the last block, the model produces a 32-channel feature map at a 512x384 resolution.
Finally, a learned RGB transformation function is applied to convert the feature map into an RGB image.
We visualize the feature map for each block via the learned RGB transformation function.
Specifically, for the feature map in each block with the size of h × w × c , we ﬁrst sum along the feature dimension, forming a h × w × 1 tensor.
We then repeat the feature 32 times and generate a h × w ×new feature map.
This allows us to keep the information of all the channels and directly apply the
RGB transformation function in the last block to convert it to the RGB image.
As shown in Figure A, we ﬁnd that blocks 1 and 2 do not exhibit interpretable structure while the car shape starts to appear in blocks 3, 4 and 5.
We can observe that there is a rough car contour in block 4 which further becomes clear in block 5.
From blocks 6 to 8, the car’s shape becomes increasingly ﬁner and background scene also appears.
This supports some of our ﬁndings, i.e., the viewpoint is controlled in block 1 and 2 (ﬁrst 4 layers) while shape, texture, and background exist in the lastlayers.
While we take car model as the example, we ﬁnd this property also true for other categories, e.g. horse and bird.
We use the ofﬁcial horse model from StyleGAN2 (Karras et al., 2019b) repo and train a StyleGAN bird model by NABirds (Van Horn et al., 2015) dataset.
C D ATASET
We show more examples from our generated dataset in Fig.
B, Fig.
C and Fig.
D to show car, horse and bird, respectively.
Our dataset contains objects with various shapes, textures and viewpoints.
In particular, for the car dataset in Fig.
B, one can notice a diverse variants of car types (SUV, Sports car, Jeep, etc) ranging from old to modern styles.
We ﬁnd that StyleGAN can also produce rare car shapes like truck or bus, but with a lower probability.
[Figure]
We further visualize all of our selected viewpoints in our dataset in Fig.
E. Our car training dataset contains 39 viewpoints.
For the horse and bird datasets, we choose 22 and 8 views, respectively.
We ﬁnd these views are enough to learn reasonable 3D models.
Note that for articulated objects such as horse and bird, StyleGAN does not perfectly preserve the articulation in different viewpoints, which leads to challenges in training high accuracy models using multi-view consistency loss.
We leave further investigation in articulated objects to future work.
D C AMERA I NITIALIZATION
Inverse graphics tasks require camera pose information, which is challenging to acquire for real imagery.
It is generally obtained by annotating keypoints for each object and running structurefrom-motion (SFM) techniques Welinder et al.
Xiang et al.
Keypoint annotation is time consuming – requiring roughly 3-5minutes per object.
In our work, we utilize StyleGAN
[Table]
to signiﬁcantly reduce annotation effort since samples with the same w ∗ v share the same viewpoint.
Therefore, we only need to annotate 39, 22 and 8 views for car, horse and bird, respectively.
Furthermore, instead of accurately annotating keypoints and running SFM, we annotate each view with a rough absolute camera pose (which we further optimize during training).
To be speciﬁc, we ﬁrst select 12 azimuth angles: [0 ◦ , 30 ◦ , 60 ◦ , 90 ◦ , 120 ◦ , 150 ◦ , 180 ◦ , 210 ◦ , 240 ◦ , 270 ◦ , 300 ◦ , 330 ◦ ].
Given a StyleGAN viewpoint, we manually classify which azimuth angle it is close to and assign it to the corresponding label with ﬁxed elevation(0 ◦ ) and camera distance.
This allows us to annotate all the views (and thus entire dataset) in only 1 minute , without any keypoint annotation.
To demonstrate the effectiveness of our camera initialization, we make a comparison with another inverse graphics network trained with accurate SFM camera initialization.
Such initialization is done by manually annotating keypoints for all views of one car example, which takes about 3hours (around 200 minutes, 39 views).
We refer to the two models as view -model and keypoint -model, respectively.
We quantitatively evaluate two initialization methods in Table.
A. We ﬁrst compare the annotation time and training time.
While it takes the same amount of time to train, view -model saves more annotation time.
The performance of two methods are comparable with almost the same 2D IOU re-projection score on StyleGAN testing set.
Moreover, the two camera systems converge to the same position.
We evaluate by converting all the views into quaternions and compare the difference between the rotation axes and rotation angles.
Among all views, the average difference of the rotation axes is only 1.43 ◦ and the rotation angle is 0.42 ◦ .
The maximum difference of the rotation axes is only 2.95 ◦ and the rotation angle is 1.11 ◦ .
We further qualitatively compare two methods in Fig.
F, showing that the two methods perform almost the same.
Both of qualitative and quantitative comparison have demonstrated view -camera initialization is enough in training inverse graphics networks and no additional annotation is required.
v i e w -I n it .
k e y p o
i n t -I n it .
[Figure]
E 3D I NFERENCE
We here present additional 3D prediction results and compare our model, which is trained on our
StyleGAN generated dataset (StyleGAN-model), with the one trained on Pascal 3D dataset (Xiang et al., 2014) (PASCAL-model).
We qualitatively compare two models on the Pascal3D test set in Fig.
G and web imagery in Fig.
H. Our StyleGAN-model produces better shape and texture predictions in all testing datasets, especially visible at different views of the object.
We also present additional 3D prediction results on horses and cars in Fig.
I.
P a s ca l
D
O u r s
P a s ca l
D
O u r s
P a s ca l
D
O u r s
P a s ca l
D
O u r s
P a s ca l
D
O u r s
P a s ca l
D
O u r s
[Figure]
P a s ca l
D
O u r s
P a s ca l
D
O u r s
P a s ca l
D
O u r s
P a s ca l
D
O u r s
P a s ca l
D
O u r s
[Figure] [Figure]
F U SER S TUDY
We report user study details in this section.
We provide users (AMT turkers) with the input image and predictions rendered in 6 views.
We ask users to choose more realistic shape and texture prediction.
For each image in the test set we asked three annotators perform evaluation, which results in 660 examples in total.
Altogether, 65 annotators participated in our user study.
We show our AMT interface in Fig.
J.
G S TYLE GAN-R D ISENTANGLEMENT
Given an input image, we infer 3D properties of an object (shape, texture, background) using our inverse graphics network, but can also map these properties back to the latent code and use our
StyleGAN-R to synthesize a new image.
We show the results in Fig.
K. Similar to Fig.
8 in the main paper, we show DIB-R-rendered predictions and neural rendering StyleGAN-R’s predictions, and manipulate their viewpoints in rows (1, 4) and (2, 5).
We further show “neural rendering” results from the original StyleGAN in row (3, 6), where we only learn the mapping network but keep the
StyleGAN’s weights ﬁxed.
We ﬁnd that ﬁne-tuning is necessary and StyleGAN-R produces more consistent shape, texture and background.
Real Image Editting:
We show additional real image editing examples in Fig.
L. With our
StyleGAN-R, we can easily change the car’s size, azimuth and elevation and synthesize a new image while preserving the shape and texture of the car with a consistent background.
H F AILURE C ASES
We ﬁnd that our inverse graphic model fails when there’s a severe domain discrepancy,as shown in
Fig.
M.
For example, the reconstruction results for Batmobile and Flinstone car are not representative of the input cars.
We anticipate that this challenge can be addressed by augmenting the StyleGAN to incorporate more diverse domains, so that it can generate more diverse data for training the inverse graphic model.
We leave this for future work.
I A BLATION S TUDIES
We ﬁnd that the multi-view consistency loss and perceptual loss play an import role in our training, as shown in Fig.
N. Multi-view consistency loss helps to generate a good model in all views while the perceptual loss helps to keep texture more realistic.
J L IMITATIONS
As mentioned in the paper, our simple spherical harmonic model fails to separate light from textures.
We show several examples in Fig.
O. We leave this for future work.
K REBUTTAL
As suggested by Reviewer3, we add the StyleGAN-R manipulation results for bird in Fig P and
Fig Q. We also compare the StyleGAN with StyleGAN-R for the camera controler in Fig R and provide improved optimization results for the experiment in Fig 7 of the main paper, following the reviewer’s suggestion to incorporate an encoder to predict the latent code, in Fig S
D I B - R
S t y l e
GAN - R
S t y l e
GAN
D I B - R
S t y l e
GAN - R
S t y l e
GAN
[Figure]
Pred.
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
S ca l e
A z i m u t h
E l e v a ti on
[Figure]
B i r d
B i r d
B i r d
S t y l e
GAN
S t y l e
GAN - R
S t y l e
GAN
S t y l e
GAN - R
S t y l e
GAN
S t y l e
GAN - R
[Figure] [Figure]
Moreover, when we try to adjust the style for the optimized code, we get low quality results (col 6, 7).
We add a further experiment (second line) where we learn a CNN encoder to map the image to the latent code and then perform a few steps of optimization.
This lead to better reconstructions than direct optimization.
Texture in this case is better than in the case of StyleGAN-R, which indicates further improvements are possible in predicting texture maps using the inverse graphics network.