Original: D O 2D GAN S K NOW 3D S HAPE ? U NSUPERVISED 3D SHAPE RECONSTRUCTION FROM 2D I MAGE GAN S
Revised: D O 2D GAN S K NOW 3D S HAPE ? U NSUPERVISED 3D SHAPE RECONSTRUCTION FROM 2D I MAGE GAN S

Original: [Figure]
Revised: [Figure]

Original: Rotation
Revised: Rotation

Original: [Figure]

Original: Rotation

Original: [Figure]
Revised: [Figure]

Original: Rotation
Revised: Rotation

Original: [Figure]
Revised: [Figure]

Original: A BSTRACT
Revised: A BSTRACT

Original: Natural images are projections of 3D objects on a 2D image plane.
Revised: Natural images are projections of 3D objects on a 2D image plane.

Original: While state-ofthe-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures.
Revised: While state-ofthe-art 2D generative models like GANs show unprecedented quality in modeling the natural image manifold, it is unclear whether they implicitly capture the underlying 3D object structures.

Original: And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images?
Revised: And if so, how could we exploit such knowledge to recover the 3D shapes of objects in the images?

Original: To answer these questions, in this work, we present the ﬁrst attempt to directly mine 3D geometric clues from an offthe-shelf 2D GAN that is trained on RGB images only.
Revised: To answer these questions, in this work, we present the ﬁrst attempt to directly mine 3D geometric cues from an offthe-shelf 2D GAN that is trained on RGB images only.

Original: Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner.
Revised: Through our investigation, we found that such a pre-trained GAN indeed contains rich 3D knowledge and thus can be used to recover 3D shape from a single 2D image in an unsupervised manner.

Original: The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold.
Revised: The core of our framework is an iterative strategy that explores and exploits diverse viewpoint and lighting variations in the GAN image manifold.

Original: The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes ( e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cars, buildings, etc .
Revised: The framework does not require 2D keypoint or 3D annotations, or strong assumptions on object shapes ( e.g. shapes are symmetric), yet it successfully recovers 3D shapes with high precision for human faces, cars, buildings, etc .

Original: The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation.
Revised: The recovered 3D shapes immediately allow high-quality image editing like relighting and object rotation.

Original: We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation.
Revised: We quantitatively demonstrate the effectiveness of our approach compared to previous methods in both 3D shape reconstruction and face rotation.

Original: Our code is available at https://github.com/XingangPan/GAN2Shape .
Revised: Our code is available at https://github.com/XingangPan/GAN2Shape .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative adversarial networks (GANs) (Goodfellow et al., 2014) are capable of modeling the 2D natural image manifold (Zhu et al., 2016) of diverse object categories with high ﬁdelity.
Revised: Generative adversarial networks (GANs) (Goodfellow et al., 2014) are capable of modeling the 2D natural image manifold (Zhu et al., 2016) of diverse object categories with high ﬁdelity.

Original: Recall the fact that natural images are actually the projections of 3D objects to the 2D plane, an ideal 2D image
Revised: Recall the fact that natural images are actually the projections of 3D objects to the 2D plane, an ideal 2D image

Original: [Figure]
Revised: [Figure]

Original: manifold should be able to reﬂect some underlying 3D geometrical properties.
Revised: manifold should be able to reﬂect some underlying 3D geometrical properties.

Original: For example, it is shown that a GAN could shift the object in its generated images ( e.g. , human faces) in a 3D rotation manner, as a direction in the GAN image manifold may correspond to viewpoint change (Shen et al., 2020).
Revised: For example, it is shown that a GAN could shift the object in its generated images ( e.g. , human faces) in a 3D rotation manner, as a direction in the GAN image manifold may correspond to viewpoint change (Shen et al., 2020).

Original: This phenomenon motivates us to ask - “ Is it possible to reconstruct the 3D shape of a single 2D image by exploiting the 3D-alike image manipulation effects produced by GANs? ”
Revised: This phenomenon motivates us to ask - “ Is it possible to reconstruct the 3D shape of a single 2D image by exploiting the 3D-alike image manipulation effects produced by GANs? ”

Original: Despite its potential to serve as a powerful method to learn 3D shape from unconstrained RGB images, this problem remains much less explored.
Revised: Despite its potential to serve as a powerful method to learn 3D shape from unconstrained RGB images, this problem remains much less explored.

Original: Some previous attempts (Lunz et al., 2020; Henzler et al., 2019; Szab ´ o et al., 2019) also adopt
Revised: Some previous attempts (Lunz et al., 2020; Henzler et al., 2019; Szab ´ o et al., 2019) also adopt

Original: GANs to learn 3D shapes from images, but they rely on explicitly modeling 3D representation and rendering during training ( e.g. 3D voxels, 3D models).
Revised: GANs to learn 3D shapes from images, but they rely on explicitly modeling 3D representation and rendering during training ( e.g. 3D voxels, 3D models).

Original: Due to either heavy memory consumption or additional training difﬁculty brought by the rendering process, the qualities of their generated samples notably lag behind their 2D GAN counterparts.
Revised: Due to either heavy memory consumption or additional training difﬁculty brought by the rendering process, the qualities of their generated samples notably lag behind their 2D GAN counterparts.

Original: Another line of works (Wu et al., 2020; Goel et al., 2020; Tulsiani et al., 2020; Li et al., 2020) for unsupervised 3D shape learning generally learns to infer the viewpoint and shape for each image in an ‘analysis by synthesis’ manner.
Revised: Another line of works (Wu et al., 2020; Goel et al., 2020; Tulsiani et al., 2020; Li et al., 2020) for unsupervised 3D shape learning generally learns to infer the viewpoint and shape for each image in an ‘analysis by synthesis’ manner.

Original: Despite their impressive results, these methods often assume object shapes are symmetric (symmetry assumption) to prevent trivial solutions, which is hard to generalize to asymmetric objects such as ‘building’.
Revised: Despite their impressive results, these methods often assume object shapes are symmetric (symmetry assumption) to prevent trivial solutions, which is hard to generalize to asymmetric objects such as ‘building’.

Original: We believe that existing pre-trained 2D GANs, without above speciﬁc designs, already capture sufﬁcient knowledge for us to recover the 3D shapes of objects from 2D images.
Revised: We believe that existing pre-trained 2D GANs, without above speciﬁc designs, already capture sufﬁcient knowledge for us to recover the 3D shapes of objects from 2D images.

Original: Since the 3D structure of an instance could be inferred from images of the same instance with multiple viewpoint and lighting variations, our insight is that we may create these variations by exploiting the image manifold captured by 2D GANs.
Revised: Since the 3D structure of an instance could be inferred from images of the same instance with multiple viewpoint and lighting variations, our insight is that we may create these variations by exploiting the image manifold captured by 2D GANs.

Original: However, the main challenge is to discover well-disentangled semantic directions in the image manifold that control viewpoint and lighting in an unsupervised manner, as manually inspect and annotate the samples in the image manifold is laborious and time-consuming.
Revised: However, the main challenge is to discover well-disentangled semantic directions in the image manifold that control viewpoint and lighting in an unsupervised manner, as manually inspect and annotate the samples in the image manifold is laborious and time-consuming.

Original: To tackle the above challenge, we observe that for many objects such as faces and cars, a convex shape prior like ellipsoid could provide a hint on the change of their viewpoints and lighting conditions.
Revised: To tackle the above challenge, we observe that for many objects such as faces and cars, a convex shape prior like ellipsoid could provide a hint on the change of their viewpoints and lighting conditions.

Original: Inspired by this, given an image generated by GAN, we employ an ellipsoid as its initial 3D object shape, and render a number of unnatural images, called ‘pseudo samples’ , with various randomly-sampled viewpoints and lighting conditions as shown in Fig.
Revised: Inspired by this, given an image generated by GAN, we employ an ellipsoid as its initial 3D object shape, and render a number of unnatural images, called ‘pseudo samples’ , with various randomly-sampled viewpoints and lighting conditions as shown in Fig.

Original: By reconstructing them using the GAN, these pseudo samples could guide the original image towards the sampled viewpoints and lighting conditions in the GAN manifold, producing a number of natural-looking images, called ‘projected samples’ .
Revised: By reconstructing them using the GAN, these pseudo samples could guide the original image towards the sampled viewpoints and lighting conditions in the GAN manifold, producing a number of natural-looking images, called ‘projected samples’ .

Original: These projected samples could be adopted as the ground truth of the differentiable rendering process to reﬁne the prior 3D shape ( i.e. an ellipsoid).
Revised: These projected samples could be adopted as the ground truth of the differentiable rendering process to reﬁne the prior 3D shape ( i.e. an ellipsoid).

Original: To achieve more precise results, we further regard the reﬁned shape as the initial shape and repeat the above steps to progressively reﬁne the 3D shape.
Revised: To achieve more precise results, we further regard the reﬁned shape as the initial shape and repeat the above steps to progressively reﬁne the 3D shape.

Original: With the proposed approach,  we show that existing 2D GANs trained on images only are sufﬁcient to accurately reconstruct the 3D shape of a single image for many object categories such as human faces, cars, buildings, etc .
Revised: With the proposed approach, namely GAN2Shape , we show that existing 2D GANs trained on images only are sufﬁcient to accurately reconstruct the 3D shape of a single image for many object categories such as human faces, cars, buildings, etc .

Original: Our method thus is an effective approach for unsupervised 3D shape reconstruction from unconstrained 2D images without any 2D keypoint or 3D annotations.
Revised: Our method thus is an effective approach for unsupervised 3D shape reconstruction from unconstrained 2D images without any 2D keypoint or 3D annotations.

Original: With an improved GAN-inversion strategy, our method works not only for GAN samples, but also for real natural images.
Revised: With an improved GAN inversion strategy, our method works not only for GAN samples, but also for real natural images.

Original: On the BFM benchmark (Paysan et al., 2009), our method outperforms a recent strong baseline designed speciﬁcally for 3D shape learning (Wu et al., 2020).
Revised: On the BFM benchmark (Paysan et al., 2009), our method outperforms a recent strong baseline designed speciﬁcally for 3D shape learning (Wu et al., 2020).

Original: We also show high-quality 3D-aware image manipulations using the semantic latent directions discovered by our approach, which achieves more accurate human face rotation than other competitors.
Revised: We also show high-quality 3D-aware image manipulations using the semantic latent directions discovered by our approach, which achieves more accurate human face rotation than other competitors.

Original: Our contributions are summarized as follows.
Revised: Our contributions are summarized as follows.

Original: 1) We present the ﬁrst attempt to reconstruct the 3D object shapes using GANs that are pre-trained on 2D images only.
Revised: 1) We present the ﬁrst attempt to reconstruct the 3D object shapes using GANs that are pre-trained on 2D images only.

Original: Our work shows that 2D GANs inherently capture rich 3D knowledge for different object categories, and provides a new perspective for 3D shape generation.
Revised: Our work shows that 2D GANs inherently capture rich 3D knowledge for different object categories, and provides a new perspective for 3D shape generation.

Original: Our work also provides an alternative unsupervised 3D shape learning method, and does not rely on the symmetry assumption of object shapes.
Revised: Our work also provides an alternative unsupervised 3D shape learning method, and does not rely on the symmetry assumption of object shapes.

Original: 3) We achieve highly photo-realistic 3D-aware image manipulations including rotation and relighting without using any external 3D models.
Revised: 3) We achieve highly photo-realistic 3D-aware image manipulations including rotation and relighting without using any external 3D models.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Generative Adversarial Networks.
Revised: Generative Adversarial Networks.

Original: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Xiangli et al., 2020) have achieved great success in 2D natural image modeling.
Revised: Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Xiangli et al., 2020) have achieved great success in 2D natural image modeling.

Original: After a series of improvements, the state-of-the-art GANs like StyleGAN (Karras et al., 2019; 2020b) and
Revised: After a series of improvements, the state-of-the-art GANs like StyleGAN (Karras et al., 2019; 2020b) and

Original: BigGAN (Brock et al., 2019) could synthesis images with high ﬁdelity and resolution.
Revised: BigGAN (Brock et al., 2019) could synthesis images with high ﬁdelity and resolution.

Original: These GANs are composed of 2D operations, like 2D convolutions.
Revised: These GANs are composed of 2D operations, like 2D convolutions.

Original: Recently, there is a surge of interest to model 3D-aware image distributions with 3D GAN architectures and 3D representations (Nguyen-Phuoc et al., 2019;
Revised: Recently, there is a surge of interest to model 3D-aware image distributions with 3D GAN architectures and 3D representations (Nguyen-Phuoc et al., 2019;

Original: Wu et al., 2016; Szab´o et al., 2019; Henzler et al., 2019; Lunz et al., 2020).
Revised: Wu et al., 2016; Szab´o et al., 2019; Henzler et al., 2019; Lunz et al., 2020).

Original: However, due to either heavy memory consumption or increased difﬁculty in training, there are still gaps between their image qualities and those of 2D GANs.
Revised: However, due to either heavy memory consumption or increased difﬁculty in training, there are still gaps between their image qualities and those of 2D GANs.

Original: Therefore, in this work, we are interested in investigating whether we can recover the 3D object shape by directly mining the knowledge of 2D GANs.
Revised: Therefore, in this work, we are interested in investigating whether we can recover the 3D object shape by directly mining the knowledge of 2D GANs.

Original: Our resulting method could be viewed as a memory-efﬁcient way for 3D shape and texture generation, which shares the beneﬁts of both 2D and 3D generative models, e.g. , high quality and resolution samples, and a 3D interpretation of contents.
Revised: Our resulting method could be viewed as a memory-efﬁcient way for 3D shape and texture generation, which shares the beneﬁts of both 2D and 3D generative models, e.g. , high quality and resolution samples, and a 3D interpretation of contents.

Original: 3D-aware Generative Manipulation.
Revised: 3D-aware Generative Manipulation.

Original: Given a pre-trained GAN, there has been works trying to discover the latent space directions that manipulate the image content in a 3D-controllable manner.
Revised: Given a pre-trained GAN, there has been works trying to discover the latent space directions that manipulate the image content in a 3D-controllable manner.

Original: InterFaceGAN (Shen et al., 2020) learns to rotate human faces by learning latent direction from manual annotations.
Revised: InterFaceGAN (Shen et al., 2020) learns to rotate human faces by learning latent direction from manual annotations.

Original: In contrast, SeFa (Shen & Zhou, 2020) and GANSpace (H ¨ ark ¨ onen et al., 2020) discover meaningful latent directions without supervision, where some of them is coupled with content pose.
Revised: In contrast, SeFa (Shen & Zhou, 2020) and GANSpace (H ¨ ark ¨ onen et al., 2020) discover meaningful latent directions without supervision, where some of them is coupled with content pose.

Original: Recently, StyleRig (Tewari et al., 2020), DiscoFaceGAN (Deng et al., 2020), and RotateRender (Zhou et al., 2020) achieves 3D-aware manipulation on human face by using an external 3D human face model, i.e. , 3DMM (Blanz & Vetter, 1999), as a strong 3D shape prior.
Revised: Recently, StyleRig (Tewari et al., 2020), DiscoFaceGAN (Deng et al., 2020), and RotateRender (Zhou et al., 2020) achieves 3D-aware manipulation on human face by using an external 3D human face model, i.e. , 3DMM (Blanz & Vetter, 1999), as a strong 3D shape prior.

Original: Unlike these methods, our method could accurately manipulate on both pose and lighting factors without borrowing external 3D models or supervisions .
Revised: Unlike these methods, our method could accurately manipulate on both pose and lighting factors without borrowing external 3D models or supervisions .

Original: This allows our method to generalize beyond human faces, like cats and cars.
Revised: This allows our method to generalize beyond human faces, like cats and cars.

Original: Unsupervised 3D Shape Learning.
Revised: Unsupervised 3D Shape Learning.

Original: Unsupervised learning of 3D shapes from raw, monocular view images has attracted increasing attention recently, as it relieves the heavy burden on manual annotations.
Revised: Unsupervised learning of 3D shapes from raw, monocular view images has attracted increasing attention recently, as it relieves the heavy burden on manual annotations.

Original: The key challenge of this problem is the lack of multiple view or lighting images for a single instance.
Revised: The key challenge of this problem is the lack of multiple view or lighting images for a single instance.

Original: To tackle this problem, a number of works leverage external 3D shape models or weak supervisions like 2D key-points.
Revised: To tackle this problem, a number of works leverage external 3D shape models or weak supervisions like 2D key-points.

Original: For example, Gecer et al.
Revised: For example, Gecer et al.

Original: ; Kanazawa et al.
Revised: ; Kanazawa et al.

Original: (2018a);
Revised: (2018a);

Original: Sanyal et al.
Revised: Sanyal et al.

Original: ; Shang et al.
Revised: ; Shang et al.

Original: (2020) reconstruct the 3D shape of human faces or bodies using 3DMM (Blanz & Vetter, 1999) or SMPL (Loper et al., 2015) as a prior knowledge.
Revised: (2020) reconstruct the 3D shape of human faces or bodies using 3DMM (Blanz & Vetter, 1999) or SMPL (Loper et al., 2015) as a prior knowledge.

Original: And Tran & Liu (2018); Kanazawa et al.
Revised: And Tran & Liu (2018); Kanazawa et al.

Original: (2018b) adopt the guidance of 2D key-points to learn the 3D shape of face or objects.
Revised: (2018b) adopt the guidance of 2D key-points to learn the 3D shape of face or objects.

Original: The works of Tulsiani et al.
Revised: The works of Tulsiani et al.

Original: ; Goel et al.
Revised: ; Goel et al.

Original: (2020) further discard the need of former supervisions, but still rely on initial category-speciﬁc shape templates.
Revised: (2020) further discard the need of former supervisions, but still rely on initial category-speciﬁc shape templates.

Original: Some representative works that rely only on natural 2D images include Sahasrabudhe et al.
Revised: Some representative works that rely only on natural 2D images include Sahasrabudhe et al.

Original: Wu et al.
Revised: Wu et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2020) 1 , which learn to disentangle the viewpoint and shape for each image via autoencoders that explicitly model the rendering process.
Revised: (2020) 1 , which learn to disentangle the viewpoint and shape for each image via autoencoders that explicitly model the rendering process.

Original: However, these methods still rely on certain regularization to prevent trivial solutions.
Revised: However, these methods still rely on certain regularization to prevent trivial solutions.

Original: For example, Sahasrabudhe et al.
Revised: For example, Sahasrabudhe et al.

Original: (2019) adopts an autoencoder with a small bottleneck embedding to constrain the deformation ﬁeld, which tend to produce blur and imprecise results.
Revised: (2019) adopts an autoencoder with a small bottleneck embedding to constrain the deformation ﬁeld, which tend to produce blur and imprecise results.

Original: Wu et al.
Revised: Wu et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2020) adopt the assumption that many object shapes are symmetric, thus their inferred shapes sometimes miss the unsymmetric aspects of the objects in images.
Revised: (2020) adopt the assumption that many object shapes are symmetric, thus their inferred shapes sometimes miss the unsymmetric aspects of the objects in images.

Original: Our method in this work provides an alternative solution for the fore-mentioned ‘single view’ challenge, which is to obtain multiple view and lighting images by exploiting the image manifold modeled by 2D GANs.
Revised: Our method in this work provides an alternative solution for the fore-mentioned ‘single view’ challenge, which is to obtain multiple view and lighting images by exploiting the image manifold modeled by 2D GANs.

Original: Our method does not necessarily rely on the symmetry assumption on object
Revised: Our method does not necessarily rely on the symmetry assumption on object

Original: Renderer
Revised: Renderer

Original: V
Revised: V

Original: L
Revised: L

Original: D
Revised: D

Original: A view light depth albedo
Revised: A view light depth albedo

Original: ReconstructionLoss
Revised: ReconstructionLoss

Original: E
Revised: E

Original: G generator
Revised: G generator

Original: ReconstructionLoss view light
Revised: ReconstructionLoss view light

Original: Renderer
Revised: Renderer

Original: Random sample (a) Step(b)
Revised: Random sample (a) Step(b)

Original: Step
Revised: Step

Original: Renderer
Revised: Renderer

Original: V
Revised: V

Original: L
Revised: L

Original: D
Revised: D

Original: A view light depth albedo ReconstructionLoss
Revised: A view light depth albedo ReconstructionLoss

Original: V
Revised: V

Original: L view light
Revised: L view light

Original: Renderer projected samples (c) Step
Revised: Renderer projected samples (c) Step

Original: ReconstructionLoss
Revised: ReconstructionLoss

Original: Encoder
Revised: Encoder

Original: Encoder -Decoder
Revised: Encoder -Decoder

Original: Fixed Network
Revised: Fixed Network

Original: Network to be optimized pseudo samples input reconstruction input shape, thus better captures the asymmetric aspects of objects.
Revised: Network to be optimized pseudo samples input reconstruction input shape, thus better captures the asymmetric aspects of objects.

Original: In experiments, we demonstrate superior performance over a recent state-of-the-art method Unsup3d (Wu et al., 2020).
Revised: In experiments, we demonstrate superior performance over a recent state-of-the-art method Unsup3d (Wu et al., 2020).

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Preliminaries on Generative Adversarial Networks.
Revised: Preliminaries on Generative Adversarial Networks.

Original: We ﬁrst provide some preliminaries on
Revised: We ﬁrst provide some preliminaries on

Original: GANs before discussing how we exploit GANs for 3D shape recovery.
Revised: GANs before discussing how we exploit GANs for 3D shape recovery.

Original: GANs (Goodfellow et al., 2014) have achieved great successes in modeling natural images.
Revised: GANs (Goodfellow et al., 2014) have achieved great successes in modeling natural images.

Original: A GAN consists of a generator
Revised: A GAN consists of a generator

Original: G that maps a latent vector z to an image x ∈ R 3 × H × W , and a discriminator D that distinguishes between generated images and real ones.
Revised: G that maps a latent vector z to an image x ∈ R 3 × H × W , and a discriminator D that distinguishes between generated images and real ones.

Original: z is drawn from a prior distribution such as the multivariate normal distribution N (0 , I ) d .
Revised: z is drawn from a prior distribution such as the multivariate normal distribution N (0 , I ) d .

Original: GANs trained on natural images are able to continuously sample realistic images, thus could be used as an approximation of the natural image manifold (Zhu et al., 2016).
Revised: GANs trained on natural images are able to continuously sample realistic images, thus could be used as an approximation of the natural image manifold (Zhu et al., 2016).

Original: In this work, our study is mainly based on a representative GAN, StyleGAN2 (Karras et al., 2020b).
Revised: In this work, our study is mainly based on a representative GAN, StyleGAN2 (Karras et al., 2020b).

Original: The generator of StyleGAN2 consists of two parts, a mapping network F : Z (cid:55)→ W that maps the latent vector z in the input latent space Z to an intermediate latent vector w ∈ W , and a synthesis network that maps w to the output image.
Revised: The generator of StyleGAN2 consists of two parts, a mapping network F : Z (cid:55)→ W that maps the latent vector z in the input latent space Z to an intermediate latent vector w ∈ W , and a synthesis network that maps w to the output image.

Original: In the following sections, we refer to the synthesis network as G .
Revised: In the following sections, we refer to the synthesis network as G .

Original: The image manifold of StyleGAN2 thus could be deﬁned as M = { G ( w ) | w ∈ W} .
Revised: The image manifold of StyleGAN2 thus could be deﬁned as M = { G ( w ) | w ∈ W} .

Original: Such a two-stage structure makes it easier to perform GAN-inverson on top of StyleGAN2.
Revised: Such a two-stage structure makes it easier to perform GAN-inverson on top of StyleGAN2.

Original: i.e. , searching for a w that best reconstructs a real target image (Abdal et al., 2019).
Revised: i.e. , searching for a w that best reconstructs a real target image (Abdal et al., 2019).

Original: 3. F ROM 2D GAN S TO 3D S HAPES
Revised: 3. F ROM 2D GAN S TO 3D S HAPES

Original: In order to mine the viewpoint and lighting clues in the GAN image manifold, we need to explicitly model these factors.
Revised: In order to mine the viewpoint and lighting cues in the GAN image manifold, we need to explicitly model these factors.

Original: Here we follow the photo-geometric autoencoding design in Wu et al.
Revised: Here we follow the photo-geometric autoencoding design in Wu et al.

Original: For an image I ∈ R 3 × H × W , we adopt a function that is designed to predict four factors ( d , a , v , l ) , including a depth map d ∈ R H × W , an albedo image a ∈ R 3 × H × W , a viewpoint v ∈ R 6 , and a light direction l ∈ S 2 .
Revised: For an image I ∈ R 3 × H × W , we adopt a function that is designed to predict four factors ( d , a , v , l ) , including a depth map d ∈ R H × W , an albedo image a ∈ R 3 × H × W , a viewpoint v ∈ R 6 , and a light direction l ∈ S 2 .

Original: This function thus is implemented with four corresponding sub-networks ( D, A, V, L ) , as shown in Fig.
Revised: This function thus is implemented with four corresponding sub-networks ( D, A, V, L ) , as shown in Fig.

Original: 3 (a).
Revised: 3 (a).

Original: The four factors are recomposed to form the original image I via a rendering process Φ , which consists of lighting Λ and reprojection Π steps as follows:
Revised: The four factors are recomposed to form the original image I via a rendering process Φ , which consists of lighting Λ and reprojection Π steps as follows:

Original: [Equation]
Revised: [Equation]

Original: Here Λ performs shading with albedo a , depth map d , and light direction l , while Π performs viewpoint change and generates the image viewed from viewpoint v .
Revised: Here Λ performs shading with albedo a , depth map d , and light direction l , while Π performs viewpoint change and generates the image viewed from viewpoint v .

Original: The viewpoint change is realized with a differentiable renderer (Kato et al., 2018).
Revised: The viewpoint change is realized with a differentiable renderer (Kato et al., 2018).

Original: We put more details in the Appendix.
Revised: We put more details in the Appendix.

Original: Step 1: Using a Weak Shape Prior.
Revised: Step 1: Using a Weak Shape Prior.

Original: As our goal is to explore images of the same instance in the
Revised: As our goal is to explore images of the same instance in the

Original: GAN image manifold that have novel viewpoints and lighting conditions, we would like to seek for a hint on the change of viewpoint and lighting to guide the exploration.
Revised: GAN image manifold that have novel viewpoints and lighting conditions, we would like to seek for a hint on the change of viewpoint and lighting to guide the exploration.

Original: To achieve this, we make the observation that many objects including faces and cars have a somewhat convex shape prior.
Revised: To achieve this, we make the observation that many objects including faces and cars have a somewhat convex shape prior.

Original: We could thus resort to an ellipsoid as a weak prior to create ‘pseudo samples’ with varying viewpoints and lightings as the hint.
Revised: We could thus resort to an ellipsoid as a weak prior to create ‘pseudo samples’ with varying viewpoints and lightings as the hint.

Original: Speciﬁcally, as shown in Fig.3 (a), given an image sample I from the GAN, we initialize the depth map d to have an ellipsoid shape.
Revised: Speciﬁcally, as shown in Fig.3 (a), given an image sample I from the GAN, we initialize the depth map d to have an ellipsoid shape.

Original: With an off-the-shelf scene parsing model (Zhao et al., 2017), we could position the ellipsoid to be roughly aligned with the object in the image.
Revised: With an off-the-shelf scene parsing model (Zhao et al., 2017), we could position the ellipsoid to be roughly aligned with the object in the image.

Original: Then the albedo network A is optimized with the reconstruction objective L ( I , ˆ I ) , where ˆ I is calculated via Eq.
Revised: Then the albedo network A is optimized with the reconstruction objective L ( I , ˆ I ) , where ˆ I is calculated via Eq.

Original: (1), and L is a weighted combination of L1 loss and perceptual loss (Johnson et al., 2016).
Revised: (1), and L is a weighted combination of L1 loss and perceptual loss (Johnson et al., 2016).

Original: The viewpoint v and lighting l are initialized with a canonical setting, i.e. , v 0 = 0 and lighting is from the front.
Revised: The viewpoint v and lighting l are initialized with a canonical setting, i.e. , v 0 = 0 and lighting is from the front.

Original: This allows us to have an initial guess about the four factors d 0 , a 0 , v 0 , l 0 , with d 0 being the weak shape prior.
Revised: This allows us to have an initial guess about the four factors d 0 , a 0 , v 0 , l 0 , with d 0 being the weak shape prior.

Original: Step 2: Sampling and Projecting to the GAN Image Manifold.
Revised: Step 2: Sampling and Projecting to the GAN Image Manifold.

Original: With the above shape prior as an initialization, we are able to create ‘pseudo samples’ by sampling a number of random viewpoints { v i | i = 1 , 2 , ..., m } and lighting directions { l i | i
Revised: With the above shape prior as an initialization, we are able to create ‘pseudo samples’ by sampling a number of random viewpoints { v i | i = 1 , 2 , ..., m } and lighting directions { l i | i

Original: = 1 , 2 , ..., m } .
Revised: = 1 , 2 , ..., m } .

Original: Speciﬁcally, we use { v i } and { l i } along with d 0 and a 0 to render pseudo samples { I i } via Eq.
Revised: Speciﬁcally, we use { v i } and { l i } along with d 0 and a 0 to render pseudo samples { I i } via Eq.

Original: { v i } and { l i } are randomly sampled from some prior distributions, like the multi-variate normal distribution.
Revised: { v i } and { l i } are randomly sampled from some prior distributions, like the multi-variate normal distribution.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: (b), although these pseudo samples have unnatural distortions and shadows, they provide clues on how the face rotates and how the light changes.
Revised: (b), although these pseudo samples have unnatural distortions and shadows, they provide cues on how the face rotates and how the light changes.

Original: In order to leverage such clues to guide novel viewpoint and lighting direction exploration in the
Revised: In order to leverage such cues to guide novel viewpoint and lighting direction exploration in the

Original: GAN image manifold, we perform GAN inversion to these pseudo samples, i.e. , reconstruct them with the GAN generator G .
Revised: GAN image manifold, we perform GAN inversion to these pseudo samples, i.e. , reconstruct them with the GAN generator G .

Original: Speciﬁcally, we train an encoder E that learns to predict the intermediate latent vector w i for each sample.
Revised: Speciﬁcally, we train an encoder E that learns to predict the intermediate latent vector w i for each sample.

Original: Unlike prior works that directly predict the latent vector, in our setting the latent vector w for the original sample I is known, thus the encoder E only need to predict an offset ∆ w i =
Revised: Unlike prior works that directly predict the latent vector, in our setting the latent vector w for the original sample I is known, thus the encoder E only need to predict an offset ∆ w i =

Original: E ( I i ) that encodes the difference between the pseudo sample
Revised: E ( I i ) that encodes the difference between the pseudo sample

Original: I i and the original sample I , which is much easier.
Revised: I i and the original sample I , which is much easier.

Original: Thus the optimization goal is:
Revised: Thus the optimization goal is:

Original: [Equation]
Revised: [Equation]

Original: where m is the number of samples, θ E is the parameter of the encoder E , λ 1 is the regularization coefﬁcient, and L (cid:48) is a distance metric for images.
Revised: where m is the number of samples, θ E is the parameter of the encoder E , λ 1 is the regularization coefﬁcient, and L (cid:48) is a distance metric for images.

Original: Following Pan et al.
Revised: Following Pan et al.

Original: (2020), we adopt the L distance of the discriminator features as the distance metric L (cid:48) , which is shown to work better for
Revised: (2020), we adopt the L distance of the discriminator features as the distance metric L (cid:48) , which is shown to work better for

Original: GAN samples.
Revised: GAN samples.

Original: The regularization term (cid:107) E ( I i ) (cid:107) 2 is to prevent the latent offset to grow too large during training.
Revised: The regularization term (cid:107) E ( I i ) (cid:107) 2 is to prevent the latent offset to grow too large during training.

Original: As shown in Fig.3 (b), to achieve better reconstruction, the projected samples { ˜ I i =
Revised: As shown in Fig.3 (b), to achieve better reconstruction, the projected samples { ˜ I i =

Original: G ( E ( I i )+ w ) } should have similar viewpoint and lighting changes as the pseudo samples { I i } .
Revised: G ( E ( I i )+ w ) } should have similar viewpoint and lighting changes as the pseudo samples { I i } .

Original: In the meanwhile, the generator G could regularize the projected samples to lie in the natural image manifold, thus ﬁxing the unnatural distortions and shadows in the pseudo samples.
Revised: In the meanwhile, the generator G could regularize the projected samples to lie in the natural image manifold, thus ﬁxing the unnatural distortions and shadows in the pseudo samples.

Original: In the next step, we discuss how to exploit these projected samples to infer the 3D shape.
Revised: In the next step, we discuss how to exploit these projected samples to infer the 3D shape.

Original: Step 3: Learning the 3D Shape.
Revised: Step 3: Learning the 3D Shape.

Original: The above projected samples { ˜ I i } provide images of multiple viewpoint and lighting conditions while having nearly the same object content, e.g. , face identity.
Revised: The above projected samples { ˜ I i } provide images of multiple viewpoint and lighting conditions while having nearly the same object content, e.g. , face identity.

Original: Such ‘single instance multiple view&lighting’ setting makes it possible to learn the underlying
Revised: Such ‘single instance multiple view&lighting’ setting makes it possible to learn the underlying

Original: 3D shape with aforementioned photo-geometric autoencoding model.
Revised: 3D shape with aforementioned photo-geometric autoencoding model.

Original: Speciﬁcally, as illustrated in
Revised: Speciﬁcally, as illustrated in

Original: Fig.3 (c), the viewpoint network V and the lighting network L predict instance-speciﬁc view ˜ v i and lighting
Revised: Fig.3 (c), the viewpoint network V and the lighting network L predict instance-speciﬁc view ˜ v i and lighting

Original: ˜ l
Revised: ˜ l

Original: i for each sample ˜
Revised: i for each sample ˜

Original: I
Revised: I

Original: i , while the depth network D and albedo network A output the shared depth ˜ v and albedo ˜
Revised: i , while the depth network D and albedo network A output the shared depth ˜ v and albedo ˜

Original: a with the original sample I as input.
Revised: a with the original sample I as input.

Original: The four networks are jointly optimized with the following reconstruction objective:
Revised: The four networks are jointly optimized with the following reconstruction objective:

Original: [Equation]
Revised: [Equation]

Original: where θ D , θ A , θ V , θ L are the parameters of network D, A, V, L respectively, and L smooth (cid:0) D
Revised: where θ D , θ A , θ V , θ L are the parameters of network D, A, V, L respectively, and L smooth (cid:0) D

Original: ( I ) (cid:1) is a smoothness term deﬁned the same way as in (Zhou et al., 2017).
Revised: ( I ) (cid:1) is a smoothness term deﬁned the same way as in (Zhou et al., 2017).

Original: Note that we also reconstruct the original sample I , i.e. , use I as an additional ˜
Revised: Note that we also reconstruct the original sample I , i.e. , use I as an additional ˜

Original: I i , to prevent the infered factors to deviate from the original sample.
Revised: I i , to prevent the infered factors to deviate from the original sample.

Original: Here we do not use the sampled { v i } and { l i } in step 2, as the viewpoints and lighting directions of the projected samples may not accurately match the pseudo samples, thus we learn to predict them with V and L instead.
Revised: Here we do not use the sampled { v i } and { l i } in step 2, as the viewpoints and lighting directions of the projected samples may not accurately match the pseudo samples, thus we learn to predict them with V and L instead.

Original: The above reconstruction loss would decrease only when the underlying depth, albedo, and all samples’ viewpoints and lighting directions are correctly inferred.
Revised: The above reconstruction loss would decrease only when the underlying depth, albedo, and all samples’ viewpoints and lighting directions are correctly inferred.

Original: As shown in Fig.3 (c), the object shape, parameterized by depth, does shift toward the true underlying shape, e.g. , shape of a human face.
Revised: As shown in Fig.3 (c), the object shape, parameterized by depth, does shift toward the true underlying shape, e.g. , shape of a human face.

Original: And the unnatural distortions and shadows that exist in the pseudo samples are mitigated in the reconstructions.
Revised: And the unnatural distortions and shadows that exist in the pseudo samples are mitigated in the reconstructions.

Original: Iterative Self-reﬁnement.
Revised: Iterative Self-reﬁnement.

Original: While the above three steps produce a more accurate 3D object shape than the prior shape, it may miss some details like the wrinkles on the face.
Revised: While the above three steps produce a more accurate 3D object shape than the prior shape, it may miss some details like the wrinkles on the face.

Original: This is because the projected samples { ˜ I i } may not accurately preserve all semantics of the original instance in I , as the pseudo samples are obtained from an approximated weak shape prior.
Revised: This is because the projected samples { ˜ I i } may not accurately preserve all semantics of the original instance in I , as the pseudo samples are obtained from an approximated weak shape prior.

Original: Fortunately, we can repeat the cycle of these three steps multiple times by regarding the reﬁned 3D shape and trained networks in the previous cycle as the initialized shape and networks in a new cycle.
Revised: Fortunately, we can repeat the cycle of these three steps multiple times by regarding the reﬁned 3D shape and trained networks in the previous cycle as the initialized shape and networks in a new cycle.

Original: This allows us to reﬁne the 3D shape and projected samples progressively.
Revised: This allows us to reﬁne the 3D shape and projected samples progressively.

Original: In our experiments, we use four cycles of these three steps unless otherwise stated.
Revised: In our experiments, we use four cycles of these three steps unless otherwise stated.

Original: 3. D ISCUSSION
Revised: 3. D ISCUSSION

Original: Regularizing the Latent Offset.
Revised: Regularizing the Latent Offset.

Original: In the Eq. 2 of Step 2, the latent offset ∆ w i =
Revised: In the Eq. 2 of Step 2, the latent offset ∆ w i =

Original: E ( I i ) is only regularized with a weak L2 norm term, thus the resulting latent vector ∆ w i + w may fall out of the intermediate latent distribution W of the GAN.
Revised: E ( I i ) is only regularized with a weak L2 norm term, thus the resulting latent vector ∆ w i + w may fall out of the intermediate latent distribution W of the GAN.

Original: This could possibly cause the projected samples { ˜ I i } to inherit some unnatural distortions in the pseudo samples { I i } , thus stronger regularization on ∆ w i is required.
Revised: This could possibly cause the projected samples { ˜ I i } to inherit some unnatural distortions in the pseudo samples { I i } , thus stronger regularization on ∆ w i is required.

Original: To address this issue, we propose a new regularization strategy, which is to restrict ∆ w i to be a valid offset created by the mapping network F as ∆ w i
Revised: To address this issue, we propose a new regularization strategy, which is to restrict ∆ w i to be a valid offset created by the mapping network F as ∆ w i

Original: = F ( E ( I i ))
Revised: = F ( E ( I i ))

Original: − F ( 0 ) .
Revised: − F ( 0 ) .

Original: We provide more discussion on this design in the Appendix.
Revised: We provide more discussion on this design in the Appendix.

Original: Joint Training of Multiple Instances.
Revised: Joint Training of Multiple Instances.

Original: Note that our discussion above involves only a single target image I , i.e. , our method is used in an instance-speciﬁc manner.
Revised: Note that our discussion above involves only a single target image I , i.e. , our method is used in an instance-speciﬁc manner.

Original: However, our pipeline could also be jointly trained on multiple instances to achieve a better generalization ability and a faster convergence speed on new samples.
Revised: However, our pipeline could also be jointly trained on multiple instances to achieve a better generalization ability and a faster convergence speed on new samples.

Original: The extension to joint training is straightforward, whose details are included in the Appendix.
Revised: The extension to joint training is straightforward, whose details are included in the Appendix.

Original: The key difference between training with multiple instances and a single instance is that, since there are multiple target instances with diverse viewpoint and lighting variations, the absolute latent offset E ( I i ) in Eq.
Revised: The key difference between training with multiple instances and a single instance is that, since there are multiple target instances with diverse viewpoint and lighting variations, the absolute latent offset E ( I i ) in Eq.

Original: 2 should be replaced with a relative latent offset
Revised: 2 should be replaced with a relative latent offset

Original: E ( I i ) − E
Revised: E ( I i ) − E

Original: ( I ) , which could generalize better across different target instances.
Revised: ( I ) , which could generalize better across different target instances.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Implementation Details.
Revised: Implementation Details.

Original: We ﬁrst evaluate our method on 3D shape recovery, and then show its application to 3D-aware image manipulation.
Revised: We ﬁrst evaluate our method on 3D shape recovery, and then show its application to 3D-aware image manipulation.

Original: The datasets used include CelebA (Liu et al., 2015),
Revised: The datasets used include CelebA (Liu et al., 2015),

Original: BFM (Paysan et al., 2009), a combined cat dataset (Zhang et al., 2008; Parkhi et al., 2012), LSUN
Revised: BFM (Paysan et al., 2009), a combined cat dataset (Zhang et al., 2008; Parkhi et al., 2012), LSUN

Original: Car (Yu et al., 2015), and LSUN Church (Yu et al., 2015), all of which are unconstrained RGB images.
Revised: Car (Yu et al., 2015), and LSUN Church (Yu et al., 2015), all of which are unconstrained RGB images.

Original: For BFM, we use the same one as in Wu et al.
Revised: For BFM, we use the same one as in Wu et al.

Original: We adopt StyleGAN2 pre-trained on these datasets in our experiments.
Revised: We adopt StyleGAN2 pre-trained on these datasets in our experiments.

Original: We recommend readers to refer to the Appendix for more implementation details and qualitative results.
Revised: We recommend readers to refer to the Appendix for more implementation details and qualitative results.

Original: [Figure]
Revised: [Figure]

Original: U n s up3d ( a )
Revised: U n s up3d ( a )

Original: O u r s
Revised: O u r s

Original: [Figure]
Revised: [Figure]

Original: up3d
Revised: up3d

Original: Relighting
Revised: Relighting

Original: 4. U NSUPERVISED 3D S HAPE R ECONSTRUCTION
Revised: 4. U NSUPERVISED 3D S HAPE R ECONSTRUCTION

Original: Qualitative Evaluation.
Revised: Qualitative Evaluation.

Original: The qualitative results of our method and Unsup3d (Wu et al., 2020) are shown in Fig.1, Fig.4, and Fig.5.
Revised: The qualitative results of our method and Unsup3d (Wu et al., 2020) are shown in Fig.1, Fig.4, and Fig.5.

Original: Our method recovers the 3D shapes with high-quality across human faces, cats, cars, and buildings.
Revised: Our method recovers the 3D shapes with high-quality across human faces, cats, cars, and buildings.

Original: For example, the wrinkles on human faces and the edges and ﬂat surfaces on cars are well captured.
Revised: For example, the wrinkles on human faces and the edges and ﬂat surfaces on cars are well captured.

Original: The results of Unsup3d, while good, tend to miss some asymmetric aspects of human faces, e.g. , its predicted eye directions always point to the front.
Revised: The results of Unsup3d, while good, tend to miss some asymmetric aspects of human faces, e.g. , its predicted eye directions always point to the front.

Original: Besides, as shown in Fig.4 (b) and Fig.5, Unsup3d has difﬁculty in generalizing to datasets with asymmetric objects or large viewpoint variations like buildings and cars, while our method does not.
Revised: Besides, as shown in Fig.4 (b) and Fig.5, Unsup3d has difﬁculty in generalizing to datasets with asymmetric objects or large viewpoint variations like buildings and cars, while our method does not.

Original: Quantitative Evaluation.
Revised: Quantitative Evaluation.

Original: Quantitative comparison is conducted on the BFM dataset.
Revised: Quantitative comparison is conducted on the BFM dataset.

Original: Following (Wu et al., 2020), we report the scale-invariant depth error (SIDE) and mean angle deviation (MAD) score as the evaluation metrics.
Revised: Following (Wu et al., 2020), we report the scale-invariant depth error (SIDE) and mean angle deviation (MAD) scores as the evaluation metrics.

Original: In order to evaluate on the test set images instead of GAN generated samples, we need to ﬁrst conduct GAN-inversion, i.e. , ﬁnding the intermediate latent vectors { w } that best reconstruct these images.
Revised: In order to evaluate on the test set images instead of GAN generated samples, we need to ﬁrst conduct GAN inversion, i.e. , ﬁnding the intermediate latent vectors { w } that best reconstruct these images.

Original: We adopt the method in Karras et al.
Revised: We adopt the method in Karras et al.

Original: (2020b), which achieves satisfactory reconstruction results, as shown in the ﬁrst row of Fig.6.
Revised: (2020b), which achieves satisfactory reconstruction results, as shown in the ﬁrst row of Fig.6.

Original: To achieve better generalization, our method is pre-trained on only 200 training images before testing (the training set has 160k images in total).
Revised: To achieve better generalization, our method is pre-trained on only 200 training images before testing (the training set has 160k images in total).

Original: Due to the additional testing time brought by the GAN-inversion and instance-speciﬁc training steps, we report results on the ﬁrst 500 images of the test set.
Revised: Due to the additional testing time brought by the GAN inversion and instance-speciﬁc training steps, we report results on the ﬁrst 500 images of the test set.

Original: [Table]
Revised: [Table]

Original: O u r s
Revised: O u r s

Original: U n s up3d
Revised: U n s up3d

Revised: Input

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: mesh and GANs.

Revised: The input of the ﬁrst row is a real natural image.

Revised: Our method achieves photo-realistic manipulation effects obeying the objects’ underlying 3D structures.

Original: Although the above GAN-inversion step could introduce additional error due to imperfect reconstruction, our method still noticeably outperforms Unsup3d and other trivial baselines, as shown in
Revised: Although the above GAN inversion step could introduce additional error due to imperfect reconstruction, our method still noticeably outperforms Unsup3d and other trivial baselines, as shown in

Original: Tab.1.
Revised: Tab.1.

Original: It is worth noting that our method does not necessarily rely on the symmetric assumption.
Revised: It is worth noting that our method does not necessarily rely on the symmetric assumption.

Original: To demonstrate this, we also report results of our method and Unsup3d when both are learned without the symmetric assumption.
Revised: To demonstrate this, we also report results of our method and Unsup3d when both are learned without the symmetric assumption.

Original: As No.(7)(8) in Tab.1 and Fig.6 show, in this setting our method significantly outperforms Unsup3d, which tends to produce messy results.
Revised: As No.(7)(8) in Tab.1 and Fig.6 show, in this setting our method significantly outperforms Unsup3d, which tends to produce messy results.

Original: Furthermore, No.(5) provides the results without using the latent offset regularization described in Sec.3.2, where the performance decreases notably, demonstrating the effectiveness of the proposed regularization technique.
Revised: Furthermore, No.(5) provides the results without using the latent offset regularization described in Sec.3.2, where the performance decreases notably, demonstrating the effectiveness of the proposed regularization technique.

Original: We further study the effects of using different shape prior.
Revised: We further study the effects of using different shape prior.

Original: The setting is the same as Tab.1, and the results are shown in Tab.2.
Revised: The setting is the same as Tab.1, and the results are shown in Tab.2.

Original: Some variations like asymmetric, shifted, and weaker shape prior only slightly affect the performance, showing that the result is not very sensitive to the shape prior.
Revised: Some variations like asymmetric, shifted, and weaker shape prior only slightly affect the performance, showing that the result is not very sensitive to the shape prior.

Original: But the results would get worse for ﬂat shape, as it could not indicate viewpoint and lighting change.
Revised: But the results would get worse for ﬂat shape, as it could not indicate viewpoint and lighting change.

Original: 4. 3D- AWARE I MAGE M ANIPULATION
Revised: 4. 3D- AWARE I MAGE M ANIPULATION

Original: Object Rotation and Relighting.
Revised: Object Rotation and Relighting.

Original: After training, the recovered 3D shape and the encoder E immediately allow various 3D-aware image manipulation effects.
Revised: After training, the recovered 3D shape and the encoder E immediately allow various 3D-aware image manipulation effects.

Original: In Fig.7, we show two manipulation effects, including object rotation and relighting.
Revised: In Fig.7, we show two manipulation effects, including object rotation and relighting.

Original: For each effect, we show results rendered using the recovered 3D shape and albedo, and their corresponding projections in the GAN manifold via the encoder E .
Revised: For each effect, we show results rendered using the recovered 3D shape and albedo, and their corresponding projections in the GAN manifold via the encoder E .

Original: The results rendered from 3D shapes strictly follow the underlying structure of the object, while their projections in GAN are more photo-realistic.
Revised: The results rendered from 3D shapes strictly follow the underlying structure of the object, while their projections in GAN are more photo-realistic.

Original: As mentioned before, our method is applicable to real natural images via GAN-inversion, and the ﬁrst row in Fig.7 is an example, where our method also works well.
Revised: As mentioned before, our method is applicable to real natural images via GAN inversion, and the ﬁrst row in Fig.7 is an example, where our method also works well.

Revised: The metrics are identity distances measured as angles in the ArcFace feature embeddings.

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original:  Identity-preserving Face Rotation.
Revised: t [Table] [Figure] Identity-preserving Face Rotation.

Original: We compare our method with other unsupervised methods that alsocould achieve face rotation with GANs, including HoloGAN (Nguyen-Phuoc et al., 2019),
Revised: We compare our method with other unsupervised methods that also achieve face rotation with GANs, including HoloGAN (Nguyen-Phuoc et al., 2019),

Original: GANSpace (H¨ark¨onen et al., 2020), and SeFa (Shen & Zhou, 2020).
Revised: GANSpace (H ¨ ark ¨ onen et al., 2020), and SeFa (Shen & Zhou, 2020).

Original: Speciﬁcally, for each method, we randomly sample 100 face images, and for each face, we shift its yaw angle from -20 degree to 20 degree, obtaining 20 samples with different poses.
Revised: Speciﬁcally, for each method, we randomly sample 100 face images, and for each face, we shift its yaw angle from -20 degree todegree, obtaining 20 samples with different poses.

Original: An off-the-shelf face identity detection model ArcFace (Deng et al., 2019) is then used to evaluate how much the face identities change duringface rotation.
Revised: A face identity detection model ArcFace (Deng et al., 2019) is then used to evaluate how much the face identities change during rotation.

Original: More details are included in the Appendix.
Revised: More details are included in the Appendix.

Original: Since our methodcould explicitly recover the 3D face shapes, results rendered from these recovered 3D shapes signiﬁcantly outperform other counterparts, as shown in Tab.3 and Fig.8.
Revised: Since our method explicitly recover the 3D face shapes, results rendered from these recovered 3D shapes signiﬁcantly outperform other counterparts, as shown in Tab.3 and Fig.8.

Original: In the alternative attempt where we directly operate in the GAN manifold following the projected directions of the recovered 3D shapes, face identities are also better preserved.[Figure]
Revised: In the alternative attempt where we directly operate in the GAN manifold following the projected directions of the recovered 3D shapes, face identities are also better preserved.

Original: The metrics are identity distances measured as angles in the ArcFace feature

Original: Method

Original: [Table]

Original: [Table]

Original: GANSpace

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: Discussion.
Revised: Discussion.

Original: We have shown that our method is applicable to many object categories.
Revised: We have shown that our method is applicable to many object categories.

Original: However, for objects with more sophisticated shapes like horses, a simple convex shape prior may not well reﬂect the viewpoint and lighting variations, and thus the 3D shapes could not be very accurately inferred.
Revised: However, for objects with more sophisticated shapes like horses, a simple convex shape prior may not well reﬂect the viewpoint and lighting variations, and thus the 3D shapes could not be very accurately inferred.

Original: The second row of Fig.9 provides one example.
Revised: The second row of Fig.9 provides one example.

Original: Another limitation of our method is that our 3D mesh is parameterized by a depth map, which could not model the back-side shape of objects, as the ﬁrst row of Fig.9 shows.
Revised: Another limitation of our method is that our 3D mesh is parameterized by a depth map, which could not model the back-side shape of objects, as the ﬁrst row of Fig.9 shows.

Original: This could possibly be addressed by adopting a better parameterization of the 3D mesh.
Revised: This could possibly be addressed by adopting a better parameterization of the 3D mesh.

Original: Despite these limitations, our method still captures some aspects of the horses’ shapes, like the rough shape of the head and belly, and achieves reasonable relighting effects.
Revised: Despite these limitations, our method still captures some aspects of the horses’ shapes, like the rough shape of the head and belly, and achieves reasonable relighting effects.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have presented the ﬁrst method that directly leverages off-the-shelf 2D GANs to recover 3D object shapes from images.
Revised: We have presented the ﬁrst method that directly leverages off-the-shelf 2D GANs to recover 3D object shapes from images.

Original: We found that existing 2D GANs inherently capture sufﬁcient knowledge to recover 3D shapes for many object categories, including human faces, cats, cars, and buildings.
Revised: We found that existing 2D GANs inherently capture sufﬁcient knowledge to recover 3D shapes for many object categories, including human faces, cats, cars, and buildings.

Original: Based on a weak convex shape prior, our method could explore the viewpoint and lighting variations in the GAN image manifold, and exploit these variations to reﬁne the underlying object shape in an iterative manner.
Revised: Based on a weak convex shape prior, our method could explore the viewpoint and lighting variations in the GAN image manifold, and exploit these variations to reﬁne the underlying object shape in an iterative manner.

Original: We have further shown the application of our method on 3D-aware image manipulations including object rotation and relighting.
Revised: We have further shown the application of our method on 3D-aware image manipulations including object rotation and relighting.

Original: Our results uncover the potential of 2D GANs in modeling the underlying 3D geometry of 2D image manifold. 
Revised: Our results uncover the potential of 2D GANs in modeling the underlying 3D geometry of 2D image manifold. Acknowledgment.

Revised: This research was conducted in collaboration with SenseTime.

Revised: This work is supported by NTU NAP and A*STAR through the Industry Alignment Fund - Industry Collaboration

Revised: Projects Grant.

Revised: This work is also supported by the HK General Research Fund No.27208720.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In this appendix, we provide more qualitative results and the implementation details.
Revised: In this appendix, we provide more qualitative results and the implementation details.

Original: A. Q UALITATIVE E XAMPLES
Revised: A. Q UALITATIVE E XAMPLES

Original: Fig.10 provides more qualitative results of our method, including 3D shape reconstruction and 3Daware image manipulations.
Revised: Fig.10 provides more qualitative results of our method, including 3D shape reconstruction and 3Daware image manipulations.

Original: The effects of iterative training is shown in Fig.11, where the object shapes become more precise at latter training stages.
Revised: The effects of iterative training is shown in Fig.11, where the object shapes become more precise at latter training stages.

Original: Fig.12 shows that our method works well without using the symmetry assumption.
Revised: Fig.12 shows that our method works well without using the symmetry assumption.

Original: Fig.13 shows that our method achieves reasonable results for many challenging cases.
Revised: Fig.13 shows that our method achieves reasonable results for many challenging cases.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Effects of shape prior.
Revised: Effects of shape prior.

Original: Fig.14 shows the qualitative results of using different shape priors.
Revised: Fig.14 shows the qualitative results of using different shape priors.

Original: Unlike the results in Tab.2, here joint training is not adopted to better showcase the effects of different shape.
Revised: Unlike the results in Tab.2, here joint training is not adopted to better showcase the effects of different shape.

Original: When the ellipsoid is shifted by 1/6 image width or is asymmetric, the shape could be progressively reﬁned during the iterative training process.
Revised: When the ellipsoid is shifted by 1/6 image width or is asymmetric, the shape could be progressively reﬁned during the iterative training process.

Original: But the results would get worse if the shift is too large or a ﬂat shape is used, because the viewpoint and lighting changes could not be revealed with these shapes.
Revised: But the results would get worse if the shift is too large or a ﬂat shape is used, because the viewpoint and lighting changes could not be revealed with these shapes.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Effects of dataset size.
Revised: Effects of dataset size.

Original: Fig.4 shows the effects of dataset size used for training the GAN.
Revised: Fig.4 shows the effects of dataset size used for training the GAN.

Original: It can be seen that the performance only drops marginally even when the dataset size is reduced from 160k to 3k.
Revised: It can be seen that the performance only drops marginally even when the dataset size is reduced from 160k to 3k.

Original: And the performance get notably worse when the dataset size reduces to 1k, as the image quality of the GAN starts to deteriorate given insufﬁcient data.
Revised: And the performance get notably worse when the dataset size reduces to 1k, as the image quality of the GAN starts to deteriorate given insufﬁcient data.

Original: This could potentially be addressed by recent data augmentation strategy for GANs (Karras et al., 2020a).
Revised: This could potentially be addressed by recent data augmentation strategy for GANs (Karras et al., 2020a).

Original: A. I MPLEMENTATION D ETIALS A.2. P HOTO - GEOMETRIC A UTOENCODING
Revised: A. I MPLEMENTATION D ETIALS A.2. P HOTO - GEOMETRIC A UTOENCODING

Original: The photo-geometric autoencoding model we used mainly follows the work of Wu et al.
Revised: The photo-geometric autoencoding model we used mainly follows the work of Wu et al.

Original: Here we describe the Λ and Π in Eq.1.
Revised: Here we describe the Λ and Π in Eq.1.

Original: J = Λ( d , a , l ) produces the illuminated texture with the albedo a , depth map d , and light direction l .
Revised: J = Λ( d , a , l ) produces the illuminated texture with the albedo a , depth map d , and light direction l .

Original: Speciﬁcally, we ﬁrst calculate the normal map n of the depth map d , which describes the normal direction of the surface for each pixel position ( i, j ) .
Revised: Speciﬁcally, we ﬁrst calculate the normal map n of the depth map d , which describes the normal direction of the surface for each pixel position ( i, j ) .

Original: Then the illuminated texture is calculate with
Revised: Then the illuminated texture is calculate with

Original: J ij =
Revised: J ij =

Original: ( k s + k d max 0 , (cid:104) l, n ij (cid:105) ) · a ij , where k s and k d are the scalar weight for the ambient and diffuse terms, and are predicted by the lighting network V .
Revised: ( k s + k d max 0 , (cid:104) l, n ij (cid:105) ) · a ij , where k s and k d are the scalar weight for the ambient and diffuse terms, and are predicted by the lighting network V .

Original: The lighting direction is described as l =
Revised: The lighting direction is described as l =

Original: ( l x , l y , 1) T / ( l 2 x + l 2 y + 1) 0 .
Revised: ( l x , l y , 1) T / ( l 2 x + l 2 y + 1) 0 .

Original: 5 , where l x and l y are also the predictions of the lighting network.
Revised: 5 , where l x and l y are also the predictions of the lighting network.

Original: Given the above illuminated texture J , the reprojection function Π would map it to a new viewpoint v as ˆ I =
Revised: Given the above illuminated texture J , the reprojection function Π would map it to a new viewpoint v as ˆ I =

Original: Π( J , d , v ) .
Revised: Π( J , d , v ) .

Original: To do this, we denote the 3D point in the reference frame of the camera as
Revised: To do this, we denote the 3D point in the reference frame of the camera as

Original: P =
Revised: P =

Original: ( P x , P y , P z ) ∈ R 3 , which is projected to pixel p
Revised: ( P x , P y , P z ) ∈ R 3 , which is projected to pixel p

Original: = ( i, j, 1) via:
Revised: = ( i, j, 1) via:

Original: [Equation]
Revised: [Equation]

Original: where θ FOV is the ﬁeld of view of the perspective camera, and is assume to be θ FOV ≈ 10 ◦ .
Revised: where θ FOV is the ﬁeld of view of the perspective camera, and is assume to be θ FOV ≈ 10 ◦ .

Original: Given a depth map d which describes a depth value d ij for each pixel ( i, j ) , we would have P = d ij · K − 1 p .
Revised: Given a depth map d which describes a depth value d ij for each pixel ( i, j ) , we would have P = d ij · K − 1 p .

Original: The viewpoint v ∈ R 6 represents the rotation angles ( v 1:3 ) and translations ( v 4:6 ) along x , y and z axes.
Revised: The viewpoint v ∈ R 6 represents the rotation angles ( v 1:3 ) and translations ( v 4:6 ) along x , y and z axes.

Original: v thus describes an Euclidean transform ( R , T ) ∈ SE (3) , which transforms 3D points to the predicted viewpoint.
Revised: v thus describes an Euclidean transform ( R , T ) ∈ SE (3) , which transforms 3D points to the predicted viewpoint.

Original: The mapping from a pixel ( i, j ) in the original view to the pixel ( i (cid:48) , j (cid:48) ) in the predicted view could then be described by the warping function η d,v : ( i, j )
Revised: The mapping from a pixel ( i, j ) in the original view to the pixel ( i (cid:48) , j (cid:48) ) in the predicted view could then be described by the warping function η d,v : ( i, j )

Original: (cid:55)→ ( i (cid:48) , j (cid:48) ) as:
Revised: (cid:55)→ ( i (cid:48) , j (cid:48) ) as:

Original: [Equation]
Revised: [Equation]

Original: (5) where p (cid:48)
Revised: (5) where p (cid:48)

Original: =
Revised: =

Original: ( i (cid:48) , j (cid:48) , 1) .
Revised: ( i (cid:48) , j (cid:48) , 1) .

Original: Finally, Π applies this warp to the illuminated texture J to obtain the reprojected image ˆ I =
Revised: Finally, Π applies this warp to the illuminated texture J to obtain the reprojected image ˆ I =

Original: Π( J , d , v ) as ˆ I i (cid:48)
Revised: Π( J , d , v ) as ˆ I i (cid:48)

Original: j (cid:48) =
Revised: j (cid:48) =

Original: J ij , where ( i, j ) =
Revised: J ij , where ( i, j ) =

Original: η − 1 d,v ( i (cid:48) , j (cid:48) ) .
Revised: η − 1 d,v ( i (cid:48) , j (cid:48) ) .

Original: Here η − 1 d,v is the backward warping of η d,v , which is achieved via a differentiable renderer Kato et al.
Revised: Here η − 1 d,v is the backward warping of η d,v , which is achieved via a differentiable renderer Kato et al.

Original: (2018) in the same way as Wu et al.
Revised: (2018) in the same way as Wu et al.

Original: A.2. R EGULARIZING THE L ATENT O FFSET
Revised: A.2. R EGULARIZING THE L ATENT O FFSET

Original: As described in Section 3.2, in order to keep the projected samples { ˜ I i } in the natural image manifold, stronger regularization on the predicted latent offset ∆ w i is needed.
Revised: As described in Section 3.2, in order to keep the projected samples { ˜ I i } in the natural image manifold, stronger regularization on the predicted latent offset ∆ w i is needed.

Original: One possible solution is to let the encoder predict the offset in the original latent space, i.e. , the one before the mapping network F , then the output of the generator would be G (cid:0) F ( E ( I i ) + z ) (cid:1) , where z is the original latent vector corresponding to I .
Revised: One possible solution is to let the encoder predict the offset in the original latent space, i.e. , the one before the mapping network F , then the output of the generator would be G (cid:0) F ( E ( I i ) + z ) (cid:1) , where z is the original latent vector corresponding to I .

Original: However, one issue of this solution is that, compared with the intermediate latent space W , it is much more difﬁcult to perform GAN inversion to the original latent space Z to obtain the latent vector z of a real natural image.
Revised: However, one issue of this solution is that, compared with the intermediate latent space W , it is much more difﬁcult to perform GAN inversion to the original latent space Z to obtain the latent vector z of a real natural image.

Original: This would impede this method to be applied to real natural images other than GAN samples.
Revised: This would impede this method to be applied to real natural images other than GAN samples.

Original: To address this issue, in this work we propose a new regularization strategy for the latent offset ∆ w i , which is to restrict it to be a valid offset created by the mapping network F as follows:
Revised: To address this issue, in this work we propose a new regularization strategy for the latent offset ∆ w i , which is to restrict it to be a valid offset created by the mapping network F as follows:

Original: [Equation]
Revised: [Equation]

Original: which regards ∆ w i as the shift in the W space by adding perturbation around zero in the Z space.
Revised: which regards ∆ w i as the shift in the W space by adding perturbation around zero in the Z space.

Original: Note that the point zero is a natural choice as it is the mean point in the prior distribution N (0 , I ) d .
Revised: Note that the point zero is a natural choice as it is the mean point in the prior distribution N (0 , I ) d .

Original: In order to make the regularization strength adjustable, one can further view the mapping network
Revised: In order to make the regularization strength adjustable, one can further view the mapping network

Original: F as two consecutive parts F = F 1 ◦ F 2 , and regularize ∆ w i with only F 1 as below:
Revised: F as two consecutive parts F = F 1 ◦ F 2 , and regularize ∆ w i with only F 1 as below:

Original: [Equation]
Revised: [Equation]

Original: The depth d of F 1 thus controls the strength of regularization.
Revised: The depth d of F 1 thus controls the strength of regularization.

Original: As shown in Tables 10 11 12, for the
Revised: As shown in Tables 10 11 12, for the

Original: BFM dataset, we use the full mapping network ( i.e. , d = 8 ).
Revised: BFM dataset, we use the full mapping network ( i.e. , d = 8 ).

Original: For other datasets with more complex images, the regularization of a full mapping network is too strong and sometimes lead to sub-optimal results, we thus set d = 2 .
Revised: For other datasets with more complex images, the regularization of a full mapping network is too strong and sometimes lead to sub-optimal results, we thus set d = 2 .

Original: A.2. T RAINING D ETIALS
Revised: A.2. T RAINING D ETIALS

Original: In this section, we provide the network architectures, hyper-parameters, and other details in our experiments.
Revised: In this section, we provide the network architectures, hyper-parameters, and other details in our experiments.

Original: Model Architectures .
Revised: Model Architectures .

Original: For a fair comparison on the BFM dataset, the images used in the photogeometric model are resized to 64 2 resolution, and the D, A, L, V network have the same architecture as Wu et al.
Revised: For a fair comparison on the BFM dataset, the images used in the photogeometric model are resized to 64 2 resolution, and the D, A, L, V network have the same architecture as Wu et al.

Original: For other datasets, the image resolution is resized to 128 2 , and the network architectures are added by additional layers accordingly, as shown in Tab.5 and Tab.6 .
Revised: For other datasets, the image resolution is resized to 128 2 , and the network architectures are added by additional layers accordingly, as shown in Tab.5 and Tab.6 .

Original: For the
Revised: For the

Original: GAN encoder net E , we adopt a ResNet-alike architecture (He et al., 2016), as presented in Tab. and Tab.8 .
Revised: GAN encoder net E , we adopt a ResNet-alike architecture (He et al., 2016), as presented in Tab. and Tab.8 .

Original: The abbreviations for the network layers are described below:
Revised: The abbreviations for the network layers are described below:

Original: Conv( c in , c out , k, s, p ): convolution with c in input channels, c out output channels, kernel size k , stride s , and padding p .
Revised: Conv( c in , c out , k, s, p ): convolution with c in input channels, c out output channels, kernel size k , stride s , and padding p .

Original: Deconv( c in , c out , k, s, p ): deconvolution with c in input channels, c out output channels, kernel size k , stride s
Revised: Deconv( c in , c out , k, s, p ): deconvolution with c in input channels, c out output channels, kernel size k , stride s

Original: [Table]
Revised: [Table]

Original: and padding p .
Revised: and padding p .

Original: GN( n ): group normalization (Wu & He, 2018) with n groups.
Revised: GN( n ): group normalization (Wu & He, 2018) with n groups.

Original: LReLU
Revised: LReLU

Original: [Table]
Revised: [Table]

Original: ): leaky ReLU
Revised: ): leaky ReLU

Original: [Table] [Table]
Revised: [Table] [Table]

Original: ) with a negative slope of α .
Revised: ) with a negative slope of α .

Original: [Table]
Revised: [Table]

Original: s ): nearest-neighbor upsampling with a scale of s .
Revised: s ): nearest-neighbor upsampling with a scale of s .

Original: [Table]
Revised: [Table]

Original: average
Revised: average

Original: [Table]
Revised: [Table]

Original: with a stride of s .
Revised: with a stride of s .

Original: ResBlock( c in , c out ): residual block as deﬁned in Tab.8.
Revised: ResBlock( c in , c out ): residual block as deﬁned in Tab.8.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Hyper-parameters.
Revised: Hyper-parameters.

Original: The hyper-parameters of our experiments are provided in Tables 9 10 11 12.
Revised: The hyper-parameters of our experiments are provided in Tables 9 10 11 12.

Original: The batch size here is the batch size used in the Step 2 and Step 3 of our method.
Revised: The batch size here is the batch size used in the Step 2 and Step 3 of our method.

Original: And the number of stages indicates how many times the Step 1-3 are repeated.
Revised: And the number of stages indicates how many times the Step 1-3 are repeated.

Original: Joint pre-training.
Revised: Joint pre-training.

Original: Note that for the CelebA, Cat, and BFM dataset, we adopt joint pre-training on 200 samples to achieve better generalization when ﬁne-tuning on new samples.
Revised: Note that for the CelebA, Cat, and BFM dataset, we adopt joint pre-training on 200 samples to achieve better generalization when ﬁne-tuning on new samples.

Original: Speciﬁcally, for each iteration of each step, we randomly choose a batch of samples from the 200 samples, and the training is the same as that of a single sample except that the gradient calculated from each sample are averaged after back-propagation.
Revised: Speciﬁcally, for each iteration of each step, we randomly choose a batch of samples from the 200 samples, and the training is the same as that of a single sample except that the gradient calculated from each sample are averaged after back-propagation.

Original: Symmetry assumption.
Revised: Symmetry assumption.

Original: In our method, the symmetry assumption on object shapes is an optional choice, and could further improve the results for human face and cat datasets.
Revised: In our method, the symmetry assumption on object shapes is an optional choice, and could further improve the results for human face and cat datasets.

Original: Thus we also adopt [Table] [Figure] [Table]
Revised: Thus we also adopt this assumption for CelebA, Cat, and BFM dataset unless otherwise stated.

Original: [Figure]
Revised: Note that the results

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: this assumption for CelebA, Cat, and BFM dataset unless otherwise stated.

Original: Note that the results without the symmetry assumption are also provided in Figures 6 12 , Tab.12 and other datasets, which is also satisfactory.
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: Unlike Unsup3d (Wu et al., 2020), our method does not need to predict an additional conﬁdence map for the symmetry, as we can drop the symmetry assumption after the ﬁrst training stage, as shown in Tab.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: without the symmetry assumption are also provided in Figures 6 12 , Tab.12 and other datasets, which is also satisfactory.

Revised: Unlike Unsup3d (Wu et al., 2020), our method does not need to predict an additional conﬁdence map for the symmetry, as we can drop the symmetry assumption after the ﬁrst training stage, as shown in Tab.11 .

Original: Sampling viewpoints and lighting directions.
Revised: Sampling viewpoints and lighting directions.

Original: In the Step 2 of our method, we need to randomly sample a number of viewpoints and lighting directions from some prior distributions.
Revised: In the Step 2 of our method, we need to randomly sample a number of viewpoints and lighting directions from some prior distributions.

Original: For viewpoint, we calculate the mean and covariance statistics of viewpoints estimated from the BFM, CelebA, and Cat dataset using the viewpoint prediction network of (Wu et al., 2020).
Revised: For viewpoint, we calculate the mean and covariance statistics of viewpoints estimated from the BFM, CelebA, and Cat dataset using the viewpoint prediction network of (Wu et al., 2020).

Original: These statistics are used to sample the random viewpoints from multi-variate normal distributions for the corresponding datasets.
Revised: These statistics are used to sample the random viewpoints from multi-variate normal distributions for the corresponding datasets.

Original: For LSUN Car and Church datasets, we simply use the statistics estimated from the FFHQ dataset (Karras et al., 2019), and we found it works well.
Revised: For LSUN Car and Church datasets, we simply use the statistics estimated from the FFHQ dataset (Karras et al., 2019), and we found it works well.

Original: For lighting directions, we sample from the following uniform distributions: l x ∼ U ( − 1 , 1) , l y ∼ 
Revised: For lighting directions, we sample from the following uniform distributions: l x ∼ U ( x min , x max ) , l y ∼ U ( y min , y max ) , k d ∼ U

Revised: ( d min , d max ) , and k s = αk d , where [ x min , x max , y min , y max , d min , d max , α ] =

Original: U ( − 0 .
Revised: [ − 0 .

Original: 8) ,k d ∼ U ( − 0 . 1 , 0 . 6) ,and k s = − 0 .
Revised: 9 , − 0 . 3 , 0 . 8 , − 0 . 1 , 0 . 7 , − 0 .

Original:  6 k d . 
Revised: 4] for human face datasets ( i.e. , BFM and CelebA) and [ − 1 , 1 , − 0 . 2 , 0 . 8 , − 0 . 6 , − 0 . 6] for other datasets.

Original: Note the sampled k d and k s are added to the predictions of the lighting network L as the ﬁnal diffuse and ambient term.
Revised: Note the sampled k d and k s are added to the predictions of the lighting network L as the ﬁnal diffuse and ambient term.

Original: This setting tends to increase the diffuse term and decrease the ambient term, as the diffuse term could better reﬂect the structure of the objects.
Revised: This setting tends to increase the diffuse term and decrease the ambient term, as the diffuse term could better reﬂect the structure of the objects.

Original: Masking out the background.
Revised: Masking out the background.

Original: In this work, we care about the main object rather than the background.
Revised: In this work, we care about the main object rather than the background.

Original: Thus, for GANs trained on the LSUN Car and LSUN Church datasets that contain large background area, we could use an off-the-shelf scene parsing model PSPNet50 (Zhao et al., 2017) to parse the object areas as a pre-processing for better visualization.
Revised: Thus, for GANs trained on the LSUN Car and LSUN Church datasets that contain large background area, we could use an off-the-shelf scene parsing model PSPNet50 (Zhao et al., 2017) to parse the object areas as a pre-processing for better visualization.

Original: For an input image, the scene parsing model is used only once to get the object mask for the original view, thus it does not provide signals on the 3D shape.
Revised: For an input image, the scene parsing model is used only once to get the object mask for the original view, thus it does not provide signals on the 3D shape.

Original: For novel views, the mask is warped with the current shape accordingly, and is used to mask out the image background in Eq.2 and Eq.3 .
Revised: For novel views, the mask is warped with the current shape accordingly, and is used to mask out the image background in Eq.2 and Eq.3 .

Original: For other datasets we do not mask out the background.
Revised: For other datasets we do not mask out the background.

Original: Face rotation evaluation.
Revised: Face rotation evaluation.

Original: In the identity-preserving face rotation experiments, we use a face identity detection model ArcFace (Deng et al., 2019) to evaluate the shift of face identity during rotation.
Revised: In the identity-preserving face rotation experiments, we use a face identity detection model ArcFace (Deng et al., 2019) to evaluate the shift of face identity during rotation.

Original: For each human face sample, we ﬁrst rotate its yaw angle from -20 degree to 20 degrees to obtain 20 samples with different poses.
Revised: For each human face sample, we ﬁrst rotate its yaw angle from -20 degree to 20 degrees to obtain 20 samples with different poses.

Original: The face angle is obtained with the viewpoint prediction network from Wu et al.
Revised: The face angle is obtained with the viewpoint prediction network from Wu et al.

Original: We then calculate the identity distances between the front-view face and other faces, and report both their mean and max values in Tab.11 .
Revised: We then calculate the identity distances between the front-view face and other faces, and report both their mean and max values in Tab.11 .

Original: For face identity prediction model, the identity distances are represented as angles between the ﬁnal feature embeddings.
Revised: For face identity prediction model, the identity distances are represented as angles between the ﬁnal feature embeddings.

Original: The above mean and max values are averaged across 100 samples as the ﬁnal results.
Revised: The above mean and max values are averaged across 100 samples as the ﬁnal results.