Original: 3D GENERATION ON I MAGE N ET
Revised: 3D GENERATION ON I MAGE N ET

Original: Anonymous authors
Revised: Ivan Skorokhodov ∗ KAUST Aliaksandr Siarohin Snap Inc.

Revised: Yinghao Xu ∗

Revised: CUHK

Revised: Jian Ren

Revised: Snap Inc.

Original: Paper under double-blind review
Revised: Hsin-Ying Lee Snap Inc. Peter Wonka KAUST

Revised: Sergey Tulyakov

Revised: Snap Inc.

Original: A BSTRACT
Revised: A BSTRACT

Original: All existing 3D-from-2D generators are designed for well-curated single-category datasets, where all the objects have (approximately) the same scale, 3D location and orientation, and the camera always points to the center of the scene.
Revised: All existing 3D-from-2D generators are designed for well-curated single-category datasets, where all the objects have (approximately) the same scale, 3D location and orientation, and the camera always points to the center of the scene.

Original: This makes them inapplicable to diverse, in-the-wild datasets of non-alignable scenes rendered from arbitrary camera poses.
Revised: This makes them inapplicable to diverse, in-the-wild datasets of non-alignable scenes rendered from arbitrary camera poses.

Original: In this work, we develop 3D generator with
Revised: In this work, we develop 3D generator with

Original: Generic Priors (3DGP) : a 3D synthesis framework with more general assumptions about the training data, and show that it scales to very challenging datasets, like ImageNet.
Revised: Generic Priors (3DGP) : a 3D synthesis framework with more general assumptions about the training data, and show that it scales to very challenging datasets, like ImageNet.

Original: Our model is based on three new ideas.
Revised: Our model is based on three new ideas.

Original: First, we incorporate an inaccurate off-the-shelf depth estimator into 3D GAN training via a special depth adaptation module to handle the imprecision.
Revised: First, we incorporate an inaccurate off-the-shelf depth estimator into 3D GAN training via a special depth adaptation module to handle the imprecision.

Original: Then, we create a flexible camera model and a regularization strategy for it to learn its distribution parameters during training.
Revised: Then, we create a flexible camera model and a regularization strategy for it to learn its distribution parameters during training.

Original: Finally, we extend the recent ideas of transferring knowledge from pretrained classifiers into GANs for patch-wise trained models by employing a simple distillation-based technique on top of the discriminator.
Revised: Finally, we extend the recent ideas of transferring knowledge from pretrained classifiers into GANs for patch-wise trained models by employing a simple distillation-based technique on top of the discriminator.

Original: It achieves more stable training than the analogs and speeds up the convergence by at least 40%.
Revised: It achieves more stable training than the existing methods and speeds up the convergence by at least 40%.

Original: We explore our model on four datasets: SDIP Dogs 256 2 , SDIP Elephants 256 2 , LSUN Horses 2562 , and ImageNet 256 2 and demonstrate that 3DGP outperforms the recent state-of-the-art in terms of both texture and geometry quality.
Revised: We explore our model on four datasets: SDIP Dogs 256 2 , SDIP Elephants 256 2 , LSUN Horses 256 and ImageNet 256 2 and demonstrate that 3DGP outperforms the recent state-of-the-art in terms of both texture and geometry quality.

Original: Visualizations: https://u2wjb9xxz9q.github.io
Revised: Code and visualizations: https://snap-research.github.io/3dgp

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Revised: EG3D models the geometry in low resolution and renders either flat shapes (when trained with the default camera distribution) or repetitive “layered” ones (when trained with a wide camera distribution).

Original: distribution radiance field in the full dataset resolution and learns high-fidelity details during training.
Revised: In contrast, our model synthesizes the radiance field in the full dataset resolution and learns high-fidelity details during training.

Original: Zoom-in for a better view.
Revised: Zoom-in for a better view.

Original: We witness remarkable progress in the domain of 3D-aware image synthesis.
Revised: We witness remarkable progress in the domain of 3D-aware image synthesis.

Original: The community is developing new methods to improve the image quality, 3D consistency and efficiency of the generators(e.g., Chan et al. ; Deng et al. ; Skorokhodov et al. ; Zhao et al.
Revised: The community is developing new methods to improve the image quality, 3D consistency and efficiency of the generators

Original: ; Schwarz et al.

Original: However, all the existing frameworks are designed for well-curated and aligned datasets consisting of objects of the same category, scale and global scene structure, like human or cat faces (Chan et al., 2021).

Original: Such curation requires domain-specific 3D knowledge about the object

Original: [Figure]
Revised: [Figure]

Original:  r e a l / f a k e
Revised: ϕ CameraGenerator r e a l / f a k e

Original: D tri-planes offline dataset preprocessing
Revised: D tri-planes offline dataset preprocessing

Original: ResNet
Revised: ResNet

Original: LeReS r e a l / f a k e D e ˆ e !
Revised: LeReS r e a l / f a k e D e ˆ e !

Original: e − ˆ e !
Revised: e − ˆ e !

Original: knowledgedistillation ϕ !
Revised: knowledgedistillation ϕ !

Original:  category at hand, since one needs to infer the underlying 3D  to properly crop, rotate and scale the images (Deng et al., 2022; Chan et al., 2022).
Revised: (e.g., Chan et et However, all the existing frameworks are designed for well-curated and aligned datasets consisting of objects of the same category, scale and global scene structure, like human or cat faces (Chan et al., 2021). Such curation requires domain-specific 3D knowledge about the object category at hand, since one needs to infer the underlying 3D keypoints to properly crop, rotate and scale the images (Deng et al., 2022; Chan et al., 2022).

Original: This makes it infeasible to perform a similar alignment procedure for large-scale multi-category datasets that are inherently “non-alignable”: there does not exist a single canonical position which all the objects could be transformed into (e.g., it is impossible to align a landscape panorama with a spoon).
Revised: This makes it infeasible to perform a similar alignment procedure for large-scale multi-category datasets that are inherently “non-alignable”: there does not exist a single canonical position which all the objects could be transformed into (e.g., it is impossible to align a landscape panorama with a spoon).

Original: To extend 3D synthesis to in-the-wild datasets, one needs a framework which relies on more universal 3D priors.
Revised: To extend 3D synthesis to in-the-wild datasets, one needs a framework which relies on more universal 3D priors.

Original: In this work, we make a step towards this direction and develop a 3D generator with
Revised: In this work, we make a step towards this direction and develop a 3D generator with

Original: Generic Priors (3DGP): a 3D synthesis model which is guided only by (imperfect) depth predictions from an off-the-shelf monocular depth estimator.
Revised: Generic Priors (3DGP): a 3D synthesis model which is guided only by (imperfect) depth predictions from an off-the-shelf monocular depth estimator.

Original: Surprisingly, such 3D cues are enough to learn reasonable scenes from loosely curated, non-aligned datasets, such as ImageNet (Deng et al., 2009).
Revised: Surprisingly, such 3D cues are enough to learn reasonable scenes from loosely curated, non-aligned datasets, such as ImageNet (Deng et al., 2009).

Original: Training a 3D generator on in-the-wild datasets comes with three main challenges: 1) extrinsic camera parameters of real images are unknown and impossible to infer; 2) objects appear in different shapes, positions, rotations and scales, hindering learning of the underlying3D geometry; and 3) the dataset typically contains a lot of variation in terms of texture and structure, and is difficult to fit even for 2D generators.
Revised: Training a 3D generator on in-the-wild datasets comes with three main challenges: 1) extrinsic camera parameters of real images are unknown and impossible to infer; 2) objects appear in different shapes, positions, rotations and scales, complicating the learning of the underlying geometry; and 3) the dataset typically contains a lot of variation in terms of texture and structure, and is difficult to fit even for 2D generators.

Original: As shown in Fig 1 (left), state-of-the-art 3D-aware generators, such as
Revised: As shown in Fig 1 (left), state-of-the-art 3D-aware generators, such as

Original: EG3D (Chan et al., 2022), struggle to learn the proper geometry in such a challenging scenario.
Revised: EG3D (Chan et al., 2022), struggle to learn the proper geometry in such a challenging scenario.

Original: In this work, we develop three novel techniques to address those problems.
Revised: In this work, we develop three novel techniques to address those problems.

Original: Learnable “Ball-in-Sphere” camera distribution .
Revised: Learnable “Ball-in-Sphere” camera distribution .

Original: Most existing methods utilize a restricted camera model (e.g., (Schwarz et al., 2020; Niemeyer & Geiger, 2021b;
Revised: Most existing methods utilize a restricted camera model (e.g., (Schwarz et al., 2020; Niemeyer & Geiger, 2021b;

Original: Chan et al., 2021)): the camera is positioned on a sphere with a constant radius, always points to the world center and has fixed intrinsics.
Revised: Chan et al., 2021)): the camera is positioned on a sphere with a constant radius, always points to the world center and has fixed intrinsics.

Original: But diverse, non-aligned datasets violate these assumptions: e.g., dogs datasets have images of both close-up photos of a snout and photos of full-body dogs, which implies the variability in the focal length and look-at positions.
Revised: But diverse, non-aligned datasets violate these assumptions: e.g., dogs datasets have images of both close-up photos of a snout and photos of full-body dogs, which implies the variability in the focal length and look-at positions.

Original: Thus, we introduce a novel camera model with 6 degrees of freedom to address this variability.
Revised: Thus, we introduce a novel camera model with 6 degrees of freedom to address this variability.

Original: We optimize its distribution parameters during training and develop an efficient gradient penalty for it to prevent its collapse to  delta distribution.
Revised: We optimize its distribution parameters during training and develop an efficient gradient penalty for it to prevent its collapse to a delta distribution.

Original: Adversarial depth supervision (ADS).
Revised: Adversarial depth supervision (ADS).

Original: A generic image dataset features a wide diversity of objects with different shapes and poses.
Revised: A generic image dataset features a wide diversity of objects with different shapes and poses.

Original: That is why learning a meaningful 3D geometry together with the camera distribution is an ill-posed problem, as the incorrect scale can be well compensated by an incorrect camera model (Hartley & Zisserman, 2003), or flat geometry (Zhao et al., 2022; Chan et al., 2022).
Revised: That is why learning a meaningful 3D geometry together with the camera distribution is an ill-posed problem, as the incorrect scale can be well compensated by an incorrect camera model (Hartley & Zisserman, 2003), or flat geometry (Zhao et al., 2022; Chan et al., 2022).

Original: To instill the 3D bias, we provide the scene geometry information to the discriminator by concatenating the depth map of a scene as the 4-th channel of its RGB input.
Revised: To instill the 3D bias, we provide the scene geometry information to the discriminator by concatenating the depth map of a scene as the 4-th channel of its RGB input.

Original: For real images, we use their ( imperfect ) estimates from a generic off-the-shelf monocular depth predictor (Miangoleh et al., 2021).
Revised: For real images, we use their ( imperfect ) estimates from a generic off-the-shelf monocular depth predictor (Miangoleh et al., 2021).

Original: For fake images, we render the depth from the synthesized radiance field, and process it with a shallow depth adaptor module, bridging the distribution gap between the estimated and rendered depth maps.
Revised: For fake images, we render the depth from the synthesized radiance field, and process it with a shallow depth adaptor module, bridging the distribution gap between the estimated and rendered depth maps.

Original: This ultimately guides the generator to learn the proper 3D geometry.
Revised: This ultimately guides the generator to learn the proper 3D geometry.

Original: Knowledge distillation into Discriminator.
Revised: Knowledge distillation into Discriminator.

Original: Prior works have shown benefits from transferring the knowledge from conventional 2D image encoders into a synthesis model (Sauer et al., 2022).
Revised: Prior works found it beneficial to transfer the knowledge from off-the-shelf 2D image encoders into a synthesis model (Sauer et al., 2022).

Original: The state-of-the-art techniques utilize pretrained image classifiers as the discriminator backbone and develop regularization strategies on topof them (Sauer et al., 2021; Kumari et al., 2022).
Revised: They typically utilize pre-trained image classifiers as the discriminator backbone with additional regularization strategies on top (Sauer et al., 2021; Kumari et al., 2022).

Original: These techniques, however, are only applicable when the discriminator has a similar input distribution compared to what the encoder was trained on.
Revised: Such techniques, however, are only applicable when the discriminator has an input distribution similar to what the encoder was trained on.

Original: This makes it difficult to use efficient patch-based discriminators (Skorokhodov et al., 2022) or passing depth maps as the 4-th channel to the discriminator.
Revised: This does not suit the setup of patch-wise training (Schwarz et al., 2020) or allows to provide depth maps via the 4-th channel to the discriminator.

Original: That is why, we develop a more general and efficient knowledge transfer strategy based on knowledge distillation.
Revised: That is why we develop a more general and efficient knowledge transfer strategy based on knowledge distillation.

Original: It consists in forcing the discriminator to predict features of a pre-trained ResNet50 (He et al., 2016) model, effectively transferring the knowledge into our model.
Revised: It consists in forcing the discriminator to predict features of a pre-trained ResNet50 (He et al., 2016) model, effectively transferring the knowledge into our model.

Original: This technique has just 1% of computational overhead compared to standard training, but allows to improve FID for both 2D and 3D generators by at least 40%.
Revised: This technique has just 1% of computational overhead compared to standard training, but allows to improve FID for both 2D and 3D generators by at least 40%.

Original: First, we explore our ideas on non-aligned single-category image datasets:
Revised: First, we explore our ideas on non-aligned single-category image datasets:

Original: SDIP
Revised: SDIP

Original: Dogs 256 2 (Mokady et al., 2022), SDIP Elephants 256 2 (Mokady et al., 2022), and LSUN
Revised: Dogs 256 2 (Mokady et al., 2022), SDIP Elephants 256 2 (Mokady et al., 2022), and LSUN

Original: Horses 256 2 (Yu et al., 2015).
Revised: Horses 256 2 (Yu et al., 2015).

Original: On these datasets, our generator achieves better image appearance (measured by FID Heusel et al.
Revised: On these datasets, our generator achieves better image appearance (measured by FID Heusel et al.

Original: and geometry quality than the modern state-of-the-art 3D-aware generators.
Revised: and geometry quality than the modern state-of-the-art 3D-aware generators.

Original: Then, we train the model on all the 1 , 000 classes of ImageNet (Deng et al., 2009), showing that multi-categorical 3D synthesis is possible for non-alignable data (see Fig. 1).
Revised: Then, we train the model on all the 1 , 000 classes of ImageNet (Deng et al., 2009), showing that multi-categorical 3D synthesis is possible for non-alignable data (see Fig. 1).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: 3D-aware image synthesis .
Revised: 3D-aware image synthesis .

Original: Mildenhall et al.
Revised: Mildenhall et al.

Original: (2020) introduced Neural Radiance Fields (NeRF): a neural network-based representation of 3D volumes which is learnable from RGB supervision only.
Revised: (2020) introduced Neural Radiance Fields (NeRF): a neural network-based representation of 3D volumes which is learnable from RGB supervision only.

Original: It ignited many 3D-aware image/video generators (Schwarz et al., 2020; Niemeyer & Geiger, 2021b;
Revised: It ignited many 3D-aware image/video generators (Schwarz et al., 2020; Niemeyer & Geiger, 2021b;

Original: Chan et al., 2021; Xue et al., 2022;
Revised: Chan et al., 2021; Xue et al., 2022;

Original: Zhou et al., 2021; Wang et al., 2022; Gu et al., 2022; Or-El et al., 2021; Chan et al., 2022; Skorokhodov et al., 2022; Zhang et al., 2022; Xu et al., 2021; Bahmani et al., 2022), all of them being GAN-based (Goodfellow et al., 2014).
Revised: Zhou et al., 2021; Wang et al., 2022; Gu et al., 2022; Or-El et al., 2021; Chan et al., 2022; Skorokhodov et al., 2022; Zhang et al., 2022; Xu et al., 2021; Bahmani et al., 2022), all of them being GAN-based (Goodfellow et al., 2014).

Original: Many of them explore the techniques to reduce the cost of 3D-aware generation for high-resolution data, like patch-wise training (e.g.,
Revised: Many of them explore the techniques to reduce the cost of 3D-aware generation for high-resolution data, like patch-wise training (e.g.,

Original: Schwarz et al.
Revised: Schwarz et al.

Original: Meng et al.
Revised: Meng et al.

Original: ; Skorokhodov et al.
Revised: ; Skorokhodov et al.

Original: , MPI-based rendering (e.g.,
Revised: , MPI-based rendering (e.g.,

Original: Zhao et al.
Revised: Zhao et al.

Original: or training a separate 2D upsampler (e.g., Gu et al.
Revised: or training a separate 2D upsampler (e.g., Gu et al.

Original: Learning the camera poses .
Revised: Learning the camera poses .

Original: NeRFs require known camera poses, obtained from multi-view stereo (Sch ¨ onberger et al., 2016) or structure from motion (Sch ¨ onberger & Frahm, 2016).
Revised: NeRFs require known camera poses, obtained from multi-view stereo (Sch ¨ onberger et al., 2016) or structure from motion (Sch ¨ onberger & Frahm, 2016).

Original: Alternatively, a group of works have been introduced to either automatically estimate  camera poses (Wang et al., 2021) or finetune them during training (Lin et al., 2021; Kuang et al., 2022).
Revised: Alternatively, a group of works has been introduced to either automatically estimate the camera poses (Wang et al., 2021) or finetune them during training (Lin et al., 2021; Kuang et al., 2022).

Original: The problem we are tackling in this work is fundamentally different as it requires to learn not the camera poses from multi-view observations, but a distribution of poses, while having access to sparse, single-view data of diverse object categories.
Revised: The problem we are tackling in this work is fundamentally different as it requires learning not the camera poses from multi-view observations, but a distribution of poses, while having access to sparse, single-view data of diverse object categories.

Original: In this respect, the work closest to ours is CAMPARI (Niemeyer &
Revised: In this respect, the work closest to ours is CAMPARI (Niemeyer &

Original: Geiger, 2021a) as it also learns a camera distribution.
Revised: Geiger, 2021a) as it also learns a camera distribution.

Original: Beyond this the similarity ends, as our “Ballin-Sphere” camera distribution allows for more flexibility and supports multiple object categories.

Original: GANs with external knowledge.
Revised: GANs with external knowledge.

Original: Several works observed improved convergence and fidelity of
Revised: Several works observed improved convergence and fidelity of

Original: GANs when using existing, generic image-based models (Kumari et al., 2022; Sauer et al., 2021; 2022; Mo et al., 2020), the most notable being StyleGAN-XL (Sauer et al., 2022), which uses a pre-trained EfficientNet (Tan & Le, 2019) followed by a couple of discriminator layers.
Revised: GANs when using existing, generic image-based models (Kumari et al., 2022; Sauer et al., 2021; 2022; Mo et al., 2020), the most notable being StyleGAN-XL (Sauer et al., 2022), which uses a pre-trained EfficientNet (Tan & Le, 2019) followed by a couple of discriminator layers.

Original: A similar technique is not suitable in our case as pre-training a generic RGB-D network on a large scale RGBD dataset is problematic due to the lack of data.
Revised: A similar technique is not suitable in our case as pre-training a generic RGB-D network on a large-scale RGB-D dataset is problematic due to the lack of data.

Original: Another notable example is FreezeD Mo et al.
Revised: Another notable example is FreezeD Mo et al.

Original: (2020), which proposes to distill discriminator features for GAN finetuning.
Revised: (2020), which proposes to distill discriminator features for GAN finetuning.

Original: Our work, on the other hand, rely on  existing model for image classification.
Revised: Our work, on the other hand, relies on an existing model for image classification.

Original: Off-the-shelf depth guidance .
Revised: Off-the-shelf depth guidance .

Original: GSN (DeVries et al., 2021) also concatenates depth maps as the 4-th channel of the discriminator’s input, but they utilize ground truth depths, which are not available for large-scale datasets.
Revised: GSN (DeVries et al., 2021) also concatenates depth maps as the 4-th channel of the discriminator’s input, but they utilize ground truth depths, which are not available for large-scale datasets.

Original: DepthGAN (Shi et al., 2022) uses predictions from a depth estimator to guide  training of a 2D GAN.
Revised: DepthGAN (Shi et al., 2022) uses predictions from a depth estimator to guide the training of a 2D GAN.

Original: Exploiting monocular depth estimators for improving neural rendering was also explored in concurrent work (Yu et al., 2022), however their goal is just geometry reconstruction.
Revised: Exploiting monocular depth estimators for improving neural rendering was also explored in concurrent work (Yu et al., 2022), however, their goal is just geometry reconstruction.

Original: The core characteristic of our approach is taking the depth estimator imprecision into account by training a depth adaptor module to mitigate it (see §3.2).
Revised: The core characteristic of our approach is taking the depth estimator imprecision into account by training a depth adaptor module to mitigate it (see §3.2).

Original:  M ETHOD
Revised:  M ETHOD

Original: We build our generator on top of EpiGRAF (Skorokhodov et al., 2022) since its fast to train, achieves reasonable image quality and does not need a 2D upsampler, relying on patch-wise training instead (Schwarz et al., 2020).
Revised: We build our generator on top of EpiGRAF (Skorokhodov et al., 2022) since its fast to train, achieves reasonable image quality and does not need a 2D upsampler, relying on patch-wise training in- stead (Schwarz et al., 2020).

Original: Given a random latent code z , our generator G produces a tri-plane representation for the scene.
Revised: Given a random latent code z , our generator G produces a tri-plane representation for the scene.

Original: Then, a shallow 2-layer MLP predicts RGB color and density σ values from an interpolated feature vector at a 3D coordinate.
Revised: Then, a shallow 2-layer MLP predicts RGB color and density σ values from an interpolated feature vector at a 3D coordinate.

Original: Then images and depths are volumetrically rendered (Mildenhall et al., 2020) at any given c¸amera position.
Revised: Then, images and depths are volumetrically rendered (Mildenhall et al., 2020) at any given camera position.

Original: Differently to prior works (Chan et al., 2021; Niemeyer & Geiger, 2021b) that utilize fixed camera distribution, we sample camera from a trainable camera generator C (see §3.1).
Revised: Differently to prior works (Chan et al., 2021; Niemeyer & Geiger, 2021b) that utilize fixed camera distribution, we sample camera from a trainable camera generator C (see §3.1).

Original: We render depth and process it via the depth adaptor (see §3.2), bridging the domains of rendered and estimated depth maps (Miangoleh et al., 2021).
Revised: We render depth and process it via the depth adaptor (see §3.2), bridging the domains of rendered and estimated depth maps (Miangoleh et al., 2021).

Original: Our discriminator D follows the architecture of StyleGAN2 (Karras et al., 2020a), additionally taking either adapted or estimated depth as the 4-th channel.
Revised: Our discriminator D follows the architecture of StyleGAN2 (Karras et al., 2020a), additionally taking either adapted or estimated depth as the 4-th channel.

Original: To further improve the image fidelity, we propose a simple knowledge distillation technique, that enriches D with external knowledge obtained from ResNet (He et al., 2016)
Revised: To further improve the image fidelity, we propose a simple knowledge distillation technique, that enriches D with external knowledge obtained from ResNet (He et al., 2016)

Original: (see §3.3).
Revised: (see §3.3).

Original: The overall model architecture is shown in Fig.
Revised: The overall model architecture is shown in Fig.

Original: 3. L EARNABLE “B ALL - IN -S PHERE ” CAMERA DISTRIBUTION
Revised: 3. L EARNABLE “B ALL - IN -S PHERE ” CAMERA DISTRIBUTION

Original: Limitations of Existing Camera Parameterization.
Revised: Limitations of Existing Camera Parameterization.

Original: The camera parameterization of existing 3D generators follows an overly simplified distribution — its position is sampled on a fixed-radius sphere with fixed intrinsics, and the camera always points to (0 , 0 , 0) .
Revised: The camera parameterization of existing 3D generators follows an overly simplified distribution — its position is sampled on a fixed-radius sphere with fixed intrinsics, and the camera always points to (0 , 0 , 0) .

Original: This parametrization has only two degrees of freedom: pitch and yaw ( φ pos in Fig.
Revised: This parametrization has only two degrees of freedom: pitch and yaw ( φ pos in Fig.

Original: 3 (a)), implicitly assuming that all the objects could be centered, rotated and scaled with respect to some canonical alignment.
Revised: 3 (a)), implicitly assuming that all the objects could be centered, rotated and scaled with respect to some canonical alignment.

Original: However, natural in-the-wild 3D scenes are inherently non-alignable: they could consist of multiple objects, objects might have drastically different shapes and articulation, or they could even be represented only as volumes (like smoke).
Revised: However, natural in-the-wild 3D scenes are inherently non-alignable: they could consist of multiple objects, objects might have drastically different shapes and articulation, or they could even be represented only as volumes (like smoke).

Original: This makes the traditional camera conventions ill-posed for such data.
Revised: This makes the traditional camera conventions ill-posed for such data.

Original: Learnable “Ball-in-Sphere” Camera Distribution.
Revised: Learnable “Ball-in-Sphere” Camera Distribution.

Original: We introduce a new camera parametrization which we call “Ball-in-Sphere”.
Revised: We introduce a new camera parametrization which we call “Ball-in-Sphere”.

Original: Contrary to the standard parametrization, it has four additional degrees of freedom: the field of view φ fov , and pitch, yaw and radius of the inner sphere, specifying the look-at point within the outer sphere ( φ lookat in Fig.
Revised: Contrary to the standard one, it has four additional degrees of freedom: the field of view φ fov , and pitch, yaw, and radius of the inner sphere, specifying the look-at point within the outer sphere ( φ lookat in Fig.

Original: 3 (b)).
Revised: 3 (b)).

Original: Combining with the standard parameters on the outer sphere, our camera parametrization has six degrees of freedom φ =
Revised: Combining with the standard parameters on the outer sphere, our camera parametrization has six degrees of freedom φ =

Original: [ φ pos ∥ φ fov ∥
Revised: [ φ pos ∥ φ fov ∥

Original: φ lookat ] , where ∥ denotes concatenation.
Revised: φ lookat ] , where ∥ denotes concatenation.

Original: Note, that compared with the standard camera, ours allows us to learn the scale of objects and scenes, enabling zooming into different parts of the scene as shown in Fig.

Original: 3 (b).

Original: Instead of manually defining camera distributions, we learn the camera distribution during training for each dataset.
Revised: Instead of manually defining camera distributions, we learn the camera distribution during training for each dataset.

Original: In particular, we train the camera generator network C that takes camera parameters sampled from a sufficiently wide camera prior φ ′ and produces new camera parameters φ .
Revised: In particular, we train the camera generator network C that takes camera parameters sampled from a sufficiently wide camera prior φ ′ and produces new camera parameters φ .

Original: For a class conditional dataset, such as ImageNet where scenes have significantly different geometry, we additionally condition this network on the class label c and the latent code z , i.e. φ = C ( φ ′ , z , c ) .
Revised: For a class conditional dataset, such as ImageNet where scenes have significantly different geometry, we additionally condition this network on the class label c and the latent code z , i.e. φ = C ( φ ′ , z , c ) .

Original: For a single category dataset we use φ = C
Revised: For a single category dataset we use φ = C

Original: ( φ ′ , z ) .
Revised: ( φ ′ , z ) .

Original: [Figure]
Revised: [Figure]

Original: Camera Gradient Penalty.
Revised: Camera Gradient Penalty.

Original: To the best of our knowledge, CAMPARI (Niemeyer & Geiger, 2021a) is the only work which also learns the camera distribution.
Revised: To the best of our knowledge, CAMPARI (Niemeyer & Geiger, 2021a) is the only work which also learns the camera distribution.

Original: It samples a set of camera parameters from a wide distribution and passes it to a shallow neural network, which produces a residual ∆ φ = φ − φ ′ for these parameters.
Revised: It samples a set of camera parameters from a wide distribution and passes it to a shallow neural network, which produces a residual ∆ φ = φ − φ ′ for these parameters.

Original: However, we observed that such a regularization is too weak for the complex datasets we explore in this work, and leads to the distribution collapse (see Fig. 7).
Revised: However, we observed that such a regularization is too weak for the complex datasets we explore in this work, and leads to a distribution collapse (see Fig. 7).

Original: Note, that a similar observation was also made by Gu et al.
Revised: Note, that a similar observation was also made by Gu et al.

Original: To prevent the camera generator from producing collapsed camera parameters, we seek a new regularization strategy.
Revised: To prevent the camera generator from producing collapsed camera parameters, we seek a new regularization strategy.

Original: Ideally, it should prevent constant solutions, while at the same time, reducing the Lipschitz constant for C , which is shown to be crucial for stable training of generators (Odena et al., 2018).
Revised: Ideally, it should prevent constant solutions, while at the same time, reducing the Lipschitz constant for C , which is shown to be crucial for stable training of generators (Odena et al., 2018).

Original: We can achieve both if we push the derivatives of the predicted camera parameters with respect to the prior camera parameters to either one or minus one, n o r m a li z e
Revised: We can achieve both if we push the derivatives of the predicted camera parameters with respect to the prior camera parameters to either one or minus one, n o r m a li z e

Original: C o n v 2 d C o n v 2 d
Revised: C o n v 2 d C o n v 2 d

Original: C o n v 2 d
Revised: C o n v 2 d

Original: Conv2d
Revised: Conv2d

Original: Select random
Revised: Select random

Original: RGB
Revised: RGB

Original: Real image
Revised: Real image

Original: Estimated depth from LeReS d ¯ d d a d (1) d (2) d (3) d ¯ d d (1) d (2) d (3) left eye
Revised: Estimated depth from LeReS d ¯ d d a d (1) d (2) d (3) d ¯ d d (1) d (2) d (3) left eye

Original: [Figure]
Revised: [Figure]

Original: tail arriving at the following regularization term:
Revised: arriving at the following regularization term:

Original: [Equation]
Revised: [Equation]

Original: where φ ′ i ∈
Revised: where φ ′ i ∈

Original: φ ′ is the camera sampled from the prior distribution and φ i ∈ φ is produced by the camera generator.
Revised: φ ′ is the camera sampled from the prior distribution and φ i ∈ φ is produced by the camera generator.

Original: We refer to this loss as Camera Gradient Penalty.
Revised: We refer to this loss as Camera Gradient Penalty.

Original: Its first part prevents rapid camera changes, facilitating stable optimization, while the second part avoids collapsed posteriors.
Revised: Its first part prevents rapid camera changes, facilitating stable optimization, while the second part avoids collapsed posteriors.

Original: 3. A DVERSARIAL DEPTH SUPERVISION
Revised: 3. A DVERSARIAL DEPTH SUPERVISION

Original: To instill a 3D bias into our model, we develop a strategy of using depth maps predicted by an off-the-shelf estimator E (Miangoleh et al., 2021), for its advantages of being generic and readily applicable for many object categories.
Revised: To instill a 3D bias into our model, we develop a strategy of using depth maps predicted by an off-the-shelf estimator E (Miangoleh et al., 2021), for its advantages of being generic and readily applicable for many object categories.

Original: The main idea is concatenating a depth map as the 4-th channel of the RGB as the input of the discriminator.
Revised: The main idea is concatenating a depth map as the 4-th channel of the RGB as the input of the discriminator.

Original: The fake depth maps in this case is obtained with the help of neural rendering, while the real depth maps are estimated using monocular depth estimator
Revised: The fake depth maps in this case is obtained with the help of neural rendering, while the real depth maps are estimated using monocular depth estimator

Original: E .
Revised: E .

Original: However, naively utilizing the depth from E leads to training divergence.
Revised: However, naively utilizing the depth from E leads to training divergence.

Original: This happens because
Revised: This happens because

Original: E could only produce relative depth, not metric depth.
Revised: E could only produce relative depth, not metric depth.

Original: Moreover, monocular depth estimators are still not perfect, they produce noisy artifacts, ignore high-frequency details, and make prediction mistakes.
Revised: Moreover, monocular depth estimators are still not perfect, they produce noisy artifacts, ignore high-frequency details, and make prediction mistakes.

Original: Thus, we devise a mechanism that allows utilization of the imperfect depth maps.
Revised: Thus, we devise a mechanism that allows utilization of the imperfect depth maps.

Original: The central part of this mechanism is a learnable depth adaptor A , that should transform and augment the depth map obtained with neural rendering to look like a depth map from E .
Revised: The central part of this mechanism is a learnable depth adaptor A , that should transform and augment the depth map obtained with neural rendering to look like a depth map from E .

Original: More specifically, we first render raw depths d from NeRF via volumetric rendering:
Revised: More specifically, we first render raw depths d from NeRF via volumetric rendering:

Original: [Equation]
Revised: [Equation]

Original: where t n , t f ∈ R are near/far planes, T ( t ) is accumulated transmittance, and r ( t ) is a ray.
Revised: where t n , t f ∈ R are near/far planes, T ( t ) is accumulated transmittance, and r ( t ) is a ray.

Original: Raw depth is shifted and scaled from the range of [ t n , t f ] into [ − 1 , 1] to obtain normalized depth ¯ d :
Revised: Raw depth is shifted and scaled from the range of [ t n , t f ] into [ − 1 , 1] to obtain normalized depth ¯ d :

Original: [Equation]
Revised: [Equation]

Original: where b ∈ [0 , ( t n + t f ) / 2] is an additional learnable shift needed to account for the empty space in the front of the camera.
Revised: where b ∈ [0 , ( t n + t f ) / 2] is an additional learnable shift needed to account for the empty space in the front of the camera.

Original: Real depths are normalized into the [ − 1 , 1] range directly.
Revised: Real depths are normalized into the [ − 1 , 1] range directly.

Original: Learnable Depth Adaptor.
Revised: Learnable Depth Adaptor.

Original: While ¯ d has the same range as d r , it is not suitable for adversarial supervision directly due it the imprecision of E : G would be trained to simulate all its prediction artifacts.
Revised: While ¯ d has the same range as d r , it is not suitable for adversarial supervision directly due it the imprecision of E : G would be trained to simulate all its prediction artifacts.

Original: To overcome this issue, we introduce a depth adaptor A to adapt the depth map d a =
Revised: To overcome this issue, we introduce a depth adaptor A to adapt the depth map d a =

Original: A ( ¯ d ) ∈ R h × w , where h × w is a number of sampled pixels.
Revised: A ( ¯ d ) ∈ R h × w , where h × w is a number of sampled pixels.

Original: This depth (fake d a or real d r ) is concatenated with the RGB input and passed to D .
Revised: This depth (fake d a or real d r ) is concatenated with the RGB input and passed to D .

Original: The depth adaptor A models artifacts produced by E , so that the discriminator should focus only on the relevant high level geometry.
Revised: The depth adaptor A models artifacts produced by E , so that the discriminator should focus only on the relevant high level geometry.

Original: However, a too powerful A would be able to fake the depth completely, and G will not learn the geometry.
Revised: However, a too powerful A would be able to fake the depth completely, and G will not learn the geometry.

Original: This is why we structure A as just a 3-layer convolutional network (see Fig. 4).
Revised: This is why we structure A as just a 3-layer convolutional network (see Fig. 4).

Original: Each layer produces a separated depth map with different levels
Revised: Each layer produces a separated depth map with different levels

Original: [Figure]
Revised: [Figure]

Revised: I

Revised: [Figure]

Original: S
Revised: S

Original: [Figure]
Revised: [Figure]

Original: I P
Revised: I P

Original: [Figure]
Revised: [Figure]

Original: l
Revised: l

Original: [Figure]
Revised: [Figure]

Original: a n t
Revised: a n t

Original: L S U N
Revised: L S U N

Original: H
Revised: H

Original: o r s e s
Revised: o r s e s

Original: [Figure]
Revised: [Figure]

Original: of adaptation
Revised: of adaptation

Original: : d (1) a , d (2) a and d (3) a .
Revised: : d (1) a , d (2) a and d (3) a .

Original: The final adapted depth d a is randomly selected from the set of { ¯ d , d (1) a , d (2) a , d (3) a } .
Revised: The final adapted depth d a is randomly selected from the set of { ¯ d , d (1) a , d (2) a , d (3) a } .

Original: Such design can effectively learn good geometry while alleviating overfitting.
Revised: Such design can effectively learn good geometry while alleviating overfitting.

Original: For example, when D receives the original depth map ¯ d as input, it provides to G a strong signal for learning the geometry.
Revised: For example, when D receives the original depth map ¯ d as input, it provides to G a strong signal for learning the geometry.

Original: And passing an adapted depth map d ( i ) a to D allows G to simulate the imprecision artifacts of the depth estimator without degrading its original depth map ¯ d .
Revised: And passing an adapted depth map d ( i ) a to D allows G to simulate the imprecision artifacts of the depth estimator without degrading its original depth map ¯ d .

Original: 3. K NOWLEDGE DISTILLATION FOR D ISCRIMINATOR
Revised: 3. K NOWLEDGE DISTILLATION FOR D ISCRIMINATOR

Original: Knowledge from pretrained classification networks was shown to improve training stability and generation quality in 2D GANs (Sauer et al., 2021; Kumari et al., 2022; Sauer et al., 2022; Casanova et al., 2021).
Revised: Knowledge from pretrained classification networks was shown to improve training stability and generation quality in 2D GANs (Sauer et al., 2021; Kumari et al., 2022; Sauer et al., 2022; Casanova et al., 2021).

Original: A popular solution proposed by Sauer et al.
Revised: A popular solution proposed by Sauer et al.

Original: (2021; 2022) is to use an off-the-shelf model as a discriminator while freezing most of its weights.
Revised: (2021; 2022) is to use an off-the-shelf model as a discriminator while freezing most of its weights.

Original: Unfortunately, this technique is not applicable in our scenario since we modify the architecture of the discriminator by adding an additional depth input (see §3.2) and condition on the parameters of the patch similarly to EpiGRAF (Skorokhodov et al., 2022).
Revised: Unfortunately, this technique is not applicable in our scenario since we modify the architecture of the discriminator by adding an additional depth input (see §3.2) and condition on the parameters of the patch similarly to EpiGRAF (Skorokhodov et al., 2022).

Original: Thus, we devise an alternative technique that can work with arbitrary architectures of the discriminator.
Revised: Thus, we devise an alternative technique that can work with arbitrary architectures of the discriminator.

Original: Specifically, for each real sample, we obtain two feature representations: e from the pretrained ResNet (He et al., 2016) network and ˆ e extracted from the final representation of our discriminator D .
Revised: Specifically, for each real sample, we obtain two feature representations: e from the pretrained ResNet (He et al., 2016) network and ˆ e extracted from the final representation of our discriminator D .

Original: Our loss simply pushes ˆ e to e as follows:
Revised: Our loss simply pushes ˆ e to e as follows:

Original: [Equation]
Revised: [Equation]

Original: L dist can effectively distill knowledge from the pretrained ResNet (He et al., 2016) into our D .
Revised: L dist can effectively distill knowledge from the pretrained ResNet (He et al., 2016) into our D .

Original: 3. T RAINING
Revised: 3. T RAINING

Original: The overall loss for generator G consists of two parts: adversarial loss and Camera Gradient Penalty:
Revised: The overall loss for generator G consists of two parts: adversarial loss and Camera Gradient Penalty:

Original: [Equation]
Revised: [Equation]

Original: where L adv is the non-saturating loss (Goodfellow et al., 2014).
Revised: where L adv is the non-saturating loss (Goodfellow et al., 2014).

Original: We observe that a diverse distribution for camera origin is most important for leaning meaningful geometry, but it is also most prone to degrade to a constant solution.
Revised: We observe that a diverse distribution for camera origin is most important for leaning meaningful geometry, but it is also most prone to degrade to a constant solution.

Original: Therefore, we set λ φ i = 0 .
Revised: Therefore, we set λ φ i = 0 .

Original: 3 for φ pos , while set λ φ i = 0 .
Revised: 3 for φ pos , while set λ φ i = 0 .

Original: 03 for φ fov and λ φ i = 0 .
Revised: 03 for φ fov and λ φ i = 0 .

Original: 003 for φ lookat .
Revised: 003 for φ lookat .

Original: The loss for discriminator D , on the other hand, consists of three parts: adversarial loss, knowledge distillation, and R 1 gradient penalty (Mescheder et al., 2018): 
Revised: The loss for discriminator D , on the other hand, consists of three parts: adversarial loss, knowledge distillation, and R 1 gradient penalty (Mescheder et al., 2018): [Equation]

Original: [Equation]
Revised: [Equation]

Original: We use the same optimizer and hyper-parameters as EpiGRAF.
Revised: We use the same optimizer and hyper-parameters as EpiGRAF.

Original: We observe that for depth adaptor, sampling adapted depth maps with equal probability is not always beneficial, and found that using
Revised: We observe that for depth adaptor, sampling adapted depth maps with equal probability is not always beneficial, and found that using

Original: P ( ¯ d ) = 0 .
Revised: P ( ¯ d ) = 0 .

Original: 5 leads to better geometry.
Revised: 5 leads to better geometry.

Original: For additional details, see Appx B.
Revised: For additional details, see Appx B.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: Datasets.
Revised: Datasets.

Original: In our experiments, we use 4 non-aligned datasets: SDIP Dogs (Mokady et al., 2022),
Revised: In our experiments, we use 4 non-aligned datasets: SDIP Dogs (Mokady et al., 2022),

Original: SDIP Elephants (Mokady et al., 2022), LSUN Horses (Yu et al., 2015), and ImageNet (Deng et al., 2009).
Revised: SDIP Elephants (Mokady et al., 2022), LSUN Horses (Yu et al., 2015), and ImageNet (Deng et al., 2009).

Original: The first three are single-category datasets that contain objects with complex articulated geometry, making them challenging for standard 3D generators.
Revised: The first three are single-category datasets that contain objects with complex articulated geometry, making them challenging for standard 3D generators.

Original: We found it useful to remove outlier images from SDIP Dogs and LSUN Horsesvia instance selection (DeVries et al., 2020), reducing their size to 40 K samples each.
Revised: We found it useful to remove outlier images from SDIP Dogs and LSUN Horsesvia instance selection (DeVries et al., 2020), reducing their size to 40 K samples each.

Original: We refer to the filtered versions of these datasets as SDIP Dogs 40k , and LSUN Horses 40k , respectively.
Revised: We refer to the filtered versions of these datasets as SDIP Dogs 40k , and LSUN Horses 40k , respectively.

Original: We then validate our method on ImageNet, a real-world, multicategory dataset containing 1 , 000 diverse object classes, with more than 1 , 000 images per category.
Revised: We then validate our method on ImageNet, a real-world, multicategory dataset containing 1 , 000 diverse object classes, with more than 1 , 000 images per category.

Original: All the 3D generators (including the baselines) use the same filtering strategy for ImageNet, where 2/3 of its images are filtered out.
Revised: All the 3D generators (including the baselines) use the same filtering strategy for ImageNet, where 2/3 of its images are filtered out.

Original: Note that all the metrics are always measured on the full ImageNet.
Revised: Note that all the metrics are always measured on the full ImageNet.

Original: [Figure]
Revised: [Figure]

Original: Evaluation.
Revised: Evaluation.

Original: We rely on FID (Heusel et al., 2017) to measure the image quality and FID 2k , which is computed on 2,048 images instead of 50k (as for FID) for efficiency.
Revised: We rely on FID (Heusel et al., 2017) to measure the image quality and FID 2k , which is computed on 2,048 images instead of 50k (as for FID) for efficiency.

Original: For ImageNet, we additionally compute Inception Score (IS) (Salimans et al., 2016).
Revised: For ImageNet, we additionally compute Inception Score (IS) (Salimans et al., 2016).

Original: Note that while we train on the filtered ImageNet, we always compute the metrics on the full one.
Revised: Note that while we train on the filtered ImageNet, we always compute the metrics on the full one.

Original: There is no established protocol to evaluate the geometry quality of 3D generators in general, but the state-of-the-art ones are tri-plane (Chan et al., 2022;
Revised: There is no established protocol to evaluate the geometry quality of 3D generators in general, but the state-of-the-art ones are tri-plane (Chan et al., 2022;

Original: Sun et al., 2022) or MPI-based Zhao et al.
Revised: Sun et al., 2022) or MPI-based Zhao et al.

Original: , and we observed that it is possible to quantify their most popular geometry failure case: flatness of the shapes.
Revised: , and we observed that it is possible to quantify their most popular geometry failure case: flatness of the shapes.

Original: For this, we propose Non-Flatness Score (NFS) which is computed as the average entropy of the normalized depth maps histograms.
Revised: For this, we propose Non-Flatness Score (NFS) which is computed as the average entropy of the normalized depth maps histograms.

Original: We depict its intuition in Fig 6 and provide the details in Appx F.
Revised: We depict its intuition in Fig 6 and provide the details in Appx F.

Original: 4. 3D GENERATION FOR SINGLE CATEGORY DATASETS
Revised: 4. 3D GENERATION FOR SINGLE CATEGORY DATASETS

Original: We show quantitative results for single category datasets in Tab.
Revised: We show quantitative results for single category datasets in Tab.

Original: 1a, where we compare with
Revised: 1a, where we compare with

Original: EG3D (Chan et al., 2022) and EpiGRAF (Skorokhodov et al., 2022).
Revised: EG3D (Chan et al., 2022) and EpiGRAF (Skorokhodov et al., 2022).

Original: EG3D was mainly designed for FFHQ (Karras et al., 2019) and uses the true camera poses inferred from real images as the camera distribution for the generator.
Revised: EG3D was mainly designed for FFHQ (Karras et al., 2019) and uses the true camera poses inferred from real images as the camera distribution for the generator.

Original: But in our scenario, we do not have any knowledge about the true camera distribution, that is why to stay as close as possible to the setup EG3D was designed for, we use normal distribution for rotation and elevation angles with the same standard deviations as in FFHQ, which are equal to σ yaw = 0 .
Revised: But in our scenario, we do not have any knowledge about the true camera distribution, that is why to stay as close as possible to the setup EG3D was designed for, we use normal distribution for rotation and elevation angles with the same standard deviations as in FFHQ, which are equal to σ yaw = 0 .

Original: 3 and σ pitch
Revised: 3 and σ pitch

Original: = 0 .
Revised: = 0 .

Original: 155 , respectively.
Revised: 155 , respectively.

Original: Also, to learn better geometry, we additionally trained the baselines with a twice wider camera distribution: σ yaw = 0 .and
Revised: Also, to learn better geometry, we additionally trained the baselines with a twice wider camera distribution: σ yaw = 0 .and

Original: σ pitch = 0 .
Revised: σ pitch = 0 .

Original: While it indeed helped to reduce flatness, it also worsened the image quality: up to 500% as measured by FID 2k .
Revised: While it indeed helped to reduce flatness, it also worsened the image quality: up to 500% as measured by FID 2k .

Original: Our model shows substantially better (at least 2 × than EG3D, slightly worse than StyleGAN2) FID 2k and greater NFS on all the datasets.
Revised: Our model shows substantially better (at least 2 × than EG3D, slightly worse than StyleGAN2) FID 2k and greater NFS on all the datasets.

Original: Low NFS indicates flat or repetitive geometry impairing the ability of the model to generate realistic side views.
Revised: Low NFS indicates flat or repetitive geometry impairing the ability of the model to generate realistic side views.

Original: Indeed, as shown in Fig. 
Revised: Indeed, as shown in Fig. 5 (

Original: Adversarial Depth Supervision.
Revised: Adversarial Depth Supervision.

Original: We evaluate the proposed Adversarial Depth Supervision (ADS) and the depth adaptor A .
Revised: We evaluate the proposed Adversarial Depth Supervision (ADS) and the depth adaptor A .

Original: The only hyperparameter it has is the probability of using the non-adapted depth P ( ¯ d ) (see §3.2).
Revised: The only hyperparameter it has is the probability of using the non-adapted depth P ( ¯ d ) (see §3.2).

Original: We ablate this parameter in Tab. 1b.
Revised: We ablate this parameter in Tab. 1b.

Original: While FID scores are only slightly affected by varying P ( ¯ d ) , we see substantial difference in the Non-Flatness Score.
Revised: While FID scores are only slightly affected by varying P ( ¯ d ) , we see substantial difference in the Non-Flatness Score.

Original: We first verify that NFS is the worst without ADS, indicating the lack of a 3D bias.
Revised: We first verify that NFS is the worst without ADS, indicating the lack of a 3D bias.

Original: When P ( ¯ d ) = 0 , the discriminator is never presented with the rendered depth ¯ d , while the adaptor learns to fake the depth,
Revised: When P ( ¯ d ) = 0 , the discriminator is never presented with the rendered depth ¯ d , while the adaptor learns to fake the depth,

Original: [Table]
Revised: [Table]

Original: + ADS, P ( ¯ d ) = 1 .12.
Revised: + ADS, P ( ¯ d ) = 1 .12.

Original: p r
Revised: p r

Original: i o r g r a d p e n a l t y n o r e g r e
Revised: i o r g r a d p e n a l t y n o r e g r e

Original: s i du a l yaw pitch
Revised: s i du a l yaw pitch

Original: fov look-at yaw look-at pitch look-at radius yaw=0 ◦
Revised: fov look-at yaw look-at pitch look-at radius yaw=0 ◦

Original: yaw=30 ◦ yaw=60 ◦ leading to flat geometry.
Revised: yaw=30 ◦ yaw=60 ◦ leading to flat geometry.

Original: When P ( ¯ d ) = 1 , the adaptor is never used, allowing D to easily determine the generated depth from the estimated depth, as there is a large domain gap (see Fig. 4), leading to reduced FID scores.
Revised: When P ( ¯ d ) = 1 , the adaptor is never used, allowing D to easily determine the generated depth from the estimated depth, as there is a large domain gap (see Fig. 4), leading to reduced FID scores.

Original: The best results are achieved with P ( ¯ d ) = 0 .
Revised: The best results are achieved with P ( ¯ d ) = 0 .

Original: 5 , which can be visually verified from observing Fig.
Revised: 5 , which can be visually verified from observing Fig.

Original: 5 (middle).
Revised: 5 (middle).

Original: A “bare” 3DGP, P ( ¯ d ) = 0 and P ( ¯ d ) = 1 are unable to render side views, while the model trained with P ( ¯ d ) = 0 .
Revised: A “bare” 3DGP, P ( ¯ d ) = 0 and P ( ¯ d ) = 1 are unable to render side views, while the model trained with P ( ¯ d ) = 0 .

Original: 5 features the overall best geometry and side views.
Revised: 5 features the overall best geometry and side views.

Original: Knowledge Distillation.
Revised: Knowledge Distillation.

Original: Here we further provide insights for discriminator D using our knowledge distillation strategy (see §3.3).
Revised: Here we further provide insights for discriminator D using our knowledge distillation strategy (see §3.3).

Original: We find that knowledge distillation provides an additional stability for adversarial training, along with the significant improvement in FID, which can be observed by comparing results of EpiGRAF in Tab.
Revised: We find that knowledge distillation provides an additional stability for adversarial training, along with the significant improvement in FID, which can be observed by comparing results of EpiGRAF in Tab.

Original: 1a and Bare 3DGP Tab. 1a.
Revised: 1a and Bare 3DGP Tab. 1a.

Original: Additionally, we compare different knowledge distillation strategies in Appx C.
Revised: Additionally, we compare different knowledge distillation strategies in Appx C.

Original: However, as noted by Parmar et al.
Revised: However, as noted by Parmar et al.

Original: (2021), strategies which utilize additional classification networks, may provide a significant boost to FID, without corresponding improvement in visual quality.
Revised: (2021), strategies which utilize additional classification networks, may provide a significant boost to FID, without corresponding improvement in visual quality.

Original: “Ball-in-Sphere” Camera Distribution.
Revised: “Ball-in-Sphere” Camera Distribution.

Original: In Fig.
Revised: In Fig.

Original: 7, we analyze different strategies for learning the camera distribution: sampling the camera φ from the prior φ ′ without learning, predicting residuals φ − φ ′ , and using our proposed camera generator C with Camera Gradient Penalty.
Revised: 7, we analyze different strategies for learning the camera distribution: sampling the camera φ from the prior φ ′ without learning, predicting the residual φ − φ ′ , and using our proposed camera generator C with Camera Gradient Penalty.

Original: For the first two cases, the learned distributions are highly concentrated around constant values.
Revised: For the first two cases, the learned distributions are nearly deterministic.

Original: Not surprisingly, no meaningful side views can be generated as geometry becomes flat.
Revised: Not surprisingly, no meaningful side views can be generated as geometry becomes flat.

Original: In contrast, our Camera
Revised: In contrast, our regularization provides

Original: [Table]
Revised: [Table]

Original: Gradient Penalty provides sufficient regularization to C , enabling it to converge to a sufficiently wide posterior, resulting into valid geometry and realistic side views
Revised: sufficient regularization to C , enabling it to converge to a sufficiently wide posterior, resulting into valid geometry and realistic side views. After the submission

Revised: [Table]

Revised: we found a simpler and more flexible camera regularization strategy through entropy maximization, which we discuss in Appx J

Original: [Table]
Revised: [Table]

Original: 4. 3D SYNTHESIS ON I MAGE N ET
Revised: 4. 3D SYNTHESIS ON I MAGE N ET

Original: ImageNet (Deng et al., 2009) is significantly more difficult than single category datasets.
Revised: ImageNet (Deng et al., 2009) is significantly more difficult than single-category datasets.

Original: Following prior works, we trained all the methods in the conditional generation setting Brock et al.
Revised: Following prior works [Table] we trained all the methods in the conditional generation setting Brock et al.

Original: The quantitative results are presented in Tab. 2.
Revised: The quantitative results are presented in Tab. 2.

Original: We also report the results of state-of-the-art 2D generators for reference: as expected, they show much FID and IS than 3D generators, since they do not need to learnthe geometry, are trained on larger compute and had been studied for much longer.
Revised: We also report the results of state-of-the-art 2D generators for reference: as expected, they show better FID and IS than 3D generators, since they do not need to learn geometry, are trained on larger compute, and had been studied for much longer.

Original: Despite our best efforts to find a reasonable camera distribution, both EG3D and EpiGRAF produce flat or repetitive geometry, while 3DGP produces geometry with rich details (see Fig. 1).
Revised: Despite our best efforts to find a reasonable camera distribution, both EG3D and EpiGRAF produce flat or repetitive geometry, while 3DGP produces geometry with rich details (see Fig. 1).

Original: StyleNeRF (Gu et al., 2022) and VolumeGAN Xu et al.
Revised: StyleNeRF (Gu et al., 2022) and VolumeGAN Xu et al.

Original: (2021), trained for conditional ImageNet generation with the narrow camera distribution, substantially under-performed in terms of visual quality.
Revised: (2021), trained for conditional ImageNet generation with the narrow camera distribution, substantially under-performed in terms of visual quality.

Original: We hypothesize that the reason lies in their MLP/voxel-based NeRF backbones: they have  better 3D prior than tri-planes, but are considerably more expensive, which, inits turn, requires sacrifices in terms of  generator’s expressivity.
Revised: We hypothesize that the reason lies in their MLP/voxel-based NeRF backbones: they have a better 3D prior than tri-planes, but are considerably more expensive, which, in turn, requires sacrifices in terms of the generator’s expressivity.

Original: Also, we found that increasing EMA half-life for our

Original: G improves its performance (see Tab. 2).

Original: Training a 3D generator from scratch is  the only way to achieve 3D-aware synthesis: one can lift an existing 2D generator into 3D using the techniques from 3D photo “inpainting”.
Revised: Training a 3D generator from scratch is not the only way to achieve 3D-aware synthesis: one can lift an existing 2D generator into 3D using the techniques from 3D photo “inpainting”.

Original: To test this idea, we generate 10k images with StyleGAN-XL (the current SotA on ImageNet) and then run 3DPhoto
Revised: To test this idea, we generate 10k images with StyleGAN-XL (the current SotA on ImageNet) and then run 3DPhoto

Original: (Shih et al., 2020) on them.
Revised: (Shih et al., 2020) on them.

Original: This method first lifts a photo into 3D via a pre-trained depth estimator and then inpaints the holes with a separately trained GAN model to generate novel views.
Revised: This method first lifts a photo into 3D via a pre-trained depth estimator and then inpaints the holes with a separately trained GAN model to generate novel views.

Original: This method works well for negligible camera variations, but starts diverging when the camera moves at least by 5 ◦ .
Revised: This method works well for negligible camera variations, but starts diverging when the camera moves more than 10 ◦ .

Original: In Tab. 2, we report its FID/IS scores with the narrow camera distribution: σ yaw ∼ N (0 , 0 . 3) and σ pitch ∼ N
Revised: In Tab. 2, we report its FID/IS scores with the narrow camera distribution: σ yaw ∼ N (0 , 0 . 3) and σ pitch ∼ N

Original: ( π/ 2 , 0 . 15) ).
Revised: ( π/ 2 , 0 . 15) ).

Original: The details are in Appx I.
Revised: The details are in Appx I.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we present the first 3D synthesis framework for in-the-wild, multi-category datasets, such as ImageNet.
Revised: In this work, we present the first 3D synthesis framework for in-the-wild, multi-category datasets, such as ImageNet.

Original: We demonstrate how to utilize generic priors in the form of ( imprecise ) monocular depth and latent feature representation to improve the visual quality and guide the geometry.
Revised: We demonstrate how to utilize generic priors in the form of ( imprecise ) monocular depth and latent feature representation to improve the visual quality and guide the geometry.

Original: Moreover, we propose the new “Ball-in-Sphere” camera model with a novel regularization scheme that enables learning meaningful camera distribution.
Revised: Moreover, we propose the new “Ball-in-Sphere” camera model with a novel regularization scheme that enables learning meaningful camera distribution.

Original: On the other hand, our work still has several limitations, such as sticking background, lower visual quality compared to 2D generators, and no reliable quantitative measure of generated geometry.
Revised: On the other hand, our work still has several limitations, such as sticking background, lower visual quality compared to 2D generators, and no reliable quantitative measure of generated geometry.

Original: Additional discussion is provided in Appx A.
Revised: Additional discussion is provided in Appx A.

Original: This project consumed ≈ 12 NVidia A100 GPU years in total.
Revised: This project consumed ≈ 12 NVidia A100 GPU years in total.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: Most importantly, we will release 1) the source code and the checkpoints of our generator as a separate github repo; and 2) fully pre-processed datasets used in our work (with the corresponding extracted depth maps).
Revised: Most importantly, we will release 1) the source code and the checkpoints of our generator as a separate github repo; and 2) fully pre-processed datasets used in our work (with the corresponding extracted depth maps).

Original: In §3 and §4 and Appx B, and also in the figures throughout the text, we provided a complete list of architecture and optimization details needed to reproduce our results.
Revised: In §3 and §4 and Appx B, and also in the figures throughout the text, we provided a complete list of architecture and optimization details needed to reproduce our results.

Original: In §5, we also We are also open to provide any further details on our work in public and/or private correspondence.
Revised: We are also open to provide any further details on our work in public and/or private correspondence.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: There are two main ethical concerns aroundthe deep learning projects on synthesis: generation of fake content to mislead people (e.g. fake news or deepfakes 2 ) and potential licensing, authorship and distribution abuse, which recently got controversy in the research community in the context of 2D image generation (by Stable Diffusion Rombach et al.
Revised: There are two main ethical concerns around deep learning projects on synthesis: generation of fake content to mislead people (e.g. fake news or deepfakes 2 ) and potential licensing, authorship and distribution abuse. This was recently discussed in the research community in the context of 2D image generation (by Stable Diffusion Rombach et al.

Original: and DALL-E Ramesh et al.
Revised: and DALL-E Ramesh et al.

Original: and code generation (by Github Copilot 3 ).
Revised: and code generation (by Github Copilot 3 ).

Original: While the subpar synthesis quality of our generator makes it safe from such concerns, we still warn the  community to build on top of the developed ideas with caution, since they could strengthen a potentially harmful system.
Revised: While the current synthesis quality of our generator would still need to be improved to fool an attentive human observer, we still encourage the research community to discuss ideas and mechanisms for preventing abuse in the future.

Original: A L IMITATIONS
Revised: A L IMITATIONS

Original: Lower visual quality compared to 2D generators .
Revised: Lower visual quality compared to 2D generators .

Original: Despite providing a more reasonable representation of the underlining scene, 3D generators still have a lower visual quality compared to 2D generators.
Revised: Despite providing a more reasonable representation of the underlining scene, 3D generators still have a lower visual quality compared to 2D generators.

Original: Closing this gap is essential for a wide adaptation of 3D generators.
Revised: Closing this gap is essential for a wide adaptation of 3D generators.

Original: Background sticking .
Revised: Background sticking .

Original: One common 3D artifact of 3DGP is gluing of the foreground and the background.
Revised: One common 3D artifact of 3DGP is gluing of the foreground and the background.

Original: In other words our model predicts a single shape for both and there is no clear separation between the two.
Revised: In other words our model predicts a single shape for both and there is no clear separation between the two.

Original: One potential cause of this artifact is the dataset bias, where most of the photos are frontal, thus it is not beneficial for the model to explore backward views.
Revised: One potential cause of this artifact is the dataset bias, where most of the photos are frontal, thus it is not beneficial for the model to explore backward views.

Original: Another reason is the bias of tri-planes toward flat geometry.
Revised: Another reason is the bias of tri-planes toward flat geometry.

Original: However, all our attempts to replace tri-planes with an MLPbased NeRF led to much worse results (see Appx D).
Revised: However, all our attempts to replace tri-planes with an MLPbased NeRF led to much worse results (see Appx D).

Original: Inventing a different efficient parametrization may be an important direction.
Revised: Inventing a different efficient parametrization may be an important direction.

Original: No reliable quantitative measure of generated geometry .
Revised: No reliable quantitative measure of generated geometry .

Original: In this work we introduce Non-Flatness
Revised: In this work we introduce Non-Flatness

Original: Score as a proxy metric for evaluating the quality of the underlying geometry.
Revised: Score as a proxy metric for evaluating the quality of the underlying geometry.

Original: However it can capture only a single failure case, specific for generators based on tri-planes: flatness of geometry.
Revised: However it can capture only a single failure case, specific for generators based on tri-planes: flatness of geometry.

Original: Devising a reasonable metric applicable to a variety of scenarios could significantly speed up the progress in this area.
Revised: Devising a reasonable metric applicable to a variety of scenarios could significantly speed up the progress in this area.

Original: Camera model C does not learn fine-grained control .
Revised: Camera generator C does not learn fine-grained control .

Original: While our camera model is conditioned on the class label c , and, in theory, it should be able to perform fine-grained control over the class focal length distributions (which is natural since landscape panoramas and close-up view of a coffee mug typically have different focal lengths), it does not do this, as shown in Fig.
Revised: While our camera generator is conditioned on the class label c , and, in theory, it should be able to perform fine-grained control over the class focal length distributions (which is natural since landscape panoramas and close-up view of a coffee mug typically have different focal lengths), it does not do this, as shown in Fig.

Original: We attribute this problem to the implicit bias of G to produce large-FoV images due to tri-planes parametrization.
Revised: We attribute this problem to the implicit bias of G to produce large-FoV images due to tri-planes parametrization.

Original: Triplanes define a limited volume box in space, and close-up renderings with large focal length would utilize fewer tri-plane features, hence using less generator’s capacity.
Revised: Tri-planes define a limited volume box in space, and close-up renderings with large focal length would utilize fewer tri-plane features, hence using less generator’s capacity.

Original: This is why 3DGP attempts to perform modeling with larger field-of-view values.
Revised: This is why 3DGP attempts to perform modeling with larger field-of-view values.

Original: [Figure]
Revised: [Figure]

Original: B I MPLEMENTATION DETAILS
Revised: B I MPLEMENTATION DETAILS

Original: This section provides additional architectural and training details.
Revised: This section provides additional architectural and training details.

Original: Also note that wewill release the source code.
Revised: Also, note that we release the source code.

Original: Our generator G consists of M , S and T .
Revised: Our generator G consists of M , S and T .

Original: Mapping network M takes noise z ∈ Rand class label c ∈ 0 , ..., K − 1 , where K is the number of classes, and produces the style code w ∈ R 512 .
Revised: Mapping network M takes noise z ∈ Rand class label c ∈ 0 , ..., K − 1 , where K is the number of classes, and produces the style code w ∈ R 512 .

Original: Similar to StyleGAN2-ADA Karras et al.
Revised: Similar to StyleGAN2-ADA Karras et al.

Original: (2020a)
Revised: (2020a)

Original: and EpiGRAF Skorokhodov et al.
Revised: and EpiGRAF Skorokhodov et al.

Original: (2022), M is 2-layer MLP network with LeakyReLU activations and 512 neurons in each layer.
Revised: (2022), M is 2-layer MLP network with LeakyReLU activations and 512 neurons in each layer.

Original: Synthesis network is a decoder network same as StyleGAN2 Karras et al.
Revised: Synthesis network is a decoder network same as StyleGAN2 Karras et al.

Original: (2020b), except that it produce tri-plane feature features p
Revised: (2020b), except that it produce tri-plane feature features p

Original: = ( p xy , p yz , p xz ) ∈ R 3 × (512 × 512 × 32) .
Revised: = ( p xy , p yz , p xz ) ∈ R 3 × (512 × 512 × 32) .

Original: A feature vector f xyz ∈ R 32 located ( x, y, z ) ∈ R 3 is computed by projecting the coordinate back to the tri-plane representation, followed by bi-linearly interpolating the nearby features and averaging the features from different planes.
Revised: A feature vector f xyz ∈ R 32 located ( x, y, z ) ∈ R 3 is computed by projecting the coordinate back to the tri-plane representation, followed by bi-linearly interpolating the nearby features and averaging the features from different planes.

Original: Following EpiGRAF (Skorokhodov et al., 2022), tri-plane decoder is a twolayer MLP with Leaky-ReLU activations and 64 neurons in the hidden layer, that takes a tri-plane
Revised: Following EpiGRAF (Skorokhodov et al., 2022), tri-plane decoder is a twolayer MLP with Leaky-ReLU activations and 64 neurons in the hidden layer, that takes a tri-plane

Original: [Figure]
Revised: [Table]

Original: Estimator [Figure] [Equation] [Figure]
Revised: feature f xyz as input and produced the color and density ( RGB , σ ) in that point.

Original: [Equation] [Figure] [Equation] [Figure]
Revised: We use same volume rendering procedure as EpiGRAF (Skorokhodov et al., 2022).

Original: [Equation] [Figure] ϕ (a a higher Each
Revised: We also found that increasing the half-life of the exponential moving average for our G improves both FID and Inception Score.

Original: [Figure] (b) Camera gradient penalty . We structure the regularization term for C as a function
Revised: We observed this by noticing that the samples change too rapidly over the course of training.

Original: L ( | g | ) = | g | +1 / | g | (see (1)), and this function is visualized above. This allows to prevent the collapse of C into delta distribution by it either producing constant values or very large/small values (which become constant after the sigmoid normalization). feature f xyz as input and produced the color and density ( RGB , σ ) in that point. We use same volume rendering procedure as EpiGRAF (Skorokhodov et al., 2022).
Revised: In practice, we log the samples every 400k seen images and noticed that the generator could completely change the global structure of a sample during that period.

Original: Camera generator .
Revised: Camera generator .

Original: Camera generator consists of linear layers with SoftPlus activations, it architecture is depicted in Fig.
Revised: Camera generator consists of linear layers with SoftPlus activations, it architecture is depicted in Fig.

Original: We found it crucial to use SoftPlus activation and not LeakyReLU, since optimization of Camera Gradient Penalty for non-smooth functions is unstable for small learning rates (smaller 0.02).
Revised: We found it crucial to use SoftPlus activation and not LeakyReLU, since optimization of Camera Gradient Penalty for non-smooth functions is unstable for small learning rates (smaller 0.02).

Original: Our gradient penalty minimizes the function L = | g | + 1 / | g | , where | g | is the input/output scalar derivative of the camera generator C .
Revised: Our gradient penalty minimizes the function L = | g | + 1 / | g | , where | g | is the input/output scalar derivative of the camera generator C .

Original: The motivation is to prevent the collapse of C into delta distribution and the intuition is the following.
Revised: The motivation is to prevent the collapse of C into delta distribution and the intuition is the following.

Original: C can collapse into delta distribution in two ways: 1) by starting to produce the constant output for all the input values (this is being prevented by the first term | g | and 2) by starting producing ±∞ for all the inputs, which are at the end converted to constants since we apply sigmoid normalization on top of its outputs to normalize them into a proper range (e.g., pitch is bounded in (0, π )) — this, in turn, is prevented by the second term 1 / | g | .
Revised: C can collapse into delta distribution in two ways: 1) by starting to produce the constant output for all the input values (this is being prevented by the first term | g | and 2) by starting producing ±∞ for all the inputs, which are at the end converted to constants since we apply sigmoid normalization on top of its outputs to normalize them into a proper range (e.g., pitch is bounded in (0, π )) — this, in turn, is prevented by the second term 1 / | g | .

Original: The minimum value of this function is 2 (which is achieved when the gradient norm is constant and equals to 1), and the function itself is visualized in Fig. 10b.
Revised: The minimum value of this function is 2 (which is achieved when the gradient norm is constant and equals to 1), and the function itself is visualized in Fig. 9b.

Original: Adversarial depth supervision .
Revised: Adversarial depth supervision .

Original: For depth adaptor, we use a three layer convolutional neural network with 5 × 5 kernel sizes (since it is shallow, we increase its receptive field by increasing the kernel size) with LeakyReLU activations and 64 filters in each layer.
Revised: For depth adaptor, we use a three layer convolutional neural network with 5 × 5 kernel sizes (since it is shallow, we increase its receptive field by increasing the kernel size) with LeakyReLU activations and 64 filters in each layer.

Original: Additionally we use one shared convolutional layer that converts 64 × h × w features to the depth maps.
Revised: Additionally we use one shared convolutional layer that converts 64 × h × w features to the depth maps.

Original: We use the same architecture for D as EpiGRAF (Skorokhodov et al., 2022), but additionally concatenate a 1-channel depth to the 3-channel RGB input.
Revised: We use the same architecture for D as EpiGRAF (Skorokhodov et al., 2022), but additionally concatenate a 1-channel depth to the 3-channel RGB input.

Original: Finally E and F is pretrainined LeReS (Miangoleh et al., 2021) and ResNet50 (He et al., 2016) networks without any modifications.
Revised: Finally E and F is pretrainined LeReS (Miangoleh et al., 2021) and ResNet50 (He et al., 2016) networks without any modifications.

Original: We used the timm library to extract the features for real images.
Revised: We used the timm library to extract the features for real images.

Original: Similarly to EpiGRAF (Skorokhodov et al., 2022) we set the R 1 regularization (Mescheder et al., 2018) term
Revised: Similarly to EpiGRAF (Skorokhodov et al., 2022) we set the R 1 regularization (Mescheder et al., 2018) term

Original: λ r = 0 .
Revised: λ r = 0 .

Original: 1 and knowledge distillation term λ dist = 1 .
Revised: 1 and knowledge distillation term λ dist = 1 .

Original: R 1 regularization helps to stabilize
Revised: R 1 regularization helps to stabilize

Original: GAN training and is formulated as a gradient penalty on top of the discriminator’s real inputs:
Revised: GAN training and is formulated as a gradient penalty on top of the discriminator’s real inputs:

Original: [Equation]
Revised: [Equation]

Original: The (hand-wavy) intuition is that it makes the discriminator surface more flat in the vicinity of real points, making it easier for the generator to reach them.
Revised: The (hand-wavy) intuition is that it makes the discriminator surface more flat in the vicinity of real points, making it easier for the generator to reach them.

Original: We train all the models with Adam optimizer (Kingma & Ba, 2014) using the learning rate of 2 e - 3 and β 1 = 0 . 0 , β 2 = 0 .
Revised: We train all the models with Adam optimizer (Kingma & Ba, 2014) using the learning rate of 2 e - 3 and β 1 = 0 . 0 , β 2 = 0 .

Original: Following
Revised: Following

Original: EpiGRAF (Skorokhodov et al., 2022), our model uses patch-wise training with 64 × 64 -resolution patches and uses their proposed β scale sampling strategy without any modifications.
Revised: EpiGRAF (Skorokhodov et al., 2022), our model uses patch-wise training with 64 × 64 -resolution patches and uses their proposed β scale sampling strategy without any modifications.

Original: We use the batch size of 64 in all the experiments, since in early experiments we didn’t find any improvements from using a large batch size neither for our model nor for StyleGAN2, as observed by Brock et al.
Revised: We use the batch size of 64 in all the experiments, since in early experiments we didn’t find any improvements from using a large batch size neither for our model nor for StyleGAN2, as observed by Brock et al.

Original: (2018) and Sauer et al.
Revised: (2018) and Sauer et al.

Original: As being said in Section 4, we use the instance selection technique by DeVries et al.
Revised: As being said in Section 4, we use the instance selection technique by DeVries et al.

Original: (2020) to remove image outliers from the datasets since they might negatively affect the geometry.
Revised: (2020) to remove image outliers from the datasets since they might negatively affect the geometry.

Original: This procedure works by first extracting a 2,048-dimensional feature vector for each image, then fitting a multivariate gaussian distribution on the obtained dataset and removing the images, which features have low probability density.
Revised: This procedure works by first extracting a 2,048-dimensional feature vector for each image, then fitting a multivariate gaussian distribution on the obtained dataset and removing the images, which features [Figure] ϕ [Figure]

Original: For SDIP Dogs and LSUN Horses, we fit a multi-variate gaussian distribution for the whole dataset.
Revised: z [Figure] c [Figure]

Revised: ϕ (a) Camera generator C .

Revised: We condition C on class labels c when generating the camera position

Revised: ˆ φ since it might be different for different classes.

Revised: And we condition it on z when generating the look-at position and field-of-view since it might depend on the object shape (e.g., there is a higher probability to synthesize a close-up view of a dog’s snout rather than its tail).

Revised: Each MLP consists on 3 layers with Soft- plus non-linearities.

Revised: [Figure]

Revised: [Figure]

Revised: (b) Camera gradient penalty .

Revised: We structure the regularization term for C as a function

Revised: L ( | g | ) = | g | +1 / | g | (see (1)), and this function is visualized above.

Revised: This allows to prevent the collapse of C into delta distribution by it either producing constant values or very large/small values (which become constant after the sigmoid normalization).

Revised: have low probability density.

Revised: For SDIP Dogs and LSUN Horses, we fit a multi-variate gaussian distribution for the whole dataset.

Original: For ImageNet, we fit a separate model for each class with additional diagonal regularization for covariance, which is needed due to its singularity: feature vector has more dimensions than the number of images in a class.
Revised: For ImageNet, we fit a separate model for each class with additional diagonal regularization for covariance, which is needed due to its singularity: feature vector has more dimensions than the number of images in a class.

Original: We refer to (DeVries et al., 2020) for additional details.
Revised: We refer to (DeVries et al., 2020) for additional details.

Original: Also note that we will release the source code and the checkpoints, which would additionally convey all other implementation details.
Revised: Also note that we will release the source code and the checkpoints, which would additionally convey all other implementation details.

Original: C K NOWLEDGE DISTILLATION
Revised: C K NOWLEDGE DISTILLATION

Original: In this section we compare our knowledge distillation strategy with projected D (Sauer et al., 2021; 2022), another popular technique of utilizing existing classification models for GAN training.
Revised: In this section we compare our knowledge distillation strategy with projected D (Sauer et al., 2021; 2022), another popular technique of utilizing existing classification models for GAN training.

Original: Note that projected D relies on pretrained EfficientNet (Tan & Le, 2019), it freezes most of the weights and adds some random convolutions layers that project intermediate outputs of EfficientNet to the features, that will be later processed by discriminator.
Revised: Note that projected D relies on pretrained EfficientNet (Tan & Le, 2019), it freezes most of the weights and adds some random convolutions layers that project intermediate outputs of EfficientNet to the features, that will be later processed by discriminator.

Original: Note that this strategy relies on specific architecture of Discriminator D , thus in order to adapt it for 3DGP we have to rely on two discriminators.
Revised: Note that this strategy relies on specific architecture of Discriminator D , thus in order to adapt it for 3DGP we have to rely on two discriminators.

Original: First one is our disciminator D with additional depth input and conditioned on patch parameters and second one is projected D .
Revised: First one is our disciminator D with additional depth input and conditioned on patch parameters and second one is projected D .

Original: In Tab 3 we show comparison between projected D and our knowledge distillation strategy on ImageNet (Deng et al., 2009) and SDIP Dogs 40k Mokady et al.
Revised: In Tab 4 we show comparison between projected D and our knowledge distillation strategy on ImageNet (Deng et al., 2009) and SDIP Dogs 40k Mokady et al.

Original: (2022) datasets.
Revised: (2022) datasets.

Original: First we compare these strategies for 2D generator architectures: StyleGAN3-t (large) (Karras et al., 2021) and StyleGAN2 (Karras et al., 2020a).
Revised: First we compare these strategies for 2D generator architectures: StyleGAN3-t (large) (Karras et al., 2021) and StyleGAN2 (Karras et al., 2020a).

Original: It can be observed that projected
Revised: It can be observed that projected

Original: D is not generic and it results heavily varies depending on generator architecture.
Revised: D is not generic and it results heavily varies depending on generator architecture.

Original: Moreover training cost of projective D is higher.
Revised: Moreover training cost of projective D is higher.

Original: Finally we test this strategy for our 3D generator, and again we can observe that this technique lead to inferior results in both visual quality and training cost.
Revised: Finally we test this strategy for our 3D generator, and again we can observe that this technique lead to inferior results in both visual quality and training cost.

Original: [Table] [Figure] Typically, knowledge distillation in classifiers is performed using Kullback-Leibler divergence on top of logits (Hinton et al., 2015; Kim et al., 2021), rather than L 2 distance on top of hidden activations, as in our case.
Revised: Typically, knowledge distillation in classifiers is performed using Kullback-Leibler divergence on top of logits (Hinton et al., 2015; Kim et al., 2021), rather than L 2 distance on top of hidden activations, as in our case.

Original: There are two design reasons of why we use the L 2 : 1) it is more generalizable and one can transfer knowledge from other models with the same design, like CLIP (Radford et al., 2021) or Mask R-CNN (He et al., 2017); 2) our discriminator’s task shouldn’t be able to perform classification, that is why guiding it with the KL classification loss is not natural.
Revised: There are two design reasons of why we use the L 2 : 1) it is more generalizable and one can transfer knowledge from other models with the same design, like CLIP (Radford et al., 2021) or Mask R-CNN (He et al., 2017); 2) our discriminator’s task shouldn’t be able to perform classification, that is why guiding it with the KL classification loss is not natural.

Original: However, in
Revised: However, in

Original: Fig.
Revised: Fig.

Original: 11, we provide additional exploration with other knowledge distillation objectives on top of
Revised: 10, we provide additional exploration with other knowledge distillation objectives on top of

Original: StyleGAN2 (Karras et al., 2020b) trained for conditional ImageNet generation on the 128 2 resolution.
Revised: StyleGAN2 (Karras et al., 2020b) trained for conditional ImageNet generation on the 128 2 resolution.

Original: One can observe that KL and L 2 objectives perform approximately the same.
Revised: One can observe that KL and L 2 objectives perform approximately the same.

Original: Also, guiding by CLIP underperforms to ResNet guidance initially, but starts to outperform after more training is performed: we hypothesize that the reason is that it is a more general model.
Revised: Also, guiding by CLIP underperforms to ResNet guidance initially, but starts to outperform after more training is performed: we hypothesize that the reason is that it is a more general model.

Original: We do not use CLIP guidance for our generators in this paper so not to give it an unfair advantage compared to other models.
Revised: We do not use CLIP [Table] [Figure]

Original: D F AILED EXPERIMENTS
Revised: D F AILED EXPERIMENTS

Original: In this section we describe several ideas that have been tried in this project, which however did not work for us.
Revised: In this section we describe several ideas that have been tried in this project, which however did not work for us.

Original: Gradient Penalty for Depth Adaptor .
Revised: Gradient Penalty for Depth Adaptor .

Original: To prevent depth adaptor A from completely faking the depth and forcing it to still provide a useful signal to the generator, one could employ a gradient penalty that will force gradient of A close to one.
Revised: To prevent depth adaptor A from completely faking the depth and forcing it to still provide a useful signal to the generator, one could employ a gradient penalty that will force gradient of A close to one.

Original: We tried it in two setups: window-to-pixel gradient and pixel-to-pixel gradient.
Revised: We tried it in two setups: window-to-pixel gradient and pixel-to-pixel gradient.

Original: Even with the large weight for this loss and shallow adaptor with k =kernel size, it didn’t enforce good geometry 3DGP with Projected D .
Revised: Even with the large weight for this loss and shallow adaptor with k =kernel size, it didn’t enforce good geometry 3DGP with Projected D .

Original: Before devoting to knowledge distillation, we spend significant resources trying to incorporate projected D in our setup.
Revised: Before devoting to knowledge distillation, we spend significant resources trying to incorporate projected D in our setup.

Original: For double discriminators setting, see Appx C, we tested: different weighting strategies, less frequent updates for projected D , enabling projected D after some geometry was already learned.
Revised: For double discriminators setting, see Appx C, we tested: different weighting strategies, less frequent updates for projected D , enabling projected D after some geometry was already learned.

Original: We also tested single projected D , where we learned additional depth encoder that can consume our depth inputs.
Revised: We also tested single projected D , where we learned additional depth encoder that can consume our depth inputs.

Original: In all these experiments learned geometry was significantly inferior to the setting without projected D .
Revised: In all these experiments learned geometry was significantly inferior to the setting without projected D .

Original: Most of the time geometry becomes completely flat.
Revised: Most of the time geometry becomes completely flat.

Original: MLPs instead of tri-planes .
Revised: MLPs instead of tri-planes .

Original: We noticed that tri-planes are extremely biased towards flat generation.
Revised: We noticed that tri-planes are extremely biased towards flat generation.

Original: Therefore one natural idea would be to utilize different representations, one possible candidate is
Revised: Therefore one natural idea would be to utilize different representations, one possible candidate is

Original: MLPs Chan et al.
Revised: MLPs Chan et al.

Original: However our experiments with MLPs always lead to significantly inferior visual quality.
Revised: However our experiments with MLPs always lead to significantly inferior visual quality.

Original: Our hypothesis is that they just don’t have enough capacity to model large scale datasets.
Revised: Our hypothesis is that they just don’t have enough capacity to model large scale datasets.

Original: Few-shot NeRF regularization .
Revised: Few-shot NeRF regularization .

Original: We also try to improve learned geometry by incorporating ray entropy minimization loss from InfoNeRF Kim et al.
Revised: We also try to improve learned geometry by incorporating ray entropy minimization loss from InfoNeRF Kim et al.

Original: (2022) with different weights.
Revised: (2022) with different weights.

Original: But the model always diverges.
Revised: But the model always diverges.

Original: We hypothesize that this is because it is significantly more complicated to find proper geometry, if the model is stuck in local minimum where the object densities are very sharp.
Revised: We hypothesize that this is because it is significantly more complicated to find proper geometry, if the model is stuck in local minimum where the object densities are very sharp.

Original: Normal supervision .
Revised: Normal supervision .

Original: Another prominent idea was utilizing normal supervision, since generic networks for normals
Revised: Another prominent idea was utilizing normal supervision, since generic networks for normals

Original: Eftekhar et al.
Revised: Eftekhar et al.

Original: (2021) are also widely available and provide good results on the arbitrary data.
Revised: (2021) are also widely available and provide good results on the arbitrary data.

Original: Since computing normals requires gradient of density σ with respect to ( x, y, z ) and pytorch does not provide second order derivatives for F.grid sample 4 , we developed a custom
Revised: Since computing normals requires gradient of density σ with respect to ( x, y, z ) and pytorch does not provide second order derivatives for F.grid sample 4 , we developed a custom

Original: CUDA kernel that computes second derivative for F.grid sample .
Revised: CUDA kernel that computes second derivative for F.grid sample .

Original: Unfortunately we found that normals supervision is inferior to depth and it is much easier for the generator G to fake normals.
Revised: Unfortunately we found that normals supervision is inferior to depth and it is much easier for the generator G to fake normals.

Original: Preventing C collapse via variance/entropy/moments regularization .
Revised: Preventing C collapse via variance/entropy/moments regularization .

Original: Before arriving to our current form we had been experimenting with other forms of regularization: maximizing variance with some small loss coefficient, or entropy (in the assumption that the posterior distribution is gaussion), or additionally pushing mean/skewness/kurtosis to the one of the gaussian distribution.
Revised: Before arriving to our current form we had been experimenting with other forms of regularization: maximizing variance with some small loss coefficient, or entropy (in the assumption that the posterior distribution is gaussion), or additionally pushing mean/skewness/kurtosis to the one of the gaussian distribution.

Original: But each time, the generator was finding the ways to “cheat” the regularization and managing to produce either a delta distribution or a mixture of delta distributions (it very “likes” doing so since, it is able to completely flat images and cheat the geometry in this regime).
Revised: But each time, the generator was finding the ways to “cheat” the regularization and managing to produce either a delta distribution or a mixture of delta distributions (it very “likes” doing so since, it is able to completely flat images and cheat the geometry in this regime).

Original: E A DDITIONAL SAMPLES
Revised: E A DDITIONAL SAMPLES

Original: Since it is much easier to visualize the samples from NeRF-based generators as videos rather than
Revised: Since it is much easier to visualize the samples from NeRF-based generators as videos rather than

Original: RGB images, we provide all the additional visualizations on https://u2wjb9xxz9q.github.io.
Revised: RGB images, we provide all the additional visualizations on https://snap-research.github.io/3dgp.

Original: F N ON -F LATNESS S CORE DETAILS
Revised: F N ON -F LATNESS S CORE DETAILS

Original: To detect and quantify the flatness of 3D generators, we propose a Non-Flatness Score (NFS).
Revised: To detect and quantify the flatness of 3D generators, we propose a Non-Flatness Score (NFS).

Original: To compute NFS, we sample N latent codes with their corresponding radiance fields.
Revised: To compute NFS, we sample N latent codes with their corresponding radiance fields.

Original: For each radiance field, we perform integration following Eq.
Revised: For each radiance field, we perform integration following Eq.

Original: 2 to obtain its depth, however, we first set the 50% of lowest density to zero.
Revised: 2 to obtain its depth, however, we first set the 50% of lowest density to zero.

Original: This is necessary to cull spurious density artifacts.
Revised: This is necessary to cull spurious density artifacts.

Original: We then normalize each depth map according to the corresponding near and far planes and compute a histogram with B bins, showing the distribution of the depth values.
Revised: We then normalize each depth map according to the corresponding near and far planes and compute a histogram with B bins, showing the distribution of the depth values.

Original: To analyze how much the depth values are concentrated versus spread across the volume, we compute its entropy for each distribution.
Revised: To analyze how much the depth values are concentrated versus spread across the volume, we compute its entropy for each distribution.

Original: Averaging over N depth maps gives the sought score:
Revised: Averaging over N depth maps gives the sought score:

Original: [Equation]
Revised: [Equation]

Original: where b ( d ( i ) )
Revised: where b ( d ( i ) )

Original: j is the normalized number of depth values in the j -th bin of the i -th depth map d ( i ) , with N = 256 and B = 64 .
Revised: j is the normalized number of depth values in the j -th bin of the i -th depth map d ( i ) , with N = 256 and B = 64 .

Original: NFS does not directly evaluate the quality of the geometry.
Revised: NFS does not directly evaluate the quality of the geometry.

Original: Instead, it helps to detect and quantify how flat the generated geometry is.
Revised: Instead, it helps to detect and quantify how flat the generated geometry is.

Original: In Fig.
Revised: In Fig.

Original: 6, we show examples of repetitive geometry generated by EG3D and a more diverse generation produced by 3DGP along with their depth histograms and NFS.
Revised: 6, we show examples of repetitive geometry generated by EG3D and a more diverse generation produced by 3DGP along with their depth histograms and NFS.

Revised: Intuitively, NFS should provide high values to EG3D with a wide camera distribution, since its repetitive shapes are supposed to span the whole ray uniformly.

Revised: But surprisingly, it does not do so and is able to relfect this repeptitiveness artifact as well, yielding low scores, as can be observed from Tables 1 and 2.

Original: G I NVESTIGATING THE POTENTIAL DATA LEAK FROM D EPTH E STIMATOR
Revised: G I NVESTIGATING THE POTENTIAL DATA LEAK FROM D EPTH E STIMATOR

Original: LeReS (Miangoleh et al., 2021) depth estimator (which was used in our work) was pre-trained on a different set of images compared to ImageNet (or other animals datasets, explored in our work),  and this theoretically could help our generator achieve better results by implicitly giving it access to a broader image set rather than giving only texture-less geometric guidance.
Revised: LeReS (Miangoleh et al., 2021) depth estimator (which was used in our work) was pre-trained on a different set of images compared to ImageNet (or other animals datasets, explored in our work), [Figure] [Figure] and this theoretically could help our generator achieve better results by implicitly giving it access to a broader image set rather than giving only texture-less geometric guidance.

Original: In this section, we investigate this, and conclude that it is not the case: the pre-training data of LeReS is too different from the explored datasets, containing almost no animal images — while all the explored datasets are very animal-dominated, including ImageNet which contains 482 animal classes.
Revised: In this section, we investigate this, and conclude that it is not the case: the pre-training data of LeReS is too different from the explored datasets, containing almost no animal images — while all the explored datasets are very animal-dominated, including ImageNet which contains 482 animal classes.

Original: Hence, LeReS has no chance in helping our generator by giving it access to additional data.
Revised: Hence, LeReS has no chance in helping our generator by giving it access to additional data.

Original: LeReS (Miangoleh et al., 2021) trains on a combination of 4 datasets (in fact, parts of them): 1) HRWSI (Xian et al., 2020) contains outdoor city imagery (e.g., buildings, monuments, landscapes); 2)
Revised: LeReS (Miangoleh et al., 2021) trains on a combination of 4 datasets (in fact, parts of them): 1) HRWSI (Xian et al., 2020) contains outdoor city imagery (e.g., buildings, monuments, landscapes); 2)

Original: DiverseDepth (Yin et al., ) contains clips from in-the-wild movies and videos; 3)[Figure] [Equation]
Revised: DiverseDepth (Yin et al., 2021) contains clips from in-the-wild movies and videos; 3)

Original: [Figure] [Equation] [Figure] Holopix50k (Hua et al., 2020) — diverse, in-the-wild web images (it is the most similar to ImageNet in terms of underlying data distribution); and 4) Taskonomy (Zamir et al., 2018) contains indoor scenes (bedrooms, stores, etc.).
Revised: Holopix50k (Hua et al., 2020) — diverse, in-the-wild web images (it is the most similar to ImageNet in terms of underlying data distribution); and 4) Taskonomy (Zamir et al., 2018) contains indoor scenes (bedrooms, stores, etc.).

Original: In Fig.
Revised: In Fig.

Original: 15, we provide 30 random images from each dataset.
Revised: 14, we provide 30 random images from each dataset.

Original: In Tab. 5, we rigorously explore how many animal images does the pre-training data of LeReS contains.
Revised: In Tab. 6, we rigorously explore how many animal images does the pre-training data of LeReS contains.

Original: We perform this in two ways: by directly counting the amount of animals with the pre-trained
Revised: We perform this in two ways: by directly counting the amount of animals with the pre-trained

Original: Mask R-CNN model He et al.
Revised: Mask R-CNN model He et al.

Original: (2017) and by computing FID scores between the pre-training datasets of LeReS and ImageNet animal subset.
Revised: (2017) and by computing FID scores between the pre-training datasets of LeReS and ImageNet animal subset.

Original: The pre-trained Mask R-CNN model from TorchVision is able to detect 10 animal classes: birds, cats, dogs, horses, sheeps, cows, elephants, bears, zebras and giraffes.
Revised: The pre-trained Mask R-CNN model from TorchVision is able to detect 10 animal classes: birds, cats, dogs, horses, sheeps, cows, elephants, bears, zebras and giraffes.

Original: Following the official tutorial, we used a threshold of 0.8.
Revised: Following the official tutorial, we used a threshold of 0.8.

Original: We downloaded the datasets with the official download scripts of LeReS.
Revised: We downloaded the datasets with the official download scripts of LeReS.

Original: [Table]
Revised: [Table]

Original: 7k
Revised: All pre-training data

Original: [Table]
Revised: [Table]

Original: [Table] [Table] From Tab. 4, one can see that the pre-training data of LeReS contains just 3.79% of animal images, while ImageNet contains 48.3%.
Revised: From Tab. 5, one can see that the pre-training data of LeReS contains just 3.79% of animal images, while ImageNet contains 48.3%.

Original: In this way, it is too far away in terms of distribution from the
Revised: In this way, it is too far away in terms of distribution from the

Original: Animal subset of ImageNet.
Revised: Animal subset of ImageNet.

Original: From this, one can conclude the following: Depth Estimator has almost never seen animals during training, since its training datasets have 600 fewer animal images than ImageNet.
Revised: From this, one can conclude the following: Depth Estimator has almost never seen animals during training, since its training datasets have 600 fewer animal images than ImageNet.

Original: This means that our generator does not receive any unfair advantage in synthesizing animals from using adversarial depth supervision by implicitly getting access to a larger set of images.
Revised: This means that our generator does not receive any unfair advantage in synthesizing animals from using adversarial depth supervision by implicitly getting access to a larger set of images.

Original: How does it affect our generator’s performance in synthesizing animals?
Revised: How does it affect our generator’s performance in synthesizing animals?

Original: Will it have an unusually poor FID compared to non-animal data or compared to other methods?
Revised: Will it have an unusually poor FID compared to non-animal data or compared to other methods?

Original: In Tab. 5, we report FID scores of each generator on animal vs non-animal subsets of ImageNet.
Revised: In Tab. 6, we report FID scores of each generator on animal vs non-animal subsets of ImageNet.

Original: From these scores, one can see that the trend is the same as for FID on full ImageNet.
Revised: From these scores, one can see that the trend is the same as for FID on full ImageNet.

Original: This implies that our generator performs equally well on data which constitutes the main part of our training dataset and is not a part of the depth estimator pre-training.
Revised: This implies that our generator performs equally well on data which constitutes the main part of our training dataset and is not a part of the depth estimator pre-training.

Original: In this way, one can conclude that adversarial depth supervision helps to achieve better synthesis quality only through geometric guidance.
Revised: In this way, one can conclude that adversarial depth supervision helps to achieve better synthesis quality only through geometric guidance.

Original: It does not help the generator to do this by leaking the knowledge of pre-trained data from the pre-trained depth estimator.only
Revised: It does not help the generator to do this by leaking the knowledge of pre-trained data from the pre-trained depth estimator.

Original: [Table]
Revised: [Table]

Original: H G EOMETRY EVALUATION ON S HAPE N ET
Revised: H G EOMETRY EVALUATION ON S HAPE N ET

Original: In this section, we present the details and the results of an additional study, which rigorously shows that our proposed adversarial depth supervision improves the geometry quality.
Revised: In this section, we present the details and the results of an additional study, which rigorously shows that our proposed adversarial depth supervision improves the geometry quality.

Original: H. R ENDERING DETAILS
Revised: H. R ENDERING DETAILS

Original: We take the ShapeNet dataset Chang et al.
Revised: We take the ShapeNet dataset Chang et al.

Original: (2015), which consists of 50k models from 54 classes and render it from random frontal camera positions.
Revised: (2015), which consists of 50k models from 54 classes and render it from random frontal camera positions.

Original: For this, we set the camera on a sphere of radius 2 and randomly choose its position by sampling rotation and elevation angles from N (0 , π/ 8) and
Revised: For this, we set the camera on a sphere of radius 2 and randomly choose its position by sampling rotation and elevation angles from N (0 , π/ 8) and

Original: N ( π/ 2 , π/ 8) , respectively.
Revised: N ( π/ 2 , π/ 8) , respectively.

Original: We chose frontal views to be closer to the real-world scenario: the most popular image synthesis datasets are dominated by the frontal views.
Revised: We chose frontal views to be closer to the real-world scenario: the most popular image synthesis datasets are dominated by the frontal views.

Original: Also, if the dataset is not frontal, then there is less necessity in additional geometry supervision: it is a good enough 3D bias (which does not exist in modern in-the-wild datasets).
Revised: Also, if the dataset is not frontal, then there is less necessity in additional geometry supervision: it is a good enough 3D bias (which does not exist in modern in-the-wild datasets).

Original: For wide camera distribution, a generator can learn proper geometry on its own: see experiments on synthetic datasets with 360 degrees camera coverage by Skorokhodov et al.
Revised: For wide camera distribution, a generator can learn proper geometry on its own: see experiments on synthetic datasets with 360 degrees camera coverage by Skorokhodov et al.

Original: We render just a single view per model since this is also the scenario which we typically have in real datasets (e.g., in all our explored datasets, there is just a single view per scene).
Revised: We render just a single view per model since this is also the scenario which we typically have in real datasets (e.g., in all our explored datasets, there is just a single view per scene).

Original: Some of the models in ShapeNet are broken (they lack the corresponding mesh files), so we discard them.
Revised: Some of the models in ShapeNet are broken (they lack the corresponding mesh files), so we discard them.

Original: In total, this gave us 51209 training images, which are visualized in Fig.
Revised: In total, this gave us 51209 training images, which are visualized in Fig.

Original: We used BlenderProc Denninger et al.
Revised: We used BlenderProc Denninger et al.

Original: (2019) to do rendering.
Revised: (2019) to do rendering.

Original: Also, during rendering we removed the transparent elements of the meshes since they were providing aliasing issues in the depth maps.
Revised: Also, during rendering we removed the transparent elements of the meshes since they were providing aliasing issues in the depth maps.

Original: H. E XPERIMENTS
Revised: H. E XPERIMENTS

Original: Experimental setup .
Revised: Experimental setup .

Original: After rendering, we trained our main baselines, EG3D (Chan et al., 2022) and
Revised: After rendering, we trained our main baselines, EG3D (Chan et al., 2022) and

Original: EpiGRAF (Skorokhodov et al., 2022), on this dataset using the ground truth camera parametrization and distribution when sampling the camera poses.
Revised: EpiGRAF (Skorokhodov et al., 2022), on this dataset using the ground truth camera parametrization and distribution when sampling the camera poses.

Original: For 3DGP, we trained it in the following variants: 1) with the default adversarial depth supervision (i.e., using P ( ¯ d ) = 0 . 5 ); and 2-3) with the adversarial depth supervision, but with corrupted depth maps, where corruption is simulated by blurring, similar to (DeVries et al., 2021).
Revised: For 3DGP, we trained it in the following variants: 1) with the default adversarial depth supervision (i.e., using P ( ¯ d ) = 0 . 5 ); and 2-3) with the adversarial depth supervision, but with corrupted depth maps, where corruption is simulated by blurring, similar to (DeVries et al., 2021).

Original: Also, we trained StyleGAN2 (Karras et al., 2020b) as a lower bound on FID.
Revised: Also, we trained StyleGAN2 (Karras et al., 2020b) as a lower bound on FID.

Original: We disabled camera distribution learning and knowledge distillation for the experiments with 3DGP to avoid unnecessary complications: we investigate only adversarial depth supervision in this study.
Revised: We disabled camera distribution learning and knowledge distillation for the experiments with 3DGP to avoid unnecessary complications: we investigate only adversarial depth supervision in this study.

Original: For all the baselines, we use white background in volumetric rendering.
Revised: For all the baselines, we use white background in volumetric rendering.

Original: Evaluation .
Revised: Evaluation .

Original: We compare the methods in terms of FID (Heusel et al., 2017) and also Frechet Pointcloud Distance (FPD), proposed by Shu et al.
Revised: We compare the methods in terms of FID (Heusel et al., 2017) and also Frechet Pointcloud Distance (FPD), proposed by Shu et al.

Original: This metric is an FID analog for point clouds: it uses a pre-trained PointNet (Qi et al., 2017a) to extract the features from point clouds and then computes Frechet distance between real and fake representations sets, approximating their underlying distribution as multi-variate normal one.
Revised: This metric is an FID analog for point clouds: it uses a pre-trained PointNet (Qi et al., 2017a) to extract the features from point clouds and then computes Frechet distance between real and fake representations sets, approximating their underlying distribution as multi-variate normal one.

Original: [Table]
Revised: [Table]

Original: We used the official codebase of Shu et al.
Revised: We used the official codebase of Shu et al.

Original: (2019) to compute the metrics.
Revised: (2019) to compute the metrics.

Original: In their original work,
Revised: In their original work,

Original: Shu et al.
Revised: Shu et al.

Original: (2019) used a customly trained PointNet (Qi et al., 2017a) to extract the features.
Revised: (2019) used a customly trained PointNet (Qi et al., 2017a) to extract the features.

Original: But we observed an issue with it: it was sometimes producing unusually high FPD scores in the order of 10 6 , even for real point clouds.
Revised: But we observed an issue with it: it was sometimes producing unusually high FPD scores in the order of 10 6 , even for real point clouds.

Original: This is why we extracted pointclouds features with a pre-trained
Revised: This is why we extracted pointclouds features with a pre-trained

Original: PointNet++ (Qi et al., 2017b) model from a popular public implementation (Yan, 2019).
Revised: PointNet++ (Qi et al., 2017b) model from a popular public implementation (Yan, 2019).

Original: To extract point clouds from the models, we first extracted the surfaces from them via marching cubes.
Revised: To extract point clouds from the models, we first extracted the surfaces from them via marching cubes.

Original: For this, we sampled density fields in 256 3 resolution, and then used the marching cubes implementation of PyMCubes (pmneila, 2015) to extract the surfaces.
Revised: For this, we sampled density fields in 256 3 resolution, and then used the marching cubes implementation of PyMCubes (pmneila, 2015) to extract the surfaces.

Original: Following EG3D, we thresholded the surface for marching cubes at the density value of σ = 10 .
Revised: Following EG3D, we thresholded the surface for marching cubes at the density value of σ = 10 .

Original: Our overall pipeline is identical to the original procedure of SDF extraction in the EG3D repo, but simpler in terms of implementation.
Revised: Our overall pipeline is identical to the original procedure of SDF extraction in the EG3D repo, but simpler in terms of implementation.

Original: After that, we extracted point clouds by sampling 2,048 points on the surface for both real and fake meshes using uniform sampling from trimesh Dawson-Haggerty et al.
Revised: After that, we extracted point clouds by sampling 2,048 points on the surface for both real and fake meshes using uniform sampling from trimesh Dawson-Haggerty et al.

Original: H. R ESULTS
Revised: H. R ESULTS

Original: The results of these experiments are presented in Tab. 6.
Revised: The results of these experiments are presented in Tab. 7.

Original: We also provide random sample examples (seed 1, classes 1-8) in Fig.
Revised: We also provide random sample examples (seed 1, classes 1-8) in Fig.

Original: EG3D (Chan et al., 2022) fails to recover the geometry and generates inverted shapes with hollow geometry.
Revised: EG3D (Chan et al., 2022) fails to recover the geometry and generates inverted shapes with hollow geometry.

Original: This is highlighted by its very high FPD score.
Revised: This is highlighted by its very high FPD score.

Original: EpiGRAF (Skorokhodov et al., 2022) is able to recover shapes, but it models the white background via an additional sphere, which also damages its FPD.
Revised: EpiGRAF (Skorokhodov et al., 2022) is able to recover shapes, but it models the white background via an additional sphere, which also damages its FPD.

Original: Our method, in contrast, learns proper geometry and correctly drops the background.
Revised: Our method, in contrast, learns proper geometry and correctly drops the background.

Original: When the depth maps are corrupted with blurring, it deteriorates both its geometry quality and texture quality, which is also confirmed by the metrics and the provided samples.
Revised: When the depth maps are corrupted with blurring, it deteriorates both its geometry quality and texture quality, which is also confirmed by the metrics and the provided samples.

Original: I D ETAILS ON THE EXPERIMENTS WITH 3D P HOTO I NPAINTING
Revised: I D ETAILS ON THE EXPERIMENTS WITH 3D P HOTO I NPAINTING

Original: In this section, we provide the details of our experiments on combining 2D generation with 3D Photo
Revised: In this section, we provide the details of our experiments on combining 2D generation with 3D Photo

Original: Inpainting techniques.
Revised: Inpainting techniques.

Original: For a 2D generator, we chose StyleGAN-XL (Sauer et al., 2022) since it achieves state-of-theart visual quality on ImageNet, as measured by FID (Heusel et al., 2017).
Revised: For a 2D generator, we chose StyleGAN-XL (Sauer et al., 2022) since it achieves state-of-theart visual quality on ImageNet, as measured by FID (Heusel et al., 2017).

Original: We took the original checkpoint for 256 2 generation from the official repository.
Revised: We took the original checkpoint for 256 2 generation from the official repository.

Original: First, we generated 50,000 images with the model and computed their FID: this gave the value of 2.51 vs 2.26, reported by the authors, which is a negligible difference.
Revised: First, we generated 50,000 images with the model and computed their FID: this gave the value of 2.51 vs 2.26, reported by the authors, which is a negligible difference.

Original: After that, we used the original codebase of 3DPhoto (Shih et al., 2020) to synthesize the 3D variations of 10k random images.
Revised: After that, we used the original codebase of 3DPhoto (Shih et al., 2020) to synthesize the 3D variations of 10k random images.

Original: We kept all the hyperparameters the same.
Revised: We kept all the hyperparameters the same.

Original: In 3DPhoto, there is no spherical camera parametrization, this is why we had to simulate it the following way.
Revised: In 3DPhoto, there is no spherical camera parametrization, this is why we had to simulate it the following way.

Original: We assumed that the sphere center lies at the median depth value from the original camera position (which is (0 , 0 , 0) ), and had been rotating the camera around that point.
Revised: We assumed that the sphere center lies at the median depth value from the original camera position (which is (0 , 0 , 0) ), and had been rotating the camera around that point.

Original: As discussed in §4, we used the narrow camera distribution to sample the points.
Revised: As discussed in §4, we used the narrow camera distribution to sample the points.

Original: Namely, we used the normal distributions with standard deviations of σ yaw = 0 .
Revised: Namely, we used the normal distributions with standard deviations of σ yaw = 0 .

Original: 3 and σ pitch = 0 .
Revised: 3 and σ pitch = 0 .

Original: 15 for sampling rotation and elevation angles, respectively.
Revised: 15 for sampling rotation and elevation angles, respectively.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: For negligible camera variations, this strategy produces excellent results.
Revised: For negligible camera variations, this strategy produces excellent results.

Original: While it is not perfectly view consistent, it inherits state-of-the-art image quality from the 2D generator, which was used to synthesize the original images.
Revised: While it is not perfectly view consistent, it inherits state-of-the-art image quality from the 2D generator, which was used to synthesize the original images.

Original: But for larger camera variations, the quality becomes to deteriorate very quickly.
Revised: But for larger camera variations, the quality becomes to deteriorate very quickly.

Original: Not only noticeable inpainting artifacts start to appear, but also outpainting problem emerges.
Revised: Not only noticeable inpainting artifacts start to appear, but also outpainting problem emerges.

Original: The model was not trained to perform image extrapolation or inpaint large regions and fails to fill the holes which appear in larger camera movements.
Revised: The model was not trained to perform image extrapolation or inpaint large regions and fails to fill the holes which appear in larger camera movements.

Original: We provide the visualizations for it in Fig.
Revised: We provide the visualizations for it in Fig.

Revised: J E NTROPY REGULARIZATION

Revised: After the submission, we explored another strategy to regularize the camera generator and observed that it is more flexible and easier to tune.

Revised: Intuitively, the strategy is to maximize the entropy of each predicted camera parameter ϕ

Revised: i :

Revised: [Equation]

Revised: [Equation]

Revised: where p G ( φ i ) is the camera generator’s distribution over φ i , H is differential entropy.

Revised: To do this, we used the POT package Flamary et al.

Revised: (2021) to minimize the Earth Mover’s Distance with the uniform distribution:

Revised: [Equation]

Revised: where EMD ( P, Q ) denotes the Earth Mover’s Distance between the distributions P and Q , U [ a, b ] is the uniform distribution on [ a, b ] and m φ

Revised: i , M φ i are minimum and maximum values for a given camera parameter φ i .

Revised: In practice, we used 64 samples to approximate the EMD and a non-regularized Wasserstein distance.