Original: 3D GENERATION ON I MAGE N ET
Revised: 3D GENERATION ON I MAGE N ET

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: All existing 3D-from-2D generators are designed for well-curated single-category datasets, where all the objects have (approximately) the same scale, 3D location and orientation, and the camera always points to the center of the scene.
Revised: All existing 3D-from-2D generators are designed for well-curated single-category datasets, where all the objects have (approximately) the same scale, 3D location and orientation, and the camera always points to the center of the scene.

Original: This makes them inapplicable to diverse, in-the-wild datasets of non-alignable scenes rendered from arbitrary camera poses.
Revised: This makes them inapplicable to diverse, in-the-wild datasets of non-alignable scenes rendered from arbitrary camera poses.

Original: In this work, we develop 3D generator with
Revised: In this work, we develop 3D generator with

Original: Generic Priors (3DGP) : a 3D synthesis framework with more general assumptions about the training data, and show that it scales to very challenging datasets, like ImageNet.
Revised: Generic Priors (3DGP) : a 3D synthesis framework with more general assumptions about the training data, and show that it scales to very challenging datasets, like ImageNet.

Original: Our model is based on three new ideas.
Revised: Our model is based on three new ideas.

Original: First, we incorporate an inaccurate off-the-shelf depth estimator into 3D GAN training via a special depth adaptation module to handle the imprecision.
Revised: First, we incorporate an inaccurate off-the-shelf depth estimator into 3D GAN training via a special depth adaptation module to handle the imprecision.

Original: Then, we create a flexible camera model and a regularization strategy for it to learn its distribution parameters during training.
Revised: Then, we create a flexible camera model and a regularization strategy for it to learn its distribution parameters during training.

Original: Finally, we extend the recent ideas of transferring knowledge from pretrained classifiers into GANs for patch-wise trained models by employing a simple distillation-based technique on top of the discriminator.
Revised: Finally, we extend the recent ideas of transferring knowledge from pretrained classifiers into GANs for patch-wise trained models by employing a simple distillation-based technique on top of the discriminator.

Original: It achieves more stable training than the analogs and speeds up the convergence by at least 40%.
Revised: It achieves more stable training than the analogs and speeds up the convergence by at least 40%.

Original: We explore our model on four datasets: SDIP Dogs 256 2 , SDIP Elephants 256 2 ,
Revised: We explore our model on four datasets: SDIP Dogs 256 2 , SDIP Elephants 256 2 ,

Original: LSUN Horses 256 2 , and ImageNet 256 2 and demonstrate that 3DGP outperforms the recent state-of-the-art in terms of both texture and geometry quality.
Revised: LSUN Horses 256 2 , and ImageNet 256 2 and demonstrate that 3DGP outperforms the recent state-of-the-art in terms of both texture and geometry quality.

Original: Visualizations: https://u2wjb9xxz9q.github.io/
Revised: Visualizations: https://u2wjb9xxz9q.github.io

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: the radiance field in the full dataset resolution and learns high-fidelity details during training.
Revised: distribution radiance field in the full dataset resolution and learns high-fidelity details during training.

Original: Zoom-in for a better view.
Revised: Zoom-in for a better view.

Original: We witness remarkable progress in the domain of 3D-aware image synthesis.
Revised: We witness remarkable progress in the domain of 3D-aware image synthesis.

Original: The community is developing new methods to improve the image quality, 3D consistency and efficiency of the generators (e.g., Chan et al.
Revised: The community is developing new methods to improve the image quality, 3D consistency and efficiency of the generators (e.g., Chan et al.

Original: ; Deng et al.
Revised: ; Deng et al.

Original: ; Skorokhodov et al.
Revised: ; Skorokhodov et al.

Original: ; Zhao et al.
Revised: ; Zhao et al.

Original: ; Schwarz et al.
Revised: ; Schwarz et al.

Original: However, all the existing frameworks are designed for well-curated and aligned datasets consisting of objects of the same category, scale and global scene structure, like human or cat faces (Chan et al., 2021).
Revised: However, all the existing frameworks are designed for well-curated and aligned datasets consisting of objects of the same category, scale and global scene structure, like human or cat faces (Chan et al., 2021).

Original: Such curation requires domain-specific 3D knowledge about the object
Revised: Such curation requires domain-specific 3D knowledge about the object

Original: [Figure]
Revised: [Figure]

Original: r e a l / f a k e
Revised: r e a l / f a k e

Original: D tri-planes offline dataset preprocessing
Revised: D tri-planes offline dataset preprocessing

Original: ResNet
Revised: ResNet

Original: LeReS r e a l / f a k e D e ˆ e !
Revised: LeReS r e a l / f a k e D e ˆ e !

Original: e − ˆ e !
Revised: e − ˆ e !

Original: knowledgedistillation ϕ !
Revised: knowledgedistillation ϕ !

Original: category at hand, since one needs to infer the underlying 3D to properly crop, rotate and scale the images (Deng et al., 2022; Chan et al., 2022).
Revised: category at hand, since one needs to infer the underlying 3D to properly crop, rotate and scale the images (Deng et al., 2022; Chan et al., 2022).

Original: This makes it infeasible to perform a similar alignment procedure for large-scale multi-category datasets that are inherently “non-alignable”: there does not exist a single canonical position which all the objects could be transformed into (e.g., it is impossible to align a landscape panorama with a spoon).
Revised: This makes it infeasible to perform a similar alignment procedure for large-scale multi-category datasets that are inherently “non-alignable”: there does not exist a single canonical position which all the objects could be transformed into (e.g., it is impossible to align a landscape panorama with a spoon).

Original: To extend 3D synthesis to in-the-wild datasets, one needs a framework which relies on more universal 3D priors.
Revised: To extend 3D synthesis to in-the-wild datasets, one needs a framework which relies on more universal 3D priors.

Original: In this work, we make a step towards this direction and develop a 3D generator with
Revised: In this work, we make a step towards this direction and develop a 3D generator with

Original: Generic Priors (3DGP): a 3D synthesis model which is guided only by (imperfect) depth predictions from an off-the-shelf monocular depth estimator.
Revised: Generic Priors (3DGP): a 3D synthesis model which is guided only by (imperfect) depth predictions from an off-the-shelf monocular depth estimator.

Original: Surprisingly, such 3D cues are enough to learn reasonable scenes from loosely curated, non-aligned datasets, such as ImageNet (Deng et al., 2009).
Revised: Surprisingly, such 3D cues are enough to learn reasonable scenes from loosely curated, non-aligned datasets, such as ImageNet (Deng et al., 2009).

Original: Training a 3D generator on in-the-wild datasets comes with three main challenges: 1) extrinsic camera parameters of real images are unknown and impossible to infer; 2) objects appear in different shapes, positions, rotations and scales, hindering learning of the underlying 3D geometry; and 3) the dataset typically contains a lot of variation in terms of texture and structure, and is difficult to fit even for 2D generators.
Revised: Training a 3D generator on in-the-wild datasets comes with three main challenges: 1) extrinsic camera parameters of real images are unknown and impossible to infer; 2) objects appear in different shapes, positions, rotations and scales, hindering learning of the underlying 3D geometry; and 3) the dataset typically contains a lot of variation in terms of texture and structure, and is difficult to fit even for 2D generators.

Original: As shown in Fig 1 (left), state-of-the-art 3D-aware generators, such as
Revised: As shown in Fig 1 (left), state-of-the-art 3D-aware generators, such as

Original: EG3D (Chan et al., 2022), struggle to learn the proper geometry in such a challenging scenario.
Revised: EG3D (Chan et al., 2022), struggle to learn the proper geometry in such a challenging scenario.

Original: In this work, we develop three novel techniques to address those problems.
Revised: In this work, we develop three novel techniques to address those problems.

Original: Learnable “Ball-in-Sphere” camera distribution .
Revised: Learnable “Ball-in-Sphere” camera distribution .

Original: Most existing methods utilize a restricted camera model (e.g., (Schwarz et al., 2020; Niemeyer & Geiger, 2021b;
Revised: Most existing methods utilize a restricted camera model (e.g., (Schwarz et al., 2020; Niemeyer & Geiger, 2021b;

Original: Chan et al., 2021)): the camera is positioned on a sphere with a constant radius, always points to the world center and has fixed intrinsics.
Revised: Chan et al., 2021)): the camera is positioned on a sphere with a constant radius, always points to the world center and has fixed intrinsics.

Original: But diverse, non-aligned datasets violate these assumptions: e.g., dogs datasets have images of both close-up photos of a snout and photos of full-body dogs, which implies the variability in the focal length and look-at positions.
Revised: But diverse, non-aligned datasets violate these assumptions: e.g., dogs datasets have images of both close-up photos of a snout and photos of full-body dogs, which implies the variability in the focal length and look-at positions.

Original: Thus, we introduce a novel camera model with 6 degrees of freedom to address this variability.
Revised: Thus, we introduce a novel camera model with 6 degrees of freedom to address this variability.

Original: We optimize its distribution parameters during training and develop an efficient gradient penalty for it to prevent its collapse to delta distribution.
Revised: We optimize its distribution parameters during training and develop an efficient gradient penalty for it to prevent its collapse to delta distribution.

Original: Adversarial depth supervision (ADS).
Revised: Adversarial depth supervision (ADS).

Original: A generic image dataset features a wide diversity of objects with different shapes and poses.
Revised: A generic image dataset features a wide diversity of objects with different shapes and poses.

Original: That is why learning a meaningful 3D geometry together with the camera distribution is an ill-posed problem, as the incorrect scale can be well compensated by an incorrect camera model (Hartley & Zisserman, 2003), or flat geometry (Zhao et al., 2022; Chan et al., 2022).
Revised: That is why learning a meaningful 3D geometry together with the camera distribution is an ill-posed problem, as the incorrect scale can be well compensated by an incorrect camera model (Hartley & Zisserman, 2003), or flat geometry (Zhao et al., 2022; Chan et al., 2022).

Original: To instill the 3D bias, we provide the scene geometry information to the discriminator by concatenating the depth map of a scene as the 4-th channel of its RGB input.
Revised: To instill the 3D bias, we provide the scene geometry information to the discriminator by concatenating the depth map of a scene as the 4-th channel of its RGB input.

Original: For real images, we use their ( imperfect ) estimates from a generic off-the-shelf monocular depth predictor (Miangoleh et al., 2021).
Revised: For real images, we use their ( imperfect ) estimates from a generic off-the-shelf monocular depth predictor (Miangoleh et al., 2021).

Original: For fake images, we render the depth from the synthesized radiance field, and process it with a shallow depth adaptor module, bridging the distribution gap between the estimated and rendered depth maps.
Revised: For fake images, we render the depth from the synthesized radiance field, and process it with a shallow depth adaptor module, bridging the distribution gap between the estimated and rendered depth maps.

Original: This ultimately guides the generator to learn the proper 3D geometry.
Revised: This ultimately guides the generator to learn the proper 3D geometry.

Original: Knowledge distillation into Discriminator.
Revised: Knowledge distillation into Discriminator.

Original: Prior works have shown benefits from transferring the knowledge from conventional 2D image encoders into a synthesis model (Sauer et al., 2022).
Revised: Prior works have shown benefits from transferring the knowledge from conventional 2D image encoders into a synthesis model (Sauer et al., 2022).

Original: The state-of-the-art techniques utilize pretrained image classifiers as the discriminator backbone and develop regularization strategies on top of them (Sauer et al., 2021; Kumari et al., 2022).
Revised: The state-of-the-art techniques utilize pretrained image classifiers as the discriminator backbone and develop regularization strategies on top of them (Sauer et al., 2021; Kumari et al., 2022).

Original: These techniques, however, are only applicable when the discriminator has a similar input distribution compared to what the encoder was trained on.
Revised: These techniques, however, are only applicable when the discriminator has a similar input distribution compared to what the encoder was trained on.

Original: This makes it difficult to use efficient patch-based discriminators (Skorokhodov et al., 2022) or passing depth maps as the 4-th channel to the discriminator.
Revised: This makes it difficult to use efficient patch-based discriminators (Skorokhodov et al., 2022) or passing depth maps as the 4-th channel to the discriminator.

Original: That is why, we develop a more general and efficient knowledge transfer strategy based on knowledge distillation.
Revised: That is why, we develop a more general and efficient knowledge transfer strategy based on knowledge distillation.

Original: It consists in forcing the discriminator to predict features of a pre-trained
Revised: It consists in forcing the discriminator to predict features of a pre-trained

Original: ResNet50 (He et al., 2016) model, effectively transferring the knowledge into our model.
Revised: ResNet50 (He et al., 2016) model, effectively transferring the knowledge into our model.

Original: This technique has just 1% of computational overhead compared to standard training, but allows to improve
Revised: This technique has just 1% of computational overhead compared to standard training, but allows to improve

Original: FID for both 2D and 3D generators by at least 40%.
Revised: FID for both 2D and 3D generators by at least 40%.

Original: First, we explore our ideas on non-aligned single-category image datasets:
Revised: First, we explore our ideas on non-aligned single-category image datasets:

Original: SDIP
Revised: SDIP

Original: Dogs 256 2 (Mokady et al., 2022), SDIP Elephants 256 2 (Mokady et al., 2022), and LSUN
Revised: Dogs 256 2 (Mokady et al., 2022), SDIP Elephants 256 2 (Mokady et al., 2022), and LSUN

Original: Horses 256 2 (Yu et al., 2015).
Revised: Horses 256 2 (Yu et al., 2015).

Original: On these datasets, our generator achieves better image appearance (measured by FID Heusel et al.
Revised: On these datasets, our generator achieves better image appearance (measured by FID Heusel et al.

Original: and geometry quality than the modern state-of-the-art 3D-aware generators.
Revised: and geometry quality than the modern state-of-the-art 3D-aware generators.

Original: Then, we train the model on all the 1 , 000 classes of ImageNet (Deng et al., 2009), showing that multi-categorical 3D synthesis is possible for non-alignable data (see Fig. 1).
Revised: Then, we train the model on all the 1 , 000 classes of ImageNet (Deng et al., 2009), showing that multi-categorical 3D synthesis is possible for non-alignable data (see Fig. 1).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: 3D-aware image synthesis .
Revised: 3D-aware image synthesis .

Original: Mildenhall et al.
Revised: Mildenhall et al.

Original: (2020) introduced Neural Radiance Fields (NeRF): a neural network-based representation of 3D volumes which is learnable from RGB supervision only.
Revised: (2020) introduced Neural Radiance Fields (NeRF): a neural network-based representation of 3D volumes which is learnable from RGB supervision only.

Original: They ignited a stream of 3D-aware image and video generators (Schwarz et al., 2020; Niemeyer & 
Revised: It ignited many 3D-aware image/video generators (Schwarz et al., 2020; Niemeyer & Geiger, 2021b;

Original: Geiger, 2021b; Chan et al., 2021; Xue et al., 2022; Zhou et al., 2021; Wang et al., 2022; Gu et al., 2022; Or-El et al., 2021; Chan et al., 2022; Skorokhodov et al., 2022; Zhang et al., 2022; Xu et al., 2021; Bahmani et al., 2022), all of them being GAN-based (Goodfellow et al., 2014).
Revised: Chan et al., 2021; Xue et al., 2022; Zhou et al., 2021; Wang et al., 2022; Gu et al., 2022; Or-El et al., 2021; Chan et al., 2022; Skorokhodov et al., 2022; Zhang et al., 2022; Xu et al., 2021; Bahmani et al., 2022), all of them being GAN-based (Goodfellow et al., 2014).

Original: Rendering low-resolution images is expensive, those works rely to either patch-wise training (e.g., Schwarz et al.
Revised: Many of them explore the techniques to reduce the cost of 3D-aware generation for high-resolution data, like patch-wise training (e.g., Schwarz et al.

Original: Meng et al.
Revised: Meng et al.

Original: ; Skorokhodov et al.
Revised: ; Skorokhodov et al.

Original: , MPI-based rendering (e.g.,Deng et al.
Revised: , MPI-based rendering (e.g.,

Original: ; Zhao et al.
Revised: Zhao et al.

Original: or use a  2Dconvolutional upsampler (e.g.,Niemeyer & Geiger (2021b); Gu et al.
Revised: or training a separate 2D upsampler (e.g., Gu et al.

Original: Learning the camera poses .
Revised: Learning the camera poses .

Original: NeRFs require known camera poses, obtained from multi-view stereo (Sch ¨ onberger et al., 2016) or structure from motion (Sch ¨ onberger & Frahm, 2016).
Revised: NeRFs require known camera poses, obtained from multi-view stereo (Sch ¨ onberger et al., 2016) or structure from motion (Sch ¨ onberger & Frahm, 2016).

Original: Alternatively, a group of works have been introduced to either automatically estimate camera poses (Wang et al., 2021) or finetune them during training (Lin et al., 2021; Kuang et al., 2022).
Revised: Alternatively, a group of works have been introduced to either automatically estimate camera poses (Wang et al., 2021) or finetune them during training (Lin et al., 2021; Kuang et al., 2022).

Original: The problem we are tackling in this work is fundamentally different as it requires to learn not the camera poses from multi-view observations, but a distribution of poses, while having access to sparse, single-view data of diverse object categories.
Revised: The problem we are tackling in this work is fundamentally different as it requires to learn not the camera poses from multi-view observations, but a distribution of poses, while having access to sparse, single-view data of diverse object categories.

Original: In this respect, the work closest to ours is CAMPARI (Niemeyer &
Revised: In this respect, the work closest to ours is CAMPARI (Niemeyer &

Original: Geiger, 2021a) as it also learns a camera distribution.
Revised: Geiger, 2021a) as it also learns a camera distribution.

Original: Beyond this the similarity ends, as our “Ballin-Sphere” camera distribution allows for more flexibility and supports multiple object categories.
Revised: Beyond this the similarity ends, as our “Ballin-Sphere” camera distribution allows for more flexibility and supports multiple object categories.

Original: GANs with external knowledge.
Revised: GANs with external knowledge.

Original: Several works observed improved convergence and fidelity of
Revised: Several works observed improved convergence and fidelity of

Original: GANs when using existing, generic image-based models (Kumari et al., 2022; Sauer et al., 2021; 2022; Mo et al., 2020), the most notable being StyleGAN-XL (Sauer et al., 2022), which uses a pre-trained EfficientNet (Tan & Le, 2019) followed by a couple of discriminator layers.
Revised: GANs when using existing, generic image-based models (Kumari et al., 2022; Sauer et al., 2021; 2022; Mo et al., 2020), the most notable being StyleGAN-XL (Sauer et al., 2022), which uses a pre-trained EfficientNet (Tan & Le, 2019) followed by a couple of discriminator layers.

Original: A similar technique is not suitable in our case as pre-training a generic RGB-D network on a large scale RGBD dataset is problematic due to the lack of data.
Revised: A similar technique is not suitable in our case as pre-training a generic RGB-D network on a large scale RGBD dataset is problematic due to the lack of data.

Original: Another notable example is FreezeD Mo et al.
Revised: Another notable example is FreezeD Mo et al.

Original: (2020), which proposes to distill discriminator features for GAN finetuning.
Revised: (2020), which proposes to distill discriminator features for GAN finetuning.

Original: Our work, on the other hand, rely on existing model for image classification.
Revised: Our work, on the other hand, rely on existing model for image classification.

Original: Off-the-shelf depth guidance .
Revised: Off-the-shelf depth guidance .

Original: GSN (DeVries et al., 2021) also concatenates depth maps as the 4-th channel of the discriminator’s input, but they utilize ground truth depths, which are not available for large-scale datasets.
Revised: GSN (DeVries et al., 2021) also concatenates depth maps as the 4-th channel of the discriminator’s input, but they utilize ground truth depths, which are not available for large-scale datasets.

Original: DepthGAN (Shi et al., 2022) uses predictions from a depth estimator to guide training of a 2D GAN.
Revised: DepthGAN (Shi et al., 2022) uses predictions from a depth estimator to guide training of a 2D GAN.

Original: Exploiting monocular depth estimators for improving neural rendering was also explored in concurrent work (Yu et al., 2022), however their goal is just geometry reconstruction.
Revised: Exploiting monocular depth estimators for improving neural rendering was also explored in concurrent work (Yu et al., 2022), however their goal is just geometry reconstruction.

Original: The core characteristic of our approach is taking the depth estimator imprecision into account by training a depth adaptor module to mitigate it (see §3.2).
Revised: The core characteristic of our approach is taking the depth estimator imprecision into account by training a depth adaptor module to mitigate it (see §3.2).

Original:  M ETHOD
Revised:  M ETHOD

Original: We build our generator on top of EpiGRAF (Skorokhodov et al., 2022) since its fast to train, achieves reasonable image quality and does not need a 2D upsampler, relying on patch-wise training instead (Schwarz et al., 2020).
Revised: We build our generator on top of EpiGRAF (Skorokhodov et al., 2022) since its fast to train, achieves reasonable image quality and does not need a 2D upsampler, relying on patch-wise training instead (Schwarz et al., 2020).

Original: Given a random latent code z , our generator G produces a tri-plane representation for the scene.
Revised: Given a random latent code z , our generator G produces a tri-plane representation for the scene.

Original: Then, a shallow 2-layer MLP predicts RGB color and density σ values from an interpolated feature vector at a 3D coordinate.
Revised: Then, a shallow 2-layer MLP predicts RGB color and density σ values from an interpolated feature vector at a 3D coordinate.

Original: Then images and depths are volumetrically rendered (Mildenhall et al., 2020) at any given c¸amera position.
Revised: Then images and depths are volumetrically rendered (Mildenhall et al., 2020) at any given c¸amera position.

Original: Differently to prior works (Chan et al., 2021; Niemeyer & Geiger, 2021b) that utilize fixed camera distribution, we sample camera from a trainable camera generator C (see §3.1).
Revised: Differently to prior works (Chan et al., 2021; Niemeyer & Geiger, 2021b) that utilize fixed camera distribution, we sample camera from a trainable camera generator C (see §3.1).

Original: We render depth and process it via the depth adaptor (see §3.2), bridging the domains of rendered and estimated depth maps (Miangoleh et al., 2021).
Revised: We render depth and process it via the depth adaptor (see §3.2), bridging the domains of rendered and estimated depth maps (Miangoleh et al., 2021).

Original: Our discriminator D follows the architecture of StyleGAN2 (Karras et al., 2020a), additionally taking either adapted or estimated depth as the 4-th channel.
Revised: Our discriminator D follows the architecture of StyleGAN2 (Karras et al., 2020a), additionally taking either adapted or estimated depth as the 4-th channel.

Original: To further improve the image fidelity, we propose a simple knowledge distillation technique, that enriches D with external knowledge obtained from ResNet (He et al., 2016)
Revised: To further improve the image fidelity, we propose a simple knowledge distillation technique, that enriches D with external knowledge obtained from ResNet (He et al., 2016)

Original: (see §3.3).
Revised: (see §3.3).

Original: The overall model architecture is shown in Fig.
Revised: The overall model architecture is shown in Fig.

Original: 3. L EARNABLE “B ALL - IN -S PHERE ” CAMERA DISTRIBUTION
Revised: 3. L EARNABLE “B ALL - IN -S PHERE ” CAMERA DISTRIBUTION

Original: Limitations of Existing Camera Parameterization.
Revised: Limitations of Existing Camera Parameterization.

Original: The camera parameterization of existing 3D generators follows an overly simplified distribution — its position is sampled on a fixed-radius sphere with fixed intrinsics, and the camera always points to (0 , 0 , 0) .
Revised: The camera parameterization of existing 3D generators follows an overly simplified distribution — its position is sampled on a fixed-radius sphere with fixed intrinsics, and the camera always points to (0 , 0 , 0) .

Original: This parametrization has only two degrees of freedom: pitch and yaw ( φ pos in Fig.
Revised: This parametrization has only two degrees of freedom: pitch and yaw ( φ pos in Fig.

Original: 3 (a)), implicitly assuming that all the objects could be centered, rotated and scaled with respect to some canonical alignment.
Revised: 3 (a)), implicitly assuming that all the objects could be centered, rotated and scaled with respect to some canonical alignment.

Original: However, natural in-the-wild 3D scenes are inherently non-alignable: they could consist of multiple objects, objects might have drastically different shapes and articulation, or they could even be represented only as volumes (like smoke).
Revised: However, natural in-the-wild 3D scenes are inherently non-alignable: they could consist of multiple objects, objects might have drastically different shapes and articulation, or they could even be represented only as volumes (like smoke).

Original: This makes the traditional camera conventions ill-posed for such data.
Revised: This makes the traditional camera conventions ill-posed for such data.

Original: Learnable “Ball-in-Sphere” Camera Distribution.
Revised: Learnable “Ball-in-Sphere” Camera Distribution.

Original: We introduce a new camera parametrization which we call “Ball-in-Sphere”.
Revised: We introduce a new camera parametrization which we call “Ball-in-Sphere”.

Original: Contrary to the standard parametrization, it has four additional degrees of freedom: the field of view φ fov , and pitch, yaw and radius of the inner sphere, specifying the look-at point within the outer sphere ( φ lookat in Fig.
Revised: Contrary to the standard parametrization, it has four additional degrees of freedom: the field of view φ fov , and pitch, yaw and radius of the inner sphere, specifying the look-at point within the outer sphere ( φ lookat in Fig.

Original: 3 (b)).
Revised: 3 (b)).

Original: Combining with the standard parameters on the outer sphere, our camera parametrization has six degrees of freedom φ =
Revised: Combining with the standard parameters on the outer sphere, our camera parametrization has six degrees of freedom φ =

Original: [ φ pos ∥ φ fov ∥
Revised: [ φ pos ∥ φ fov ∥

Original: φ lookat ] , where ∥ denotes concatenation.
Revised: φ lookat ] , where ∥ denotes concatenation.

Original: Note, that compared with the standard camera, ours allows us to learn the scale of objects and scenes, enabling zooming into different parts of the scene as shown in Fig.
Revised: Note, that compared with the standard camera, ours allows us to learn the scale of objects and scenes, enabling zooming into different parts of the scene as shown in Fig.

Original: 3 (b).
Revised: 3 (b).

Original: Instead of manually defining camera distributions, we learn the camera distribution during training for each dataset.
Revised: Instead of manually defining camera distributions, we learn the camera distribution during training for each dataset.

Original: In particular, we train the camera generator network C that takes camera parameters sampled from a sufficiently wide camera prior φ ′ and produces new camera parameters φ .
Revised: In particular, we train the camera generator network C that takes camera parameters sampled from a sufficiently wide camera prior φ ′ and produces new camera parameters φ .

Original: For a class conditional dataset, such as ImageNet where scenes have significantly different geometry, we additionally condition this network on the class label c and the latent code z , i.e. φ = C ( φ ′ , z , c ) .
Revised: For a class conditional dataset, such as ImageNet where scenes have significantly different geometry, we additionally condition this network on the class label c and the latent code z , i.e. φ = C ( φ ′ , z , c ) .

Original: For a single category dataset we use φ = C
Revised: For a single category dataset we use φ = C

Original: ( φ ′ , z ) .
Revised: ( φ ′ , z ) .

Original: [Figure]
Revised: [Figure]

Original: Camera Gradient Penalty.
Revised: Camera Gradient Penalty.

Original: To the best of our knowledge, CAMPARI (Niemeyer & Geiger, 2021a) is the only work which also learns the camera distribution.
Revised: To the best of our knowledge, CAMPARI (Niemeyer & Geiger, 2021a) is the only work which also learns the camera distribution.

Original: It samples a set of camera parameters from a wide distribution and passes it to a shallow neural network, which produces a residual ∆ φ = φ − φ ′ for these parameters.
Revised: It samples a set of camera parameters from a wide distribution and passes it to a shallow neural network, which produces a residual ∆ φ = φ − φ ′ for these parameters.

Original: However, we observed that such a regularization is too weak for the complex datasets we explore in this work, and leads to the distribution collapse (see Fig. 7).
Revised: However, we observed that such a regularization is too weak for the complex datasets we explore in this work, and leads to the distribution collapse (see Fig. 7).

Original: Note, that a similar observation was also made by Gu et al.
Revised: Note, that a similar observation was also made by Gu et al.

Original: To prevent the camera generator from producing collapsed camera parameters, we seek a new regularization strategy.
Revised: To prevent the camera generator from producing collapsed camera parameters, we seek a new regularization strategy.

Original: Ideally, it should prevent constant solutions, while at the same time, reducing the Lipschitz constant for C , which is shown to be crucial for stable training of generators (Odena et al., 2018).
Revised: Ideally, it should prevent constant solutions, while at the same time, reducing the Lipschitz constant for C , which is shown to be crucial for stable training of generators (Odena et al., 2018).

Original: We can achieve both if we push the derivatives of the predicted camera parameters with respect to the prior camera parameters to either one or minus one, n o r m a li z e
Revised: We can achieve both if we push the derivatives of the predicted camera parameters with respect to the prior camera parameters to either one or minus one, n o r m a li z e

Original: C o n v 2 d C o n v 2 d
Revised: C o n v 2 d C o n v 2 d

Original: C o n v 2 d
Revised: C o n v 2 d

Original: Conv2d
Revised: Conv2d

Original: Select random
Revised: Select random

Original: RGB
Revised: RGB

Original: Real image
Revised: Real image

Original: Estimated depth from LeReS d ¯ d d a d (1) d (2) d (3) d ¯ d d (1) d (2) d (3) left eye right eye legs tail arriving at the following regularization term:
Revised: Estimated depth from LeReS d ¯ d d a d (1) d (2) d (3) d ¯ d d (1) d (2) d (3) left eye [Figure] tail arriving at the following regularization term:

Original: [Equation]
Revised: [Equation]

Original: where φ ′ i ∈
Revised: where φ ′ i ∈

Original: φ ′ is the camera sampled from the prior distribution and φ i ∈ φ is produced by the camera generator.
Revised: φ ′ is the camera sampled from the prior distribution and φ i ∈ φ is produced by the camera generator.

Original: We refer to this loss as Camera Gradient Penalty.
Revised: We refer to this loss as Camera Gradient Penalty.

Original: Note, that the first partof the loss prevents rapid changes in camera, facilitating stable optimization, while the second part avoids collapsedcamera posteriors.
Revised: Its first part prevents rapid camera changes, facilitating stable optimization, while the second part avoids collapsed posteriors.

Original: 3. A DVERSARIAL DEPTH SUPERVISION
Revised: 3. A DVERSARIAL DEPTH SUPERVISION

Original: To instill a 3D bias into our model, we develop a strategy of using depth maps predicted by an off-the-shelf estimator E (Miangoleh et al., 2021), for its advantages of being generic and readily applicable for many object categories.
Revised: To instill a 3D bias into our model, we develop a strategy of using depth maps predicted by an off-the-shelf estimator E (Miangoleh et al., 2021), for its advantages of being generic and readily applicable for many object categories.

Original: The main idea is concatenating the depth map as a 4-th channel of the RGB as the input of the discriminator.
Revised: The main idea is concatenating a depth map as the 4-th channel of the RGB as the input of the discriminator.

Original: The fake depth maps in this case is obtained with the help of neural rendering, while the real depth maps are estimated using monocular depth estimator E .
Revised: The fake depth maps in this case is obtained with the help of neural rendering, while the real depth maps are estimated using monocular depth estimator E .

Original: However, naively utilizing the depth from E leads to training divergence.
Revised: However, naively utilizing the depth from E leads to training divergence.

Original: This happens because E could only produce relative depth, not metric depth.
Revised: This happens because E could only produce relative depth, not metric depth.

Original: Moreover, monocular depth estimators are still not perfect, they produce noisy artifacts, ignore high-frequency details, and make prediction mistakes.
Revised: Moreover, monocular depth estimators are still not perfect, they produce noisy artifacts, ignore high-frequency details, and make prediction mistakes.

Original: Thus, we devise a mechanism that allows utilization of the imperfect depth maps.
Revised: Thus, we devise a mechanism that allows utilization of the imperfect depth maps.

Original: The central part of this mechanism is a learnable depth adaptor A , that should transform and augment the depth map obtained with neural rendering to look like a depth map from E .
Revised: The central part of this mechanism is a learnable depth adaptor A , that should transform and augment the depth map obtained with neural rendering to look like a depth map from E .

Original: More specifically, we first render raw depths d from NeRF via volumetric rendering:
Revised: More specifically, we first render raw depths d from NeRF via volumetric rendering:

Original: [Equation]
Revised: [Equation]

Original: where t n , t f ∈ R are near/far planes, T ( t ) is accumulated transmittance, and r ( t ) is a ray.
Revised: where t n , t f ∈ R are near/far planes, T ( t ) is accumulated transmittance, and r ( t ) is a ray.

Original: Raw depth is shifted and scaled from the range of [ t n , t f ] into [ − 1 , 1] to obtain normalized depth ¯ d :
Revised: Raw depth is shifted and scaled from the range of [ t n , t f ] into [ − 1 , 1] to obtain normalized depth ¯ d :

Original: [Equation]
Revised: [Equation]

Original: where b ∈ [0 , ( t n + t f ) / 2] is an additional learnable shift needed to account for the empty space in the front of the camera.
Revised: where b ∈ [0 , ( t n + t f ) / 2] is an additional learnable shift needed to account for the empty space in the front of the camera.

Original: Note that depth values obtained using E span over the entire valid range of [0 , 2 16 ] , thus empty space does not appear in them.

Original: Therefore, we map them into the [ − 1 , 1] range to get depth d r for real input.
Revised: Real depths are normalized into the [ − 1 , 1] range directly.

Original: Learnable Depth Adaptor.
Revised: Learnable Depth Adaptor.

Original: Although ¯ d is distributed in the same range as d r , ¯ d isstill not suitable for discriminator D . The reason is that ¯ d forces G to learn the prediction artifacts from E and creates an additional confusion for G , given E provides only relative depth.
Revised: While ¯ d has the same range as d r , it is not suitable for adversarial supervision directly due it the imprecision of E : G would be trained to simulate all its prediction artifacts.

Original: Therefore, objects in the different images with the same metric depth may still have different relative depth.

Original: To overcome such issue, we introduce the depth adaptor A to produce an adapted depth map d a =
Revised: To overcome this issue, we introduce a depth adaptor A to adapt the depth map d a =

Original: A 
Revised: A ( ¯ d ) ∈ R h × w , where h × w is a number of sampled pixels. This depth (fake d a or real d r ) is concatenated with the RGB input and passed to D . The depth adaptor A models artifacts produced by E , so that the discriminator should focus only on the relevant high level geometry.

Original: ( ¯ d ) ∈ R h × w ,
Revised: However, a too powerful A would be able to fake the depth completely, and G will not learn the geometry. This is why we structure A as just a 3-layer convolutional network (see Fig. 4). Each layer produces a separated depth map with different levels

Original: [Figure]
Revised: [Figure]

Original: I
Revised: S

Original: [Figure]
Revised: [Figure]

Original: S D I P
Revised: I P

Revised: [Figure]

Original: E l e ph a n t
Revised: l [Figure] a n t

Original: L S U N
Revised: L S U N

Original: H
Revised: H

Original: o r s e s
Revised: o r s e s

Original: [Figure]
Revised: [Figure]

Original: k

Original: [Figure] where h × w is a number of sampled pixels. The depth (fake d a or real d r ) is concatenated with the RGB input and passed to D .
Revised: of adaptation

Original: The depth adaptor A models artifacts produced by E , so that the discriminator should focus only on the relevant high level geometry. However, if A is too powerful, it could fake the depth completely, and the generator G will not receive any meaningful signal. To this end, we utilize a depth adaptor that is based on 3-layer convolutional network (shown in Fig 4). After each layer, we produce a separated depth map with different levels of adaptation: d (1) a , d (2) a and d (3) a .
Revised: : d (1) a , d (2) a and d (3) a .

Original: The  adapted depth d a is randomly selected from the set of (cid:110) ¯ d , d (1) a , d (2) a , d (3) a (cid:111) .
Revised: The final adapted depth d a is randomly selected from the set of { ¯ d , d (1) a , d (2) a , d (3) a } .

Original: Such design can effectively learn good geometry while alleviating overfitting.
Revised: Such design can effectively learn good geometry while alleviating overfitting.

Original: For example, when  the generator provides ¯ d to the discriminator, it receives a strong signal for learning the geometry.
Revised: For example, when D receives the original depth map ¯ d as input, it provides to G a strong signal for learning the geometry.

Original: On the other hand, if the discriminator sees highly adapted depth  d (3) a, it unlikely overfits to different unrelated estimation artifacts. Finally, d (1) a and d (2) a provide some trade-off between geometry learning and precise adaptation.
Revised: And passing an adapted depth map d ( i ) a to D allows G to simulate the imprecision artifacts of the depth estimator without degrading its original depth map ¯ d .

Original: 3. K NOWLEDGE DISTILLATION FOR D ISCRIMINATOR
Revised: 3. K NOWLEDGE DISTILLATION FOR D ISCRIMINATOR

Original: Knowledge from pretrained classification networks was shown to improve training stability and generation quality in 2D GANs (Sauer et al., 2021; Kumari et al., 2022; Sauer et al., 2022; Casanova et al., 2021).
Revised: Knowledge from pretrained classification networks was shown to improve training stability and generation quality in 2D GANs (Sauer et al., 2021; Kumari et al., 2022; Sauer et al., 2022; Casanova et al., 2021).

Original: A popular solution proposed by Sauer et al.
Revised: A popular solution proposed by Sauer et al.

Original: (2021; 2022) is to use an off-the-shelf model as a discriminator while freezing most of its weights.
Revised: (2021; 2022) is to use an off-the-shelf model as a discriminator while freezing most of its weights.

Original: Unfortunately, this technique is not applicable in our scenario since we modify the architecture of the discriminator by adding an additional depth input (see §3.2) and condition on the parameters of the patch similarly to EpiGRAF (Skorokhodov et al., 2022).
Revised: Unfortunately, this technique is not applicable in our scenario since we modify the architecture of the discriminator by adding an additional depth input (see §3.2) and condition on the parameters of the patch similarly to EpiGRAF (Skorokhodov et al., 2022).

Original: Thus, we devise an alternative technique that can work with arbitrary architectures of the discriminator.
Revised: Thus, we devise an alternative technique that can work with arbitrary architectures of the discriminator.

Original: Specifically, for each real sample, we obtain two feature representations: e from the pretrained ResNet (He et al., 2016) network and ˆ e extracted from the final representation of our discriminator D .
Revised: Specifically, for each real sample, we obtain two feature representations: e from the pretrained ResNet (He et al., 2016) network and ˆ e extracted from the final representation of our discriminator D .

Original: Our loss simply pushes ˆ e to e as follows:
Revised: Our loss simply pushes ˆ e to e as follows:

Original: [Equation]
Revised: [Equation]

Original: L dist can effectively distill knowledge from the pretrained ResNet (He et al., 2016) into our D .
Revised: L dist can effectively distill knowledge from the pretrained ResNet (He et al., 2016) into our D .

Original: 3. T RAINING
Revised: 3. T RAINING

Original: Our overall loss for generator G consists of two parts: adversarial loss and Camera Gradient Penalty for each camera parameter:
Revised: The overall loss for generator G consists of two parts: adversarial loss and Camera Gradient Penalty:

Original: [Equation]
Revised: [Equation]

Original: where L adv is the non-saturating loss (Goodfellow et al., 2014).
Revised: where L adv is the non-saturating loss (Goodfellow et al., 2014).

Original: We observe that a diverse distribution for camera origin is most important for leaning meaningful geometry, but it is also most prone to degrade to a constant solution.
Revised: We observe that a diverse distribution for camera origin is most important for leaning meaningful geometry, but it is also most prone to degrade to a constant solution.

Original: Therefore, we set λ φ i = 0 .
Revised: Therefore, we set λ φ i = 0 .

Original: 3 for φ pos , while set λ φ i = 0 .
Revised: 3 for φ pos , while set λ φ i = 0 .

Original: 03 for φ fov and λ φ i = 0 .
Revised: 03 for φ fov and λ φ i = 0 .

Original: 003 for φ lookat .
Revised: 003 for φ lookat .

Original: The loss for discriminator D , on the other hand, consists of three parts: adversarial loss, knowledge distillation, and R 1 gradient penalty (Mescheder et al., 2018):
Revised: The loss for discriminator D , on the other hand, consists of three parts: adversarial loss, knowledge distillation, and R 1 gradient penalty (Mescheder et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: We use the same optimizer and hyper-parameters as EpiGRAF.
Revised: We use the same optimizer and hyper-parameters as EpiGRAF.

Original: We observe that for depth adaptor, sampling adapted depth maps with equal probability is not always beneficial found that using
Revised: We observe that for depth adaptor, sampling adapted depth maps with equal probability is not always beneficial, and found that using

Original: P ( ¯ d ) = 0 .
Revised: P ( ¯ d ) = 0 .

Original: 5 leads to better geometry.
Revised: 5 leads to better geometry.

Original: For additional details, see Appx B.
Revised: For additional details, see Appx B.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: Datasets.
Revised: Datasets.

Original: In our experiments, we use 4 non-aligned datasets: SDIP Dogs (Mokady et al., 2022),
Revised: In our experiments, we use 4 non-aligned datasets: SDIP Dogs (Mokady et al., 2022),

Original: SDIP Elephants (Mokady et al., 2022), LSUN Horses (Yu et al., 2015), and ImageNet (Deng et al., 2009).
Revised: SDIP Elephants (Mokady et al., 2022), LSUN Horses (Yu et al., 2015), and ImageNet (Deng et al., 2009).

Original: The first three are single-category datasets that contain objects with complex articulated geometry, making them challenging for standard 3D generators.
Revised: The first three are single-category datasets that contain objects with complex articulated geometry, making them challenging for standard 3D generators.

Original: We found it useful to remove outlier images from SDIP Dogs and LSUN Horseswith the instance selectiontechnique (DeVries et al., 2020), reducing their size to 40 K samples each.
Revised: We found it useful to remove outlier images from SDIP Dogs and LSUN Horsesvia instance selection (DeVries et al., 2020), reducing their size to 40 K samples each.

Original: We refer to the filtered versions of these datasets as SDIP Dogs 40k , and LSUN Horses 40k , respectively.
Revised: We refer to the filtered versions of these datasets as SDIP Dogs 40k , and LSUN Horses 40k , respectively.

Original: We then validate our method on ImageNet, a real-world, multi-category dataset containing 1 , 000 diverse object classes, with more than 1 ,images per category.
Revised: We then validate our method on ImageNet, a real-world, multicategory dataset containing 1 , 000 diverse object classes, with more than 1 , 000 images per category.

Original: To train all our 3D generatorson ImageNet (including the baselines), we use the same filtering strategy for it, keeping 1/3 of its images (note that we always measure the metrics for it on the full ImageNet).
Revised: All the 3D generators (including the baselines) use the same filtering strategy for ImageNet, where 2/3 of its images are filtered out. Note that all the metrics are always measured on the full ImageNet.

Original: [Figure]
Revised: [Figure]

Original: Evaluation.
Revised: Evaluation.

Original: We rely on FID (Heusel et al., 2017) to measure the image quality and FID 2k , which is computed on 2,048 images instead of 50k (as for FID) for efficiency.
Revised: We rely on FID (Heusel et al., 2017) to measure the image quality and FID 2k , which is computed on 2,048 images instead of 50k (as for FID) for efficiency.

Original: For ImageNet, we additionally compute Inception Score (IS) (Salimans et al., 2016).
Revised: For ImageNet, we additionally compute Inception Score (IS) (Salimans et al., 2016).

Original: Note that while we train on the filtered ImageNet, we always compute the metrics on the full one.
Revised: Note that while we train on the filtered ImageNet, we always compute the metrics on the full one.

Original: There is no established protocol to evaluate the geometry quality of 3D generators in general, but the state-of-the-art ones are tri-plane (Chan et al., 2022;
Revised: There is no established protocol to evaluate the geometry quality of 3D generators in general, but the state-of-the-art ones are tri-plane (Chan et al., 2022;

Original: Sun et al., 2022) or MPI-based Zhao et al.
Revised: Sun et al., 2022) or MPI-based Zhao et al.

Original: , and we observed that it is possible to quantify their most popular geometry failure case: flatness of the shapes.
Revised: , and we observed that it is possible to quantify their most popular geometry failure case: flatness of the shapes.

Original: For this, we propose Non-Flatness Score (NFS) which is computed as the average entropy of the normalized depth maps histograms.
Revised: For this, we propose Non-Flatness Score (NFS) which is computed as the average entropy of the normalized depth maps histograms.

Original: We depict its intuition in Fig 6 and provide the details in Appx F.
Revised: We depict its intuition in Fig 6 and provide the details in Appx F.

Original: 4. 3D GENERATION FOR SINGLE CATEGORY DATASETS
Revised: 4. 3D GENERATION FOR SINGLE CATEGORY DATASETS

Original: We show quantitative results for single category datasets in Tab.
Revised: We show quantitative results for single category datasets in Tab.

Original: 1a, where we compare with
Revised: 1a, where we compare with

Original: EG3D (Chan et al., 2022) and EpiGRAF (Skorokhodov et al., 2022).
Revised: EG3D (Chan et al., 2022) and EpiGRAF (Skorokhodov et al., 2022).

Original: EG3D was mainly designed for FFHQ (Karras et al., 2019) and uses the true camera poses inferred from real images as the camera distribution for the generator.
Revised: EG3D was mainly designed for FFHQ (Karras et al., 2019) and uses the true camera poses inferred from real images as the camera distribution for the generator.

Original: But in our scenario, we do not have any knowledge about the true camera distribution, that is why to stay as close as possible to the setup EG3D was designed for, we use normal distribution for rotation and elevation angles with the same standard deviations as in FFHQ, which are equal to σ yaw = 0 .
Revised: But in our scenario, we do not have any knowledge about the true camera distribution, that is why to stay as close as possible to the setup EG3D was designed for, we use normal distribution for rotation and elevation angles with the same standard deviations as in FFHQ, which are equal to σ yaw = 0 .

Original: 3 and σ pitch
Revised: 3 and σ pitch

Original: = 0 .
Revised: = 0 .

Original: 155 , respectively.
Revised: 155 , respectively.

Original: Also, tomake it learn better[Table] + ADS, P ( ¯ d ) = 1 .12. geometry, we  trained it with a twice as wide camera distribution: σ yaw = 0 .
Revised: Also, to learn better geometry, we additionally trained the baselines with a twice wider camera distribution: σ yaw = 0 .and

Original: 6 and σ pitch = 0 .
Revised: σ pitch = 0 .

Original: While it indeed help to improve the flatness of the shapes, it alsoconsiderably worsened the image quality, from 20% to to 500% as measured by FID 2k .
Revised: While it indeed helped to reduce flatness, it also worsened the image quality: up to 500% as measured by FID 2k .

Original: For EpiGRAF, we could train it only with the default camera distribution of FFHQ and observed that it diverges with a wider one.

Original: Our model shows substantially better (at least 2 × than EG3D, slightly worse than StyleGAN2) FID 2k and greater NFS on all the datasets.
Revised: Our model shows substantially better (at least 2 × than EG3D, slightly worse than StyleGAN2) FID 2k and greater NFS on all the datasets.

Original: Low NFS indicates flat or repetitive geometry impairing the ability of the model to generate realistic side views.
Revised: Low NFS indicates flat or repetitive geometry impairing the ability of the model to generate realistic side views.

Original: Indeed, as shown in Fig.5 (left), both EG3D (with the default camera distribution) and EpiGRAF struggle to generate side views, while our method (3DGP) renders realistic side views on all three datasets.
Revised: Indeed, as shown in Fig.

Original: Adversarial Depth Supervision.
Revised: Adversarial Depth Supervision.

Original: We evaluate the proposed Adversarial Depth Supervision (ADS) and the depth adaptor A .
Revised: We evaluate the proposed Adversarial Depth Supervision (ADS) and the depth adaptor A .

Original: The only hyperparameter it has is the probability of using the non-adapted depth P ( ¯ d ) (see §3.2).
Revised: The only hyperparameter it has is the probability of using the non-adapted depth P ( ¯ d ) (see §3.2).

Original: We ablate this parameter in Tab. 1b.
Revised: We ablate this parameter in Tab. 1b.

Original: While FID scores are only slightly affected by varying P ( ¯ d ) , we see substantial difference in the Non-Flatness Score.
Revised: While FID scores are only slightly affected by varying P ( ¯ d ) , we see substantial difference in the Non-Flatness Score.

Original: We first verify that NFS is the worst without ADS, indicating the lack of a 3D bias.
Revised: We first verify that NFS is the worst without ADS, indicating the lack of a 3D bias.

Original: When P ( ¯ d ) = 0 , the discriminator is never presented with the rendered depth ¯ d , while the adaptor learns to fake the depth, leading to flat geometry.
Revised: When P ( ¯ d ) = 0 , the discriminator is never presented with the rendered depth ¯ d , while the adaptor learns to fake the depth, [Table] + ADS, P ( ¯ d ) = 1 .12. p r

Revised: i o r g r a d p e n a l t y n o r e g r e

Original:  When P ( ¯ d ) = 1 , the adaptor is never used, allowing the discriminator to easily determine the generated depth from the estimated depth, as there is a large domain gap (see Fig. 4), leading to reduced FID scores.
Revised: s i du a l yaw pitch fov look-at yaw look-at pitch look-at radius yaw=0 ◦ yaw=30 ◦ yaw=60 ◦ leading to flat geometry. When P ( ¯ d ) = 1 , the adaptor is never used, allowing D to easily determine the generated depth from the estimated depth, as there is a large domain gap (see Fig. 4), leading to reduced FID scores.

Original: We observe that the best results are obtained when P ( ¯ d ) = 0 .
Revised: The best results are achieved with P ( ¯ d ) = 0 .

Original: The same observations can be visually verified in Fig.
Revised: 5 , which can be visually verified from observing Fig.

Original: 5 [-(middle), where the Bare 3DGP, P ( ¯ d ) = 
 and-]
Revised: 5 [-(middle), where the Bare 3DGP, P ( ¯ d ) = 
 and-](middle).

Original:  P ( ¯ d ) = 1 are unable to render side views, while the model trained with P ( ¯ d ) = 0 .
Revised: A “bare” 3DGP, P ( ¯ d ) = 0 and P ( ¯ d ) = 1 are unable to render side views, while the model trained with P ( ¯ d ) = 0 .

Original: 5 features the overall best geometry and side views.
Revised: 5 features the overall best geometry and side views.

Original: Knowledge Distillation.
Revised: Knowledge Distillation.

Original: Here we further provide insights for discriminator D using our knowledge distillation strategy (see §3.3).
Revised: Here we further provide insights for discriminator D using our knowledge distillation strategy (see §3.3).

Original: We find that knowledge distillation provides an additional stability for adversarial training, along with the significant improvement in FID, which can be observed by comparing results of EpiGRAF in Tab.
Revised: We find that knowledge distillation provides an additional stability for adversarial training, along with the significant improvement in FID, which can be observed by comparing results of EpiGRAF in Tab.

Original: 1a and Bare 3DGP Tab. 1a.
Revised: 1a and Bare 3DGP Tab. 1a.

Original: Additionally, we compare different knowledge distillation strategies in Appx C.
Revised: Additionally, we compare different knowledge distillation strategies in Appx C.

Original: However, as noted by Parmar et al.
Revised: However, as noted by Parmar et al.

Original: (2021), strategies which utilize additional classification networks, may provide a significant boost to FID, without corresponding improvement in visual quality.
Revised: (2021), strategies which utilize additional classification networks, may provide a significant boost to FID, without corresponding improvement in visual quality.

Original: “Ball-in-Sphere” Camera Distribution.
Revised: “Ball-in-Sphere” Camera Distribution.

Original: In Fig.
Revised: In Fig.

Original: 7, we analyze different strategies for learning the camera distribution: sampling the camera φ from the prior φ ′ without learning, predicting residuals φ − φ ′ , and using our proposed camera generator C with Camera Gradient Penalty.
Revised: 7, we analyze different strategies for learning the camera distribution: sampling the camera φ from the prior φ ′ without learning, predicting residuals φ − φ ′ , and using our proposed camera generator C with Camera Gradient Penalty.

Original: For the first two cases, the learned distributions are highly concentrated around constant values.
Revised: For the first two cases, the learned distributions are highly concentrated around constant values.

Original: Not surprisingly, no meaningful side views can be generated as geometry becomes flat.
Revised: Not surprisingly, no meaningful side views can be generated as geometry becomes flat.

Original: In contrast, our Camera
Revised: In contrast, our Camera

Original:  Gradient Penalty provides sufficient regularization to C , enabling it to converge to a sufficiently wide posterior, resulting into valid geometry and realistic side views. p r i o r g r a d p
Revised: [Table] Gradient Penalty provides sufficient regularization to C , enabling it to converge to a sufficiently wide posterior, resulting into valid geometry and realistic side views

Original: e n a l t y n o r e g

Original: r

Original: e

Original: s i du a l yaw pitch

Original: fov look-at yaw look-at pitch look-at radius yaw=0 ◦

Original: yaw=30 ◦ yaw=60 ◦

Original: [Table]
Revised: [Table]

Original: 4. 3D SYNTHESIS ON I MAGE N ET
Revised: 4. 3D SYNTHESIS ON I MAGE N ET

Original: ImageNet (Deng et al., 2009) is significantly more difficult than single category datasets.
Revised: ImageNet (Deng et al., 2009) is significantly more difficult than single category datasets.

Original: Following prior work [Table] we use class labels c as an additional guidance for the generator (Sauer et al., 2022; Brock et al., 2018).
Revised: Following prior works, we trained all the methods in the conditional generation setting Brock et al.

Original: We modify the baselines accordingly.

Original: The quantitative resultsof ImageNet experiments are presented in Tab. 2.
Revised: The quantitative results are presented in Tab. 2.

Original: For the reference, we also report  results of  2D generators, such asGAN-based methods of BigGAN (Brock et al., 2018) and StyleGAN-XL (Sauer et al., 2022) and a diffusion-based approach of ADM (Dhariwal & Nichol, 2021). As expected, 2D generators show superior FID and IS, compared to the 3D counterparts as the latter use their capacity to learn geometric details.
Revised: We also report the results of state-of-the-art 2D generators for reference: as expected, they show much FID and IS than 3D generators, since they do not need to learn the geometry, are trained on larger compute and had been studied for much longer. Despite our best efforts to find a reasonable camera distribution, both EG3D and EpiGRAF produce flat or repetitive geometry, while 3DGP produces geometry with rich details (see Fig. 1). StyleNeRF (Gu et al., 2022) and VolumeGAN Xu et al.

Original: As discussed before, the “wideness” of  the camera distribution is an important factor to consider, hence, for fair comparisons, we report the standard camera ranges as well as wide ones. Despite ourbest efforts to find a reasonable camera distribution, both EG3D and EpiGRAF produce flat or repetitive geometry, confirming similar experiments on single category articulated datasets.
Revised: (2021), trained for conditional ImageNet generation with the narrow camera distribution, substantially under-performed in terms of visual quality. We hypothesize that the reason lies in their MLP/voxel-based NeRF backbones: they have better 3D prior than tri-planes, but are considerably more expensive, which, in its turn, requires sacrifices in terms of generator’s expressivity. Also, we found that increasing EMA half-life for our

Original: 3DGP produces geometry with rich details (see Fig. 1).
Revised: G improves its performance (see Tab. 2). Training a 3D generator from scratch is the only way to achieve 3D-aware synthesis: one can lift an existing 2D generator into 3D using the techniques from 3D photo “inpainting”. To test this idea, we generate 10k images with StyleGAN-XL (the current SotA on ImageNet) and then run 3DPhoto (Shih et al., 2020) on them.

Revised: This method first lifts a photo into 3D via a pre-trained depth estimator and then inpaints the holes with a separately trained GAN model to generate novel views.

Revised: This method works well for negligible camera variations, but starts diverging when the camera moves at least by 5 ◦ .

Revised: In Tab. 2, we report its FID/IS scores with the narrow camera distribution: σ yaw ∼ N (0 , 0 . 3) and σ pitch ∼ N

Revised: ( π/ 2 , 0 . 15) ).

Revised: The details are in Appx I.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we present the first 3D synthesis framework for in-the-wild, multi-category datasets, such as ImageNet.
Revised: In this work, we present the first 3D synthesis framework for in-the-wild, multi-category datasets, such as ImageNet.

Original: We demonstrate how to utilize generic priors in the form of (imprecise) monocular depth and latent feature representation to improve the visual quality and guide the geometry.
Revised: We demonstrate how to utilize generic priors in the form of ( imprecise ) monocular depth and latent feature representation to improve the visual quality and guide the geometry.

Original: Moreover, we propose the new “Ball-in-Sphere” camera model with a novel regularization scheme that enables learning meaningful camera distribution.
Revised: Moreover, we propose the new “Ball-in-Sphere” camera model with a novel regularization scheme that enables learning meaningful camera distribution.

Original: On the other hand, our work still has several limitations, such as sticking background, lower visual quality compared to 2D generators, and no reliable quantitative measure of generated geometry.
Revised: On the other hand, our work still has several limitations, such as sticking background, lower visual quality compared to 2D generators, and no reliable quantitative measure of generated geometry.

Original: Additional discussion is provided in Appx A.
Revised: Additional discussion is provided in Appx A.

Original: This project consumed ≈ 12 NVidia A100 GPU years in total.
Revised: This project consumed ≈ 12 NVidia A100 GPU years in total.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: Most importantly, we will release 1) the source code and the checkpoints of our generator as a separate github repo; and 2) fully pre-processed datasets used in our work (with the corresponding extracted depth maps).
Revised: Most importantly, we will release 1) the source code and the checkpoints of our generator as a separate github repo; and 2) fully pre-processed datasets used in our work (with the corresponding extracted depth maps).

Original: In §3 and §4 and Appx B, and also in the figures throughout the text, we provided a complete list of architecture and optimization details needed to reproduce our results.
Revised: In §3 and §4 and Appx B, and also in the figures throughout the text, we provided a complete list of architecture and optimization details needed to reproduce our results.

Original: In §5, we also We are also open to provide any further details on our work in public and/or private correspondence.
Revised: In §5, we also We are also open to provide any further details on our work in public and/or private correspondence.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: There are two main ethical concerns around the deep learning projects on synthesis: generation of fake content to mislead people (e.g. fake news or deepfakes 1 ) and potential licensing, authorship and distribution abuse, which recently got controversy in the research community in the context of 2D image generation (by Stable Diffusion Rombach et al.
Revised: There are two main ethical concerns around the deep learning projects on synthesis: generation of fake content to mislead people (e.g. fake news or deepfakes 2 ) and potential licensing, authorship and distribution abuse, which recently got controversy in the research community in the context of 2D image generation (by Stable Diffusion Rombach et al.

Original: and DALL-E Ramesh et al.
Revised: and DALL-E Ramesh et al.

Original: and code generation (by Github Copilot 2 ).
Revised: and code generation (by Github Copilot 3 ).

Original: While the subpar synthesis quality of our generator makes it safe from such concerns, we still warn the community to build on top of the developed ideas with caution, since they could strengthen a potentially harmful system.
Revised: While the subpar synthesis quality of our generator makes it safe from such concerns, we still warn the community to build on top of the developed ideas with caution, since they could strengthen a potentially harmful system.

Original: A L IMITATIONS
Revised: A L IMITATIONS

Original: Lower visual quality compared to 2D generators .
Revised: Lower visual quality compared to 2D generators .

Original: Despite providing a more reasonable representation of the underlining scene, 3D generators still have a lower visual quality compared to 2D generators.
Revised: Despite providing a more reasonable representation of the underlining scene, 3D generators still have a lower visual quality compared to 2D generators.

Original: Closing this gap is essential for a wide adaptation of 3D generators.
Revised: Closing this gap is essential for a wide adaptation of 3D generators.

Original: Background sticking .
Revised: Background sticking .

Original: One common 3D artifact of 3DGP is gluing of the foreground and the background.
Revised: One common 3D artifact of 3DGP is gluing of the foreground and the background.

Original: In other words our model predicts a single shape for both and there is no clear separation between the two.
Revised: In other words our model predicts a single shape for both and there is no clear separation between the two.

Original: One potential cause of this artifact is the dataset bias, where most of the photos are frontal, thus it is not beneficial for the model to explore backward views.
Revised: One potential cause of this artifact is the dataset bias, where most of the photos are frontal, thus it is not beneficial for the model to explore backward views.

Original: Another reason is the bias of tri-planes toward flat geometry.
Revised: Another reason is the bias of tri-planes toward flat geometry.

Original: However, all our attempts to replace tri-planes with an MLPbased NeRF led to much worse results (see Appx D).
Revised: However, all our attempts to replace tri-planes with an MLPbased NeRF led to much worse results (see Appx D).

Original: Inventing a different efficient parametrization may be an important direction.
Revised: Inventing a different efficient parametrization may be an important direction.

Original: No reliable quantitative measure of generated geometry .
Revised: No reliable quantitative measure of generated geometry .

Original: In this work we introduce Non-Flatness
Revised: In this work we introduce Non-Flatness

Original: Score as a proxy metric for evaluating the quality of the underlying geometry.
Revised: Score as a proxy metric for evaluating the quality of the underlying geometry.

Original: However it can capture only a single failure case, specific for generators based on tri-planes: flatness of geometry.
Revised: However it can capture only a single failure case, specific for generators based on tri-planes: flatness of geometry.

Original: Devising a reasonable metric applicable to a variety of scenarios could significantly speed up the progress in this area.
Revised: Devising a reasonable metric applicable to a variety of scenarios could significantly speed up the progress in this area.

Original: Camera model C does not learn fine-grained controlover the classes distribution .
Revised: Camera model C does not learn fine-grained control .

Original: While our camera model is conditioned on the class label c , and, in theory, it should be able to perform finegrained control over the class focal length distributions (which is natural since landscape panoramas and close-up view of a coffee mug typically have different focal lengths), it does not do this, as shown in Fig.
Revised: While our camera model is conditioned on the class label c , and, in theory, it should be able to perform fine-grained control over the class focal length distributions (which is natural since landscape panoramas and close-up view of a coffee mug typically have different focal lengths), it does not do this, as shown in Fig.

Original: We attribute this problem to the implicit bias of G to produce large-FoV images due to tri-planes parametrization.
Revised: We attribute this problem to the implicit bias of G to produce large-FoV images due to tri-planes parametrization.

Original: Tri-planes define a limited volume box in space, and close-up renderings with large focal length would utilize fewer tri-plane features, hence using less generator’s capacity.
Revised: Triplanes define a limited volume box in space, and close-up renderings with large focal length would utilize fewer tri-plane features, hence using less generator’s capacity.

Original: This is why 3DGP attempts to perform modeling with larger field-of-view values.
Revised: This is why 3DGP attempts to perform modeling with larger field-of-view values.

Original: [Figure]
Revised: [Figure]

Original: B I MPLEMENTATION DETAILS
Revised: B I MPLEMENTATION DETAILS

Original: [Table]

Original: This section provides additional architectural and training details.
Revised: This section provides additional architectural and training details.

Original: Also note that we will release the source code.
Revised: Also note that we will release the source code.

Original: Our generator G consists of M , S and T .
Revised: Our generator G consists of M , S and T .

Original: Mapping network M takes noise z ∈ R 512 and class label c ∈ 0 , ..., K − 1 , where K is the number of classes, and produces the style code w ∈ R 512 .
Revised: Mapping network M takes noise z ∈ Rand class label c ∈ 0 , ..., K − 1 , where K is the number of classes, and produces the style code w ∈ R 512 .

Original: Similar to StyleGAN2-ADA Karras et al.
Revised: Similar to StyleGAN2-ADA Karras et al.

Original: (2020a)
Revised: (2020a)

Original: and EpiGRAF Skorokhodov et al.
Revised: and EpiGRAF Skorokhodov et al.

Original: (2022), M is 2-layer MLP network with LeakyReLU activations and 512 neurons in each layer.
Revised: (2022), M is 2-layer MLP network with LeakyReLU activations and 512 neurons in each layer.

Original: Synthesis network is a decoder network same as StyleGAN2 Karras et al.
Revised: Synthesis network is a decoder network same as StyleGAN2 Karras et al.

Original: (2020b), except that it produce tri-plane feature features p
Revised: (2020b), except that it produce tri-plane feature features p

Original: = ( p xy , p yz , p xz ) ∈ R 3 × (512 × 512 × 32) .
Revised: = ( p xy , p yz , p xz ) ∈ R 3 × (512 × 512 × 32) .

Original: A 
Revised: A feature vector f xyz ∈ R 32 located ( x, y, z ) ∈ R 3 is computed by projecting the coordinate back to the tri-plane representation, followed by bi-linearly interpolating the nearby features and averaging the features from different planes. Following EpiGRAF (Skorokhodov et al., 2022), tri-plane decoder is a twolayer MLP with Leaky-ReLU activations and 64 neurons in the hidden layer, that takes a tri-plane

Original: [Figure]
Revised: [Figure]

Original: parameters ϕ
Revised: Estimator [Figure] [Equation] [Figure]

Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: z ϕ

Original: [Figure]
Revised: [Equation]

Original: lookat

Original: [Figure]
Revised: [Figure]

Original: ϕ

Original: [Figure]
Revised: [Equation]

Original: c posterior

Original: [Figure]
Revised: [Figure]

Original: parameters ϕ (a) Camera generator C .
Revised: ϕ (a a higher

Original: We condition C on class labels c when generating the camera position ˆ φ since it might be different for different classes. And we condition it on z when generating the look-at position and field-of-view since it might depend on the object shape (e.g., there is a higher probability to synthesize a close-up view of a dog’s snout rather than its tail). EachMLP consists on 3 layers with Soft- plus non-linearities.
Revised: Each

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: (b) Camera gradient penalty .
Revised: (b) Camera gradient penalty .

Original: We structure the regularization term for C as a function
Revised: We structure the regularization term for C as a function

Original: L ( | g | ) = | g | +1 / | g | (see (1)), and this function is visualized above.
Revised: L ( | g | ) = | g | +1 / | g | (see (1)), and this function is visualized above.

Original: This allows to prevent the collapse of C into delta distribution by it either producing constant values or very large/small values (which become constant after the sigmoid normalization).
Revised: This allows to prevent the collapse of C into delta distribution by it either producing constant values or very large/small values (which become constant after the sigmoid normalization).

Original: is by projecting the coordinate back to the plane , followed by bi-linearly interpolating the nearby features and averaging the features from different planes. Following EpiGRAF (Skorokhodov et al., 2022), tri-plane decoder is a two-layer MLP with Leaky-ReLU activations and 64 neurons in the hidden layer, that takes a tri-plane feature f xyz as input and produced the color and density ( RGB , σ ) in that point.
Revised: feature f xyz as input and produced the color and density ( RGB , σ ) in that point.

Original: We use same volume rendering procedure as EpiGRAF (Skorokhodov et al., 2022).
Revised: We use same volume rendering procedure as EpiGRAF (Skorokhodov et al., 2022).

Original: Camera generator .
Revised: Camera generator .

Original: Camera generator consists of linear layers with SoftPlus activations, it architecture is depicted in Fig.
Revised: Camera generator consists of linear layers with SoftPlus activations, it architecture is depicted in Fig.

Original: We found it crucial to use SoftPlus activation and not LeakyReLU, since optimization of Camera Gradient Penalty for non-smooth functions is unstable for small learning rates (smaller 0.02).
Revised: We found it crucial to use SoftPlus activation and not LeakyReLU, since optimization of Camera Gradient Penalty for non-smooth functions is unstable for small learning rates (smaller 0.02).

Original: Our gradient penalty minimizes the function L = | g | + 1 / | g | , where | g | is the input/output scalar derivative of the camera generator C .
Revised: Our gradient penalty minimizes the function L = | g | + 1 / | g | , where | g | is the input/output scalar derivative of the camera generator C .

Original: The motivation is to prevent the collapse of C into delta distribution and the intuition is the following.
Revised: The motivation is to prevent the collapse of C into delta distribution and the intuition is the following.

Original: C can collapse into delta distribution in two ways: 1) by starting to produce the constant output for all the input values (this is being prevented by the first term | g | and 2) by starting producing ±∞ for all the inputs, which are at the end converted to constants since we apply sigmoid normalization on top of its outputs to normalize them into a proper range (e.g., pitch is bounded in (0, π )) — this, in turn, is prevented by the second term 1 / | g | .
Revised: C can collapse into delta distribution in two ways: 1) by starting to produce the constant output for all the input values (this is being prevented by the first term | g | and 2) by starting producing ±∞ for all the inputs, which are at the end converted to constants since we apply sigmoid normalization on top of its outputs to normalize them into a proper range (e.g., pitch is bounded in (0, π )) — this, in turn, is prevented by the second term 1 / | g | .

Original: The minimum value of this function is 2 (which is achieved when the gradient  is linear), and the function itself is visualized in Fig. 9b.
Revised: The minimum value of this function is 2 (which is achieved when the gradient norm is constant and equals to 1), and the function itself is visualized in Fig. 10b.

Original: Adversarial depth supervision .
Revised: Adversarial depth supervision .

Original: For depth adaptor, we use a three layer convolutional neural network with 5 × 5 kernel sizes (since it is shallow, we increase its receptive field by increasing the kernel size) with LeakyReLU activations and 64 filters in each layer.
Revised: For depth adaptor, we use a three layer convolutional neural network with 5 × 5 kernel sizes (since it is shallow, we increase its receptive field by increasing the kernel size) with LeakyReLU activations and 64 filters in each layer.

Original: Additionally we use one shared convolutional layer that converts 64 × h × w features to the depth maps.
Revised: Additionally we use one shared convolutional layer that converts 64 × h × w features to the depth maps.

Original: We use the same architecture for D as EpiGRAF (Skorokhodov et al., 2022), but additionally concatenate a 1-channel depth to the 3-channel RGB input.
Revised: We use the same architecture for D as EpiGRAF (Skorokhodov et al., 2022), but additionally concatenate a 1-channel depth to the 3-channel RGB input.

Original: Finally E and F is pretrainined LeReS (Miangoleh et al., 2021) and ResNet50 (He et al., 2016) networks without any modifications.
Revised: Finally E and F is pretrainined LeReS (Miangoleh et al., 2021) and ResNet50 (He et al., 2016) networks without any modifications.

Original: We used the timm library to extract the features for real images.
Revised: We used the timm library to extract the features for real images.

Original: Similarly to EpiGRAF (Skorokhodov et al., 2022) we set the R 1 regularization (Mescheder et al., 2018) term
Revised: Similarly to EpiGRAF (Skorokhodov et al., 2022) we set the R 1 regularization (Mescheder et al., 2018) term

Original: λ r = 0 .
Revised: λ r = 0 .

Original: 1 and knowledge distillation term λ dist = 1 .
Revised: 1 and knowledge distillation term λ dist = 1 .

Original: R 1 regularization helps to stabilize
Revised: R 1 regularization helps to stabilize

Original: GAN training and is formulated as a gradient penalty on top of the discriminator’s real inputs:
Revised: GAN training and is formulated as a gradient penalty on top of the discriminator’s real inputs:

Original: [Equation]
Revised: [Equation]

Original: The (hand-wavy) intuition is that it makes the discriminator surface more flat in the vicinity of real points, making it easier for the generator to reach them.
Revised: The (hand-wavy) intuition is that it makes the discriminator surface more flat in the vicinity of real points, making it easier for the generator to reach them.

Original: We train all the models with Adam optimizer (Kingma & Ba, 2014) using the learning rate of 2 e - 3 and β 1 = 0 . 0 , β 2 = 0 .
Revised: We train all the models with Adam optimizer (Kingma & Ba, 2014) using the learning rate of 2 e - 3 and β 1 = 0 . 0 , β 2 = 0 .

Original: Following
Revised: Following

Original: EpiGRAF (Skorokhodov et al., 2022), our model uses patch-wise training with 64 × 64 -resolution patches and uses their proposed β scale sampling strategy without any modifications.
Revised: EpiGRAF (Skorokhodov et al., 2022), our model uses patch-wise training with 64 × 64 -resolution patches and uses their proposed β scale sampling strategy without any modifications.

Original: We use the batch size of 64 in all the experiments, since in early experiments we didn’t find any improvements from using a large batch size neither for our model nor for StyleGAN2, as observed by Brock et al.
Revised: We use the batch size of 64 in all the experiments, since in early experiments we didn’t find any improvements from using a large batch size neither for our model nor for StyleGAN2, as observed by Brock et al.

Original: (2018) and Sauer et al.
Revised: (2018) and Sauer et al.

Original: As being said in Section 4, we use the instance selection technique by DeVries et al.
Revised: As being said in Section 4, we use the instance selection technique by DeVries et al.

Original: (2020) to remove image outliers from the datasets since they might negatively affect the geometry.
Revised: (2020) to remove image outliers from the datasets since they might negatively affect the geometry.

Original: This procedure works by first extracting a 2,048-dimensional feature vector for each image, then fitting a multivariate gaussian distribution on the obtained dataset and removing the images, which features have low probability density.
Revised: This procedure works by first extracting a 2,048-dimensional feature vector for each image, then fitting a multivariate gaussian distribution on the obtained dataset and removing the images, which features have low probability density.

Original: For SDIP Dogs and LSUN Horses, we fit a multi-variate gaussian distribution for the whole dataset.
Revised: For SDIP Dogs and LSUN Horses, we fit a multi-variate gaussian distribution for the whole dataset.

Original: For ImageNet, we fit a separate model for each class with additional diagonal regularization for covariance, which is needed due to its singularity: feature vector has more dimensions than the number of images in a class.
Revised: For ImageNet, we fit a separate model for each class with additional diagonal regularization for covariance, which is needed due to its singularity: feature vector has more dimensions than the number of images in a class.

Original: We refer to (DeVries et al., 2020) for additional details.
Revised: We refer to (DeVries et al., 2020) for additional details.

Original: Also note that we will release the source code and the checkpoints, which would additionally convey all other implementation details.
Revised: Also note that we will release the source code and the checkpoints, which would additionally convey all other implementation details.

Original: C K NOWLEDGE DISTILLATIONCOMPARISON
Revised: C K NOWLEDGE DISTILLATION

Original: In this section we compare our knowledge distillation strategy with projected D (Sauer et al., 2021; 2022), another popular technique of utilizing existing classification models for GAN training.
Revised: In this section we compare our knowledge distillation strategy with projected D (Sauer et al., 2021; 2022), another popular technique of utilizing existing classification models for GAN training.

Original: Note that projected D relies on pretrained EfficientNet (Tan & Le, 2019), it freezes most of the weights and adds some random convolutions layers that project intermediate outputs of EfficientNet to the features, that will be later processed by discriminator.
Revised: Note that projected D relies on pretrained EfficientNet (Tan & Le, 2019), it freezes most of the weights and adds some random convolutions layers that project intermediate outputs of EfficientNet to the features, that will be later processed by discriminator.

Original: Note that this strategy relies on specific architecture of Discriminator D , thus in order to adapt it for 3DGP we have to rely on two discriminators.
Revised: Note that this strategy relies on specific architecture of Discriminator D , thus in order to adapt it for 3DGP we have to rely on two discriminators.

Original: First one is our disciminator D with additional depth input and conditioned on patch parameters and second one is projected D .
Revised: First one is our disciminator D with additional depth input and conditioned on patch parameters and second one is projected D .

Original: In Tab 4 we show comparison between projected D and our knowledge distillation strategy on ImageNet (Deng et al., 2009) and SDIP Dogs 40k Mokady et al.
Revised: In Tab 3 we show comparison between projected D and our knowledge distillation strategy on ImageNet (Deng et al., 2009) and SDIP Dogs 40k Mokady et al.

Original: (2022) datasets.
Revised: (2022) datasets.

Original: First we compare these strategies for 2D generator architectures: StyleGAN3-t (large) (Karras et al., 2021) and StyleGAN2 (Karras et al., 2020a).
Revised: First we compare these strategies for 2D generator architectures: StyleGAN3-t (large) (Karras et al., 2021) and StyleGAN2 (Karras et al., 2020a).

Original: It can be observed that projected
Revised: It can be observed that projected

Original: D is not generic and it results heavily varies depending on generator architecture.
Revised: D is not generic and it results heavily varies depending on generator architecture.

Original: Moreover training cost of projective D is higher.
Revised: Moreover training cost of projective D is higher.

Original: Finally we test this strategy for our 3D generator, and again we can observe that this technique lead to inferior results in both visual quality and training cost.
Revised: Finally we test this strategy for our 3D generator, and again we can observe that this technique lead to inferior results in both visual quality and training cost.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: Typically, knowledge distillation in classifiers is performed using Kullback-Leibler divergence on top of logits (Hinton et al., 2015; Kim et al., 2021), rather than L 2 distance on top of hidden activations, as in our case.

Revised: There are two design reasons of why we use the L 2 : 1) it is more generalizable and one can transfer knowledge from other models with the same design, like CLIP (Radford et al., 2021) or Mask R-CNN (He et al., 2017); 2) our discriminator’s task shouldn’t be able to perform classification, that is why guiding it with the KL classification loss is not natural.

Revised: However, in

Revised: Fig.

Revised: 11, we provide additional exploration with other knowledge distillation objectives on top of

Revised: StyleGAN2 (Karras et al., 2020b) trained for conditional ImageNet generation on the 128 2 resolution.

Revised: One can observe that KL and L 2 objectives perform approximately the same.

Revised: Also, guiding by CLIP underperforms to ResNet guidance initially, but starts to outperform after more training is performed: we hypothesize that the reason is that it is a more general model.

Revised: We do not use CLIP guidance for our generators in this paper so not to give it an unfair advantage compared to other models.

Original: D F AILED EXPERIMENTS
Revised: D F AILED EXPERIMENTS

Original: In this section we describe several ideas that have been tried in this project, which however did not work for us.
Revised: In this section we describe several ideas that have been tried in this project, which however did not work for us.

Original: Gradient Penalty for Depth Adaptor .
Revised: Gradient Penalty for Depth Adaptor .

Original: To prevent depth adaptor A from completely faking the depth and forcing it to still provide a useful signal to the generator, one could employ a gradient penalty that will force gradient of A close to one.
Revised: To prevent depth adaptor A from completely faking the depth and forcing it to still provide a useful signal to the generator, one could employ a gradient penalty that will force gradient of A close to one.

Original: We tried it in two setups: window-to-pixel gradient and pixel-to-pixel gradient.
Revised: We tried it in two setups: window-to-pixel gradient and pixel-to-pixel gradient.

Original: Even with the large weight for this loss and shallow adaptor with k =kernel size, it didn’t enforce good geometry 3DGP with Projected D .
Revised: Even with the large weight for this loss and shallow adaptor with k =kernel size, it didn’t enforce good geometry 3DGP with Projected D .

Original: Before devoting to knowledge distillation, we spend significant resources trying to incorporate projected D in our setup.
Revised: Before devoting to knowledge distillation, we spend significant resources trying to incorporate projected D in our setup.

Original: For double discriminators setting, see Appx C, we tested: different weighting strategies, less frequent updates for projected D , enabling projected D after some geometry was already learned.
Revised: For double discriminators setting, see Appx C, we tested: different weighting strategies, less frequent updates for projected D , enabling projected D after some geometry was already learned.

Original: We also tested single projected D , where we learned additional depth encoder that can consume our depth inputs.
Revised: We also tested single projected D , where we learned additional depth encoder that can consume our depth inputs.

Original: In all these experiments learned geometry was significantly inferior to the setting without projected D .
Revised: In all these experiments learned geometry was significantly inferior to the setting without projected D .

Original: Most of the time geometry becomes completely flat.
Revised: Most of the time geometry becomes completely flat.

Original: MLPs instead of tri-planes .
Revised: MLPs instead of tri-planes .

Original: We noticed that tri-planes are extremely biased towards flat generation.
Revised: We noticed that tri-planes are extremely biased towards flat generation.

Original: Therefore one natural idea would be to utilize different representations, one possible candidate is
Revised: Therefore one natural idea would be to utilize different representations, one possible candidate is

Original: MLPs Chan et al.
Revised: MLPs Chan et al.

Original: However our experiments with MLPs always lead to significantly inferior visual quality.
Revised: However our experiments with MLPs always lead to significantly inferior visual quality.

Original: Our hypothesis is that they just don’t have enough capacity to model large scale datasets.
Revised: Our hypothesis is that they just don’t have enough capacity to model large scale datasets.

Original: Few-shot NeRF regularization .
Revised: Few-shot NeRF regularization .

Original: We also try to improve learned geometry by incorporating ray entropy minimization loss from InfoNeRF Kim et al.
Revised: We also try to improve learned geometry by incorporating ray entropy minimization loss from InfoNeRF Kim et al.

Original: (2022) with different weights.
Revised: (2022) with different weights.

Original: But the model always diverges.
Revised: But the model always diverges.

Original: We hypothesize that this is because it is significantly more complicated to find proper geometry, if the model is stuck in local minimum where the object densities are very sharp.
Revised: We hypothesize that this is because it is significantly more complicated to find proper geometry, if the model is stuck in local minimum where the object densities are very sharp.

Original: Normal supervision .
Revised: Normal supervision .

Original: Another prominent idea was utilizing normal supervision, since generic networks for normals
Revised: Another prominent idea was utilizing normal supervision, since generic networks for normals

Original: Eftekhar et al.
Revised: Eftekhar et al.

Original: (2021) are also widely available and provide good results on the arbitrary data.
Revised: (2021) are also widely available and provide good results on the arbitrary data.

Original: Since computing normals requires gradient of density σ with respect to ( x, y, z ) and pytorch does not provide second order derivatives for F.grid sample 3 , we developed a custom
Revised: Since computing normals requires gradient of density σ with respect to ( x, y, z ) and pytorch does not provide second order derivatives for F.grid sample 4 , we developed a custom

Original: CUDA kernel that computes second derivative for F.grid sample .
Revised: CUDA kernel that computes second derivative for F.grid sample .

Original: Unfortunately we found that normals supervision is inferior to depth and it is much easier for the generator G to fake normals.
Revised: Unfortunately we found that normals supervision is inferior to depth and it is much easier for the generator G to fake normals.

Original: Preventing C collapse via variance/entropy/moments regularization .
Revised: Preventing C collapse via variance/entropy/moments regularization .

Original: Before arriving to our current form we had been experimenting with other forms of regularization: maximizing variance with some small loss coefficient, or entropy (in the assumption that the posterior distribution is gaussion), or additionally pushing mean/skewness/kurtosis to the one of the gaussian distribution.
Revised: Before arriving to our current form we had been experimenting with other forms of regularization: maximizing variance with some small loss coefficient, or entropy (in the assumption that the posterior distribution is gaussion), or additionally pushing mean/skewness/kurtosis to the one of the gaussian distribution.

Original: But each time, the generator was finding the ways to “cheat” the regularization and managing to produce either a delta distribution or a mixture of delta distributions (it very “likes” doing so since, it is able to completely flat images and cheat the geometry in this regime).
Revised: But each time, the generator was finding the ways to “cheat” the regularization and managing to produce either a delta distribution or a mixture of delta distributions (it very “likes” doing so since, it is able to completely flat images and cheat the geometry in this regime).

Original: E A DDITIONAL SAMPLES
Revised: E A DDITIONAL SAMPLES

Original: Since it is much easier to visualize the samples from NeRF-based generators as videos rather than
Revised: Since it is much easier to visualize the samples from NeRF-based generators as videos rather than

Original: RGB images, we provide all the additional visualizations on https://u2wjb9xxz9q.github.io/.
Revised: RGB images, we provide all the additional visualizations on https://u2wjb9xxz9q.github.io.

Original: F N ON -F LATNESS S CORE DETAILS
Revised: F N ON -F LATNESS S CORE DETAILS

Original: To detect and quantify the flatness of 3D generators, we propose a Non-Flatness Score (NFS).
Revised: To detect and quantify the flatness of 3D generators, we propose a Non-Flatness Score (NFS).

Original: To compute NFS, we sample N latent codes with their corresponding radiance fields.
Revised: To compute NFS, we sample N latent codes with their corresponding radiance fields.

Original: For each radiance field, we perform integration following Eq.
Revised: For each radiance field, we perform integration following Eq.

Original: 2 to obtain its depth, however, we first set the 50% of lowest density to zero.
Revised: 2 to obtain its depth, however, we first set the 50% of lowest density to zero.

Original: This is necessary to cull spurious density artifacts.
Revised: This is necessary to cull spurious density artifacts.

Original: We then normalize each depth map according to the corresponding near and far planes and compute a histogram with B bins, showing the distribution of the depth values.
Revised: We then normalize each depth map according to the corresponding near and far planes and compute a histogram with B bins, showing the distribution of the depth values.

Original: To analyze how much the depth values are concentrated versus spread across the volume, we compute its entropy for each distribution.
Revised: To analyze how much the depth values are concentrated versus spread across the volume, we compute its entropy for each distribution.

Original: Averaging over N depth maps gives the sought score:
Revised: Averaging over N depth maps gives the sought score:

Original: [Equation]
Revised: [Equation]

Original: where b ( d ( i ) )
Revised: where b ( d ( i ) )

Original: j is the normalized number of depth values in the j -th bin of the i -th depth map d ( i ) , with N = 256 and B = 64 .
Revised: j is the normalized number of depth values in the j -th bin of the i -th depth map d ( i ) , with N = 256 and B = 64 .

Original: NFS does not directly evaluate the quality of the geometry.
Revised: NFS does not directly evaluate the quality of the geometry.

Original: Instead, it helps to detect and quantify how flat the generated geometry is.
Revised: Instead, it helps to detect and quantify how flat the generated geometry is.

Original: In Fig.
Revised: In Fig.

Original: 6, we show examples of repetitive geometry generated by EG3D and a more diverse generation produced by 3DGP along with their depth histograms and NFS. 
Revised: 6, we show examples of repetitive geometry generated by EG3D and a more diverse generation produced by 3DGP along with their depth histograms and NFS. G I NVESTIGATING THE POTENTIAL DATA LEAK FROM D EPTH E STIMATOR

Revised: LeReS (Miangoleh et al., 2021) depth estimator (which was used in our work) was pre-trained on a different set of images compared to ImageNet (or other animals datasets, explored in our work), and this theoretically could help our generator achieve better results by implicitly giving it access to a broader image set rather than giving only texture-less geometric guidance.

Revised: In this section, we investigate this, and conclude that it is not the case: the pre-training data of LeReS is too different from the explored datasets, containing almost no animal images — while all the explored datasets are very animal-dominated, including ImageNet which contains 482 animal classes.

Revised: Hence, LeReS has no chance in helping our generator by giving it access to additional data.

Revised: LeReS (Miangoleh et al., 2021) trains on a combination of 4 datasets (in fact, parts of them): 1) HRWSI (Xian et al., 2020) contains outdoor city imagery (e.g., buildings, monuments, landscapes); 2)

Revised: DiverseDepth (Yin et al.,

Revised: ) contains clips from in-the-wild movies and videos; 3)

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]

Revised: Holopix50k (Hua et al., 2020) — diverse, in-the-wild web images (it is the most similar to ImageNet in terms of underlying data distribution); and 4) Taskonomy (Zamir et al., 2018) contains indoor scenes (bedrooms, stores, etc.).

Revised: In Fig.

Revised: 15, we provide 30 random images from each dataset.

Revised: In Tab. 5, we rigorously explore how many animal images does the pre-training data of LeReS contains.

Revised: We perform this in two ways: by directly counting the amount of animals with the pre-trained

Revised: Mask R-CNN model He et al.

Revised: (2017) and by computing FID scores between the pre-training datasets of LeReS and ImageNet animal subset.

Revised: The pre-trained Mask R-CNN model from TorchVision is able to detect 10 animal classes: birds, cats, dogs, horses, sheeps, cows, elephants, bears, zebras and giraffes.

Revised: Following the official tutorial, we used a threshold of 0.8.

Revised: We downloaded the datasets with the official download scripts of LeReS.

Revised: [Table]

Revised: 7k

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: From Tab. 4, one can see that the pre-training data of LeReS contains just 3.79% of animal images, while ImageNet contains 48.3%.

Revised: In this way, it is too far away in terms of distribution from the

Revised: Animal subset of ImageNet.

Revised: From this, one can conclude the following: Depth Estimator has almost never seen animals during training, since its training datasets have 600 fewer animal images than ImageNet.

Revised: This means that our generator does not receive any unfair advantage in synthesizing animals from using adversarial depth supervision by implicitly getting access to a larger set of images.

Revised: How does it affect our generator’s performance in synthesizing animals?

Revised: Will it have an unusually poor FID compared to non-animal data or compared to other methods?

Revised: In Tab. 5, we report FID scores of each generator on animal vs non-animal subsets of ImageNet.

Revised: From these scores, one can see that the trend is the same as for FID on full ImageNet.

Revised: This implies that our generator performs equally well on data which constitutes the main part of our training dataset and is not a part of the depth estimator pre-training.

Revised: In this way, one can conclude that adversarial depth supervision helps to achieve better synthesis quality only through geometric guidance.

Revised: It does not help the generator to do this by leaking the knowledge of pre-trained data from the pre-trained depth estimator.

Revised: only

Revised: [Table]

Revised: H G EOMETRY EVALUATION ON S HAPE N ET

Revised: In this section, we present the details and the results of an additional study, which rigorously shows that our proposed adversarial depth supervision improves the geometry quality.

Revised: H. R ENDERING DETAILS

Revised: We take the ShapeNet dataset Chang et al.

Revised: (2015), which consists of 50k models from 54 classes and render it from random frontal camera positions.

Revised: For this, we set the camera on a sphere of radius 2 and randomly choose its position by sampling rotation and elevation angles from N (0 , π/ 8) and

Revised: N ( π/ 2 , π/ 8) , respectively.

Revised: We chose frontal views to be closer to the real-world scenario: the most popular image synthesis datasets are dominated by the frontal views.

Revised: Also, if the dataset is not frontal, then there is less necessity in additional geometry supervision: it is a good enough 3D bias (which does not exist in modern in-the-wild datasets).

Revised: For wide camera distribution, a generator can learn proper geometry on its own: see experiments on synthetic datasets with 360 degrees camera coverage by Skorokhodov et al.

Revised: We render just a single view per model since this is also the scenario which we typically have in real datasets (e.g., in all our explored datasets, there is just a single view per scene).

Revised: Some of the models in ShapeNet are broken (they lack the corresponding mesh files), so we discard them.

Revised: In total, this gave us 51209 training images, which are visualized in Fig.

Revised: We used BlenderProc Denninger et al.

Revised: (2019) to do rendering.

Revised: Also, during rendering we removed the transparent elements of the meshes since they were providing aliasing issues in the depth maps.

Revised: H. E XPERIMENTS

Revised: Experimental setup .

Revised: After rendering, we trained our main baselines, EG3D (Chan et al., 2022) and

Revised: EpiGRAF (Skorokhodov et al., 2022), on this dataset using the ground truth camera parametrization and distribution when sampling the camera poses.

Revised: For 3DGP, we trained it in the following variants: 1) with the default adversarial depth supervision (i.e., using P ( ¯ d ) = 0 . 5 ); and 2-3) with the adversarial depth supervision, but with corrupted depth maps, where corruption is simulated by blurring, similar to (DeVries et al., 2021).

Revised: Also, we trained StyleGAN2 (Karras et al., 2020b) as a lower bound on FID.

Revised: We disabled camera distribution learning and knowledge distillation for the experiments with 3DGP to avoid unnecessary complications: we investigate only adversarial depth supervision in this study.

Revised: For all the baselines, we use white background in volumetric rendering.

Revised: Evaluation .

Revised: We compare the methods in terms of FID (Heusel et al., 2017) and also Frechet Pointcloud Distance (FPD), proposed by Shu et al.

Revised: This metric is an FID analog for point clouds: it uses a pre-trained PointNet (Qi et al., 2017a) to extract the features from point clouds and then computes Frechet distance between real and fake representations sets, approximating their underlying distribution as multi-variate normal one.

Revised: [Table]

Revised: We used the official codebase of Shu et al.

Revised: (2019) to compute the metrics.

Revised: In their original work,

Revised: Shu et al.

Revised: (2019) used a customly trained PointNet (Qi et al., 2017a) to extract the features.

Revised: But we observed an issue with it: it was sometimes producing unusually high FPD scores in the order of 10 6 , even for real point clouds.

Revised: This is why we extracted pointclouds features with a pre-trained

Revised: PointNet++ (Qi et al., 2017b) model from a popular public implementation (Yan, 2019).

Revised: To extract point clouds from the models, we first extracted the surfaces from them via marching cubes.

Revised: For this, we sampled density fields in 256 3 resolution, and then used the marching cubes implementation of PyMCubes (pmneila, 2015) to extract the surfaces.

Revised: Following EG3D, we thresholded the surface for marching cubes at the density value of σ = 10 .

Revised: Our overall pipeline is identical to the original procedure of SDF extraction in the EG3D repo, but simpler in terms of implementation.

Revised: After that, we extracted point clouds by sampling 2,048 points on the surface for both real and fake meshes using uniform sampling from trimesh Dawson-Haggerty et al.

Revised: H. R ESULTS

Revised: The results of these experiments are presented in Tab. 6.

Revised: We also provide random sample examples (seed 1, classes 1-8) in Fig.

Revised: EG3D (Chan et al., 2022) fails to recover the geometry and generates inverted shapes with hollow geometry.

Revised: This is highlighted by its very high FPD score.

Revised: EpiGRAF (Skorokhodov et al., 2022) is able to recover shapes, but it models the white background via an additional sphere, which also damages its FPD.

Revised: Our method, in contrast, learns proper geometry and correctly drops the background.

Revised: When the depth maps are corrupted with blurring, it deteriorates both its geometry quality and texture quality, which is also confirmed by the metrics and the provided samples.

Revised: I D ETAILS ON THE EXPERIMENTS WITH 3D P HOTO I NPAINTING

Revised: In this section, we provide the details of our experiments on combining 2D generation with 3D Photo

Revised: Inpainting techniques.

Revised: For a 2D generator, we chose StyleGAN-XL (Sauer et al., 2022) since it achieves state-of-theart visual quality on ImageNet, as measured by FID (Heusel et al., 2017).

Revised: We took the original checkpoint for 256 2 generation from the official repository.

Revised: First, we generated 50,000 images with the model and computed their FID: this gave the value of 2.51 vs 2.26, reported by the authors, which is a negligible difference.

Revised: After that, we used the original codebase of 3DPhoto (Shih et al., 2020) to synthesize the 3D variations of 10k random images.

Revised: We kept all the hyperparameters the same.

Revised: In 3DPhoto, there is no spherical camera parametrization, this is why we had to simulate it the following way.

Revised: We assumed that the sphere center lies at the median depth value from the original camera position (which is (0 , 0 , 0) ), and had been rotating the camera around that point.

Revised: As discussed in §4, we used the narrow camera distribution to sample the points.

Revised: Namely, we used the normal distributions with standard deviations of σ yaw = 0 .

Revised: 3 and σ pitch = 0 .

Revised: 15 for sampling rotation and elevation angles, respectively.

Revised: [Figure]

Revised: [Figure]

Revised: For negligible camera variations, this strategy produces excellent results.

Revised: While it is not perfectly view consistent, it inherits state-of-the-art image quality from the 2D generator, which was used to synthesize the original images.

Revised: But for larger camera variations, the quality becomes to deteriorate very quickly.

Revised: Not only noticeable inpainting artifacts start to appear, but also outpainting problem emerges.

Revised: The model was not trained to perform image extrapolation or inpaint large regions and fails to fill the holes which appear in larger camera movements.

Revised: We provide the visualizations for it in Fig.