P IX 2S CENE : L EARNING I MPLICIT 3D R EPRESENTATIONS FROM A S INGLE I MAGE
Anonymous authors
Paper under double-blind review
A BSTRACT
We aim to model 3D properties of the scenes from single 2D images.
Learning 3D scenes from 2D images is a long-standing problem in computer vision with applications in related ﬁelds such as simulation and robotics.
We propose pix2scene , a deep generative-based approach that represents the 3D scene in a learnt latent variable decoded into a viewpoint-dependent representation that can be rendered.
Our method learns the depth of the scene and leverages a local smoothness assumption to extract the orientation of visible scene points.
We achieve this using an encoderdecoder adversarial learning mechanism and a novel differentiable renderer to train the 3D model in an end-to-end fashion, using only images.
We showcase the generative ability of our model qualitatively on the ShapeNet dataset (Chang et al., 2015).
We also demonstrate that our model can predict the structure of scenes from various, previously unseen view points.
Finally, we evaluate the effectiveness of the learned 3D scene representation in supporting 3D spatial reasoning.
I NTRODUCTION
Understanding the 3-dimensional (3D) world from its 2-dimensional (2D) projections is a fundamental problem in computer vision with a broad range of application in robotics, simulation and design.
Given that the majority natural scene data is available exclusively in the form of 2D images, the ability to directly infer knowledge about 3D structure from these images would be of great utility in scene understanding.
Inferring the 3D structure from multiple images of a scene has been pursued extensively, such as in stereo or structure from motion tasks (Hartley & Zisserman, 2004). Since most available natural image data informative about the real world comes with only a single view of a given scene, it is perhaps more important to explore the development of models which can infer the 3D structural properties from a single image . On the other hand, single image 3D recovery is an extremely challenging and heavily under constrained task.
The system has to rely on prior knowledge and 2D visual cues such as textures, shadows or occlusions in order to provide hints to the 3D structure of the scene. Practically, building a machine learning model that learns to infer 3D structure from images requires either a strong inductive bias or supervision. While some have used the 3D ground truth as explicit supervision (Wu et al., 2016; 2015), in most cases of interest, such supervision will not be available.
Consequently, our long term goal is to infer the 3D structure of realistic scenes from single images . In this paper we take a step towards this direction via a method of unsupervised learning of the 3D structure, directly from a single 2D image of each scene.
Our method based on the adversarial learning framework (Goodfellow et al., 2014) and exploits a uniquely suitable 3D representation (i.e., surfels (Pﬁster et al., 2000)) and a differentiable renderer. Most 3D reconstruction methods rely on representing 3D objects explicitly using either voxels (Rezende et al., 2016; Yan et al., 2016) or meshes (Kanazawa et al., 2018; Wang et al., 2018).
Explicit representations store all the rendering-relevant information from a given 3D space and are easily transferable, i.e., they can be loaded with any 3D modeling software and viewed from any angle. However, approaches using explicit representations typically scale very poorly ( O ( n 3 ) or require a sparse/discrete representation which can be challenging for deep learning methods. As a result, these representations have only been applied to the reconstruction of single objects.
As an alternative we propose to learn an implicit 3D representation which produces only the 3D geometry which is directly relevant for a particular viewpoint.
Our viewpoint-speciﬁc 3D geometry is captured [Figure] using camera facing surfels (Pﬁster et al., 2000) which are surface elements deﬁned by its position, orientation and material properties.
Given an image we can infer its implicit 3D representation and then recreate novel surfel representations of the underlying scene from unobserved viewpoints.
In general, we note that in a 3D scene, only a small fraction of the entities are perceivable from the camera. As the camera moves, and the occluded regions become visible, our method then generates surfels for those newly unoccluded regions. Another advantage of this approach is that minimal number of primitives (surfels) are required to obtain a high-resolution image as the camera moves closer to a part of the scene. Moreover this representation ﬁts well with image based convolutional architectures.
Our model, Pix2Scene , is a deep generative-based approach for modelling the 3D structure of a scene directly from images.
This model is unsupervised in the sense that it does not require 3D groundtruth or any other kind of image annotations.
We base our model on Adversarially Learned Inference (ALI) approach (Dumoulin et al., 2016).
ALI extends the GAN (Goodfellow et al., 2014) framework by learning to infer the latent representation of a given image. In pix2scene the learned latent space embeds the 3D information of the underlying scene. The latent representation is mapped via a decoder network to a view-dependent 3D surface and then projected to image space by a differentiable renderer.
The resulting image is then evaluated by an adversarial critic.
While our long-term goal is to be able to infer the 3D structure of a real-world photograph, in this paper we experiment exclusively with synthetically-constructed scenes and adopt several simplifying assumptions. In particular, we assume that the world is piece-wise smooth and that for each input image the illumination, view and object materials are known.
This work has the following main contributions, (1) we propose a novel unsupervised method for 3D understanding from a single image; (2) we propose a new implicit 3D representation based on view-space surfels; (3) we propose a surfel-based differentiable 3D renderer that can be used as a layer of a neural network; and (4) we propose 3D-IQTT a new 3D understanding evaluation benchmark.
This task evaluates the model’s ability to perform mental rotation by obtaining comprehensive understanding of underlying 3D structure. We also estimate the camera pose as part of the learnt latent variable for this particular task.
R ELATED W ORK
The generation and reconstruction of 3D objects from images has been studied extensively in the computer vision and graphics communities (Saxena et al., 2009;
Chaudhuri et al., 2011; Kalogerakis et al., 2012; Chang et al., 2015; Rezende et al., 2016; Soltani et al., 2017).
Our work bears some conceptual similarities to Kulkarni et al.
(2015) which casts the 3D reconstruction problem as a more traditional inverse graphics task.
By using Variational Auto-Encoder(VAEs) (Kingma & Welling, 2014), they learn a representation of objects that disentangles factors of variations from images (i.e., object pose and conﬁguration) and use the approach for speciﬁc transformations such as out of axis rotation.
However, unlike their approach, ours is fully unsupervised and we implicitly generate 3D structure of scenes from single images.
Our mechanism learns a latent representation for the underlying scene, which can later be used to render from different views and lighting conditions.
Similar to ours, Rezende et al.
(2016) infer the 3D conﬁguration at their output. They adopt a probabilistic inference framework to build a generative model for 3D by combining a standard projection mechanism with gradient estimation methods.
In particular, their approach requires multiple runs with mechanisms such as REINFORCE (Williams, 1992) in order to infer gradient from the projection layer.
In addition, their use of mesh and voxel representations could become an obstacle to scaling their method to more complex scenes.
Our approach is not susceptible to restrictions imposed by meshes or other scaling issues and has the potential to adapt to arbitrary scene conﬁgurations.
M ETHOD 3. I MPLICIT 3D R EPRESENTATION AND S URFELS
Explicitly representing 3D structure presents different challenges for generative models (Kobbelt & Botsch, 2004).
Representing entire objects using voxels scales poorly given its ( O ( n 3 ) complexity. The vast majority of the generated voxels aren’t relevant to most viewpoints, such as the voxels that are entirely inside of objects. A common workaround is to use a sparse representation such as meshes. However, these too come with their own drawbacks, such as the need to discretise complex objects.
This makes mesh representation difﬁcult to generate using neural networks.
Current mesh based methods mainly rely on deforming a pre-existing mesh. On the other hand, our implicit approach represents the 3D scene in a high-dimensional latent variable. In our framework, this latent variable (or vector) is decoded using a generator network into a viewpoint-dependent representation of surface elements — similar to the surfels (Pﬁster et al., 2000) — that constitute the visible part of the scene.
This representation is very compact: given a renderer’s point of view, we can represent only the part of the 3D surface needed by the renderer.
Also, as the camera moves closer to a part of the scene, our generator will allocate more surfels to represent that part of the scene and thereby increasing the resolution.
Figure 1 illustrates these different representations.
For descriptive purpose, surfels are shown as squares, but in general they do not have any shape.
Formally, surfels are represented as a tuple ( P, N, ρ ) , where P =
( p x , p y , p z ) is its 3D position, N = ( n x , n y , n z ) is the surface normal vector, and
ρ =
( k r , k g , k b ) is the reﬂectance of the surface material.
Since we are interested in modelling structural properties of the scenes i.e., geometry and depth, we assume that objects in the scene have a uniform material.
We represent the surfels in the camera coordinate system.
This signiﬁcantly reduces the number of surfels by considering only the ones that will get projected onto a pixel in the rendered image.
Moreover, this allows to reduce the position parameters to only p z being this the distance along a ray going through the surfel to the center of its pixel.
3. D IFFERENTIABLE 3D R ENDERER
As the critic operates only on image space, we need to project the generated 3D representations back to the 2D space using a renderer.
In our setting, each stage of the rendering pipeline must be differentiable to allow us to take advantage of gradient-based optimization and backpropogate the critic’s error signal to the surfel representation.
The rendering process can be partitioned into two stages.
During forward-propagation, the ﬁrst stage ﬁnds the mapping between the surfels and the pixels; and the second stage computes the color of each pixel.
During back-propagation, the ﬁrst stage directs the gradients only to the surfels that get projected onto the image; and the second stage is differentiable as long as the shading operations are differentiable.
The ﬁrst stage of the rendering involves ﬁnding the mapping between the surfels and the pixels.
This requires performing the expensive operation of ray object intersection (See Figure 2a).
Our model requires a fast rendering engine as it will be use in every learning iteration.
Conventional ray tracing algorithms are optimized for generating multiple views from the same scene.
However in our setting during learning we render only one image from each scene.
Moreover ray tracing algorithms require from representing the full scene, which is very inefﬁcient as we only represent the part visible by the camera.
To resolve these issues, our generator proposes one surfel for each pixel in the camera’s [Figure] coordinate system.
Our PyTorch implementation of the differentiable renderer can render a 128 ×surfel-based scene in under 1.4 ms on a mobile NVIDIA GTX 1060 GPU.
The color of a surfel depends on the material reﬂectance, its position and orientation, and the ambient and point light source colors.
See Figure 2b.
Given a surface point P i , the color of its corresponding pixel I rc is given by the shading equation: [Equation]
[Equation]
where ρ i is the surface reﬂectance, L a is the ambient light’s color, L j is the j th positional light source’s color, with d ij =
L pos j − P i , or the direction vector from the scene point to the point light source, and k l , k q being the linear and quadratic attenuation terms respectively.
Equation 1 is an approximation of rendering equation proposed in Kajiya (1986).
3. P IX 2S CENE M ODEL
The adversarial training paradigm allows the generator network to capture the underlying target distribution by competing with an adversarial critic network.
Pix2scene employs bi-directional adversarial training to model the distribution of surfels from just 2D images.
3.3. B I -D IRECTIONAL A DVERSARIAL T RAINING
ALI (Dumoulin et al., 2016) or Bi-GAN (Donahue et al., 2016) extends the GAN (Goodfellow et al., 2014) framework by including the learning of an inference mechanism.
Speciﬁcally, in addition to the decoder network G x , ALI provides an encoder G z which maps data points x to latent representations z .
In these bi-directional models, the critic, D , discriminates in both the data space ( x versus G x ( z ) ), and latent space ( z versus G z ( x ) ), maximizing the adversarial value function over two joint distributions.
The ﬁnal min-max objective can be written as:
[Equation]
where q ( x ) and p ( z ) denote encoder and decoder marginal distributions.
3.3. M ODELLING DEPTH AND C ONSTRAINED N ORMAL E STIMATION
Based on the ALI formulation, as depicted in Figure 3, our model has an encoder network which captures the distribution over the latent space given an image data point x .
The decoder network maps a ﬁxed latent distribution p ( z ) (a standard normal distribution in our case) to the 3D surfel representation.
Next, the surfel representation are rendered into a 2D image using our differentiable renderer.
The resulting image is then given as input to the critic to distinguish from the real image data.
Note that the input to the critic comes from the joint space of data with its corresponding latent code, as in ALI.
A straightforward way to model the decoder network could be to learn a conditional distribution to produce the surfel’s depth ( p z ) and normal ( N ).
But this could lead to inconsistencies between the local shape and the surface normal.
For instance, the decoder can fake an RGB image of a 3D shape simply by changing the normals while keeping the depth ﬁxed.
To avoid this issue, we exploit the fact that real-world surfaces are locally planar, and surfaces visible to the camera have normals constrained to be in the half-space of visible normal directions from the camera’s view point.
Considering the camera to be looking along the − z direction, the estimated normal has the constraint n z > 0 .
Therefore, the local surface normal is estimated by solving the following problem for every surfel,
[Equation]
where the spatial gradient ∇ P is computed using 8 neighbour points, and P is the position of the surfels in the camera coordinate system obtained by backprojecting the generated depth along rays.
This approach enforces consistency between the predicted depth ﬁeld and the computed normals.
If the depth is incorrect, the normal-estimator outputs an incorrect set of normals, and result in an RGB image inconsistent with the data distribution, which would in-turn get penalized by the critic.
The decoder and the encoder networks are thus incentivized to predict realistic depths.
3.3. M ODEL T RAINING
The Wasserstein-GAN (Arjovsky et al., 2017) formalism provides stable training dynamics using the 1-Wasserstien distance between the distributions.
We adopt the gradient penalty setup as proposed in Gulrajani et al.
for more robust training, however, we modify the formulation to take into account the bidirectional training.
Architectures of our networks, and training hyper parameters are explained in detail in appendix A.
Breiﬂy, we used Conditional Normalization (Dumoulin et al., 2016; Perez et al., 2017) for conditioning the view point (or camera pose) in the encoder, decoder and the discriminator networks. The view point is a three dimensional vector representing positional co-ordinates of the camera.
In our training, the afﬁne parameters of the Batch-Normalization layers (Ioffe & Szegedy, 2015) are replaced by learned representations based on the view point.
The ﬁnal objective includes a bi-directional reconstruction loss as formulated in Equation 4. This in-turn enforces the reconstructions from the model to stay close to the corresponding inputs.
We also use a reconstruction error in the objective function for the encoder and decoder networks as it has been empirically shown to improve reconstructions in ALI-type models Li et al. [Equation]
where function rend ( · ) denotes rendered image on the decoder side. [Figure] [Figure] [Table]
E XPERIMENTS 4. E XPERIMENTAL S ETUP
Tasks. Our model is capable of both reconstructing 3D scenes and generate new ones. We evaluate the 3D understanding capability of the model on 3D-IQTT: a spatial reasoning based semi-supervised classiﬁcation task.
The goal of the 3D-IQTT is to quantify ability of our model to perform 3D spatial reasoning test by using large amounts of the unlabeled training data and considerably small set of labelled examples. Evaluation. In order to evaluate the 3D reconstruction ability of the model we used Hausdorff distance (Taha & Hanbury, 2015) and MSE. Hausdorff distance measures the model’s 3D reconstruction’s correspondence with the input for a given camera pose.
We measure the correctness of the recovered depth using standard MSE with respect to ground truth depth maps. We evaluate the 3D generation generation qualitatively.
Finally the evaluation metric for the 3D-IQTT is the percentage of correctly answered questions . Datasets. We have created multiple different scene datasets ranging from simple to complex in nature. Those scenes are composed by a room containing one or more objects placed at random positions and orientations.
Each 3D scene is rendered into a single 128 × 128 × 3 image taken from a camera in a random sampled uniformly on the positive octant of a sphere containing the room.
Technically, the probability of seeing the same conﬁguration of a scene from two different views is near zero.
Box scenes is created with simple box 3D shape (as depicted in Figure 14).
Shape scenes is created with basic 3D shapes (i.e., box, sphere, cone, torus, teapot etc).
ShapeNet scenes is composed by 6 objects from the ShapeNet dataset (Chang et al., 2015) (i.e., bowls, bottles, mugs, cans, caps and bags).
For 3D-IQTT task we generated a test where each IQ question instance consists of a reference image of Tetris-like shape, as well as 3 other images, one of which is a randomly rotated version of the reference (see Figure 10 for an example).
The training set is formed by 200k questions where
[Table] only a few are labelled with the information about the correct answer (i [Table] e., either 5% (10k) or 0 .
5% (1k) of the total training data).
The validation and test sets each contain 100K labelled questions.
More details on experimental setup and evaluation can be found in appendix B.
4. I MPLICIT 3D S CENE R ECONSTRUCTION
Figure 4 shows the input Shape scenes data and its corresponding reconstructions, along with its recovered depth and normal maps.
The depth map is encoded in such a way that the darkest points are closer to the camera.
The normal map colors correspond to the cardinal directions (red/green/blue for x/y/z axis respectively).
Table 1 shows a quantitative evaluation of the forward and reverse Hausdorff distances on three different datasets. The table also depicts mean squared error of the generated depth map with respect to the input depth map. Figure 6 shows the reconstructions from the model on more challenging multiple-shape scenes where the number of objects as well as their shape varies. Figure 16 in the appendix showcases more qualitative evaluations.
To showcase that our model can reconstruct unobserved views, we infer the latent code z of an image x and then we decode and render different views while rotating the camera around the scene.
Tableshows the Hausdorff distance and MSE of reconstructing a scene from different unobserved view angles.
As the view angle increases from 0 ◦ (original) to 80 ◦ for shape scenes the reconstruction Error and MSE tend to increase. However, for the multiple-shape scenes setup the trend is not as clear because of the complexity of the scene an the inter-object occlusions. Figure 5 qualitatively shows how pix2scene correctly infers the extents of the scene not in view in a consistent manner, demonstrating true 3D understanding of the scene.
[Figure]
4. I MPLICIT 3D S CENE G ENERATION
We trained pix2scene on scenes composed of ShapeNet objects.
Figure 7 shows qualitative results on unconditional generation. This shows how our model is able to generate correct 3D interpretations of the world.
We also trained our model conditionally by giving the class label of the ShapeNet object to the decoder and critic networks (Mirza & Osindero, 2014).
Figure 8 shows the results of conditioning the generator on different target classes.
In order to explore the manifold of the learned representations we select two images x 1 and x 2 from the held out data, then we linearly interpolate between their encodings z 1 and z 2 and decode the intermediary points into their corresponding images.
Figure 9 shows this for two different settings.
In each case, our representations capture the major geometrical aspects of the scene.
4. 3D-IQ T EST T ASK
We have designed a quantitative evaluation for 3D reconstruction which we refer to as the 3D-IQ test task (3D-IQTT).
In their landmark work, Shepard & Metzler (1971) introduced the mental rotation task into the toolkit of cognitive assessment. The authors presented human subjects with reference images and answer images and the subjects had to quickly decide if the answer was either a 3D-rotated version or a mirrored version of the reference. The speed and accuracy with which people can solve this mental rotation task has since become a staple of IQ tests like the Woodcock-Johnson test (Woodcock et al., 2001).
We took these as inspiration when designing a quantitative evaluation: we are using the same kind of 3D objects but instead of confronting our model with pairs of images [Figure]
and only two possible answers, we include several distractor answers and the subject (human or computer) has to to pick the correct answers out of 3 that is a 3D-rotated version of the reference object.
To verify that our approach is able to learn accurate embeddings of these shapes we ﬁrst assessed the reconstruction of these shapes qualitatively as shown in Figure 11.
4.4. S EMI - SUPERVISED CLASSIFICATION ON THE 3D-IQ T EST T ASK
For training pix2scene in a semi-supervised setting, in addition to the unlabeled data, we also used the labelled data.
The training with the unlabeled samples differs from the approach described for previous experiments as we do not assume that we have the knowledge of camera position.
Thus, part of the latent vector z encodes the actual 3D object (denoted as z scene ) and the remainder estimates the camera-pose (denoted as z view ).
For the supervised training two additional loss terms were added: (a) a loss that enforces the object component ( z scene ) to be the same for both the reference object and the correct answer, (b) a loss that maximizes the distance between object component of reference and the distractors.
Losses (a) and (b) are contained in Equation 5 where d i denotes the distractors, x ref is the reference and x ans the correct answer.
Algorithm is detailed in Table 1.
[Equation]
i =
D θ ( x ref , x d i ) where D θ ( x 1 , x 2 ) =
( || z x 1 scene − z x 2 scene || 2 ) 2 and z x = Encoder θ ( x ) (5)
During the training we also minimize the mutual information between z scene and z view to explicitly disentangle and make sure that the learnt latent code has distinct source of information present in its dimensions.
This is implemented via MINE (Belghazi et al., 2018).
The strategy of MINE is to parameterize a variational formulation of the mutual information in terms of a neural network:
[Equation]
This objective is optimized in an adversarial paradigm where T , the statistics network, plays the role of the critic and is fed with samples from the joint and marginal distribution.
We added this loss to our pix2scene objective to minimize the mutual information estimate in both unsupervised and 1: while iter < max iter do 2:
D ← MiniBatch () 3: z ∼ E ( x ref ); ∀ ( x ref , x d 1 , x d 2 , x ans ) ∈ D 4:
L ← L ALI + L recon + I Θ ( z scene , z view ) 5:
if supervised-training-interval ( iter ) then 6:
L ← L + L θ ( x ref , x d 1 , x d 2 , x ans ) 7: end if 8: optimize networks with L 9: end while
[Table] supervised training iterations. Once the model is trained, we answer 3D-IQTT questions, by inferring the latent 3D representation for each of the 4 images and we select the answer closest to the reference image as measured by L2 distance.
We compared our model to two different baselines.
The ﬁrst one is composed of 4 ResNetmodules (He et al., 2016) with shared weights followed by 3 fully-connected layers.
We trained this CNN only on the labeled samples.
Our second baseline has a similar architecture as the previous one but the fully-connected layers were removed.
Instead of the supervised loss provided in the form of correct answers, it is trained on the contrastive loss (Koch et al., 2015).
This loss reduces the feature distance between the references and correct answers and maximizes the feature distance between the references and incorrect answers.
A more detailed description of the networks and contrastive loss function can be found in the appendix D.
Table 3 shows 3D-IQTT results for our method and baselines.
The baselines were not able to interpret the underlying 3D structure of the data and its results are only slightly better than a random guess.
The poor performance of the Siamese CNN might be in part because the contrastive loss rewards similarities in pixel space and has no notion of 3D similarity.
However, pix2scene achieved signiﬁcantly better accuracy by leveraging the learned 3D knowledge of objects.
C ONCLUSIONS
In this paper we proposed a generative approach to learn 3D structural properties from single images in an unsupervised and implicit fashion. Our model receives an image of a scene with uniform material as input, estimates the depth of the scene points and then reconstructs the input scene.
We also provided quantitative evidence that support our argument by introducing a novel IQ-task in a semi-supervised setup. We hope that this evaluation metric will be used as a standard benchmark to measure the 3D understanding capability of the models across different 3D representations.
The main drawback of our current model is that it requires the knowledge of lighting and material properties.
Future work will focus on tackling the more ambitious setting of learning complex materials and texture along with modelling the lighting properties of the scene.
A A RCHITECTURE
Pix2scene is composed of an encoder network(See Table 4), a decoder network(See Table 5), and a critic network(See Table 6).
Speciﬁcally, the decoder architecture is similar to the generator in
DCGAN (Radford et al., 2015) but with LeakyReLU (Mikolov et al., 2011) as activation function and batch-normalization (Ioffe & Szegedy, 2015).
Also, we adjusted its depth and width to accommodate the high resolution images accordingly.
In order to condition the camera position on the z variable, we use conditional normalization in the alternate layers of the decoder.
We train our model for 60K iterations with a batchsize of 6 with images of resolution 128 × 128 × 3 . [Table] [Table]
[Table]
Material. In our experiments, we use diffuse materials with uniform reﬂectance.
The reﬂectance values are chosen arbitrarily and we use the same material properties for both the input and the generator side.
Camera. The camera is speciﬁed by its position, viewing direction and vector indicating the orientation of the camera. The camera positions were uniform randomly sampled on a sphere for the 3D-IQTT task and on a spherical patch contained in the positive octant, for the rest of the experiments. The viewing direction was updated based on the camera position and the center of mass of the objects, so that the camera was always looking at a ﬁxed point in the scene as its position changed.
The focal
[Figure]
length ranged between [18 mm and 25mm] in all the experiments.
and ﬁeld-of-view was ﬁxed to 24mm .
The camera properties were also shared between the input and the generator side.
However, in the 3D-IQTT task we relax the assumption that we know the camera pose and instead estimate the view as a part of the learnt latent representation.
Lights.
For the light sources, we experimented with single and multiple point-light sources, with the light colors chosen randomly.
The light positions are uniformly sampled on a sphere for the 3D
IQTT tasks, and uniformly on a spherical patch covering the positive octant for the other scenes.
The same light colors and positions are used both for rendering the input and the generated images.
The lights acted as a physical spot lights with the radiant energy attenuating quadratically with distance.
As an ablation study we relaxed this assumption of having perfect knowledge of lights by using random position and random color lights.
Those experiments show that the light information is not needed by our model to learn the 3D structure of the data.
However, as we use random lights on the generator side, the shading of the reconstructions is in different color than in the input as shown in
Figure 12.
C E VALUATION OF 3D RECONSTRUCTIONS
For evaluating 3D reconstructions, we use the Hausdorff distance (Taha & Hanbury, 2015) as a measure of similarity between two shapes as in Niu et al.
Given two point sets, A and B , the Hausdorff distance is, max (cid:8) max D + H
( A, B ) , max D + H ( B, A ) (cid:9) , where D + H is an asymmetric
Hausdorff distance between two point sets.
E.g., max D + H ( A, B ) = max D ( a, B ) , for all a ∈ A , or the largest Euclidean distance D ( · ) , from a set of points in A to B , and a similar deﬁnition for the reverse case max D + H ( B, A ) .
D A RCHITECTURE FOR 3D IQTT E VALUATIONS
Pixel2Scene architecture remains similar to the ones in previous sections but with higher capacity on decoder and critic as this task is more challenging and complex.
The more important difference is that for those experiments we do not condition the networks with the camera pose to be fair with the baselines.
In addition to the three networks we have a statistics network (see Table 7) that estimates and minimizes the mutual information between the two set of dimensions in the latent code using
MINE (Belghazi et al., 2018).
Out of 128 dimensions for z we use ﬁrst 118 dimensions for represent scene-based information and rest to encode view based info.
The architecture of the baseline networks is shown in Figure 13.
The contrastive loss using for training this baselines is shown in Figure 7.
[Table]
The contrastive loss from Equation 7 is applied to the 2048 features that are generated by each ResNet block.
x 1 and x 2 are the input images, y is either
(if the inputs are supposed to be the same) or(if the images are supposed to be different), G θ is each ResNet block, parameterized by θ , and m is the margin, which we set to 2 . 0 .
The loss function is from Hadsell et al.
but used slightly differently.
[Equation]
E M ORE S CENE R ECONSTRUCTIONS
Figure 14 shows 3D reconstructions of scenes formed by boxes in a room. In Figure 15 our model is asked to reconstruct the scenes of the ﬁrst column and then render different views of the same scene.
In this case we show the normal maps of those views.
Figure 16 shows the recovered shading, depth and normal images from reconstructions of complex scenes such as bedrooms and bunny.
[Figure]
[Figure]