U NSUPERVISED O BJECT -O RIENTED 3D S CENE R EPRESENTATION AND R ENDERING
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we propose a probabilistic generative model, called ROOTS, for unsupervised learning of object-oriented 3D-scene representation and rendering.
ROOTS bases on the Generative Query Network (GQN) framework.
However, unlike GQN, ROOTS provides independent, modular, and object-oriented decomposition of the 3D scene representation.
In ROOTS, the inferred object-oriented representation is 3D in the sense that it is 3D-viewpoint invariant as the scene-level representation of GQN is so.
ROOTS also provides hierarchical object-oriented representation: at 3D global-scene level and at 2D local-image level.
In experiments, we demonstrate on datasets of 3D rooms with multiple objects, the above properties by focusing on its abilities for disentanglement, compositionality, transferability, and generalization.
ROOTS achieves this without performance degradation on generation quality in comparison to GQN.
I NTRODUCTION
The shortcomings of contemporary deep learning such as interpretability, sample efﬁciency, ability for reasoning and causal inference, transferability, and compositionality, are where the symbolic AI has traditionally shown its strengths (Garnelo & Shanahan, 2019).
Thus, one of the grand challenges in machine learning has been to make deep learning embrace the beneﬁts of symbolic representation so that symbolic entities can emerge from high-dimensional observations such as visual scenes.
In particular, for learning from visual observations of the physical world, such representation should consider the following criteria.
First, it should focus on objects (and their relations ) which are foundational entities constructing the physical world.
These can be considered as units on which we can build a modular model.
The modular nature also helps compositionality (Andreas et al., 2016) and transferability (Kansky et al., 2017).
Second, being three-dimensional (3D) is a decisive property of the physical world.
We humans, equipped with such 3D representation in our brain (Yamane et al., 2008), can retain consistency on the identity of an object even if it is observed from different viewpoints.
Lastly, learning such representation should be unsupervised .
Although there have been remarkable advances in supervised methods to object perception (Redmon et al., 2016; Ren et al., 2015; Long et al., 2015), the technology should advance toward unsupervised learning as we humans do.
This not only avoids expensive labeling efforts but also allows adaptability and ﬂexibility to the evolving goals of various downstream tasks because “objectness” itself can vary on the situation.
In this paper, we propose a probabilistic generative model that can learn, without supervision, objectoriented 3D representation of a 3D scene from its partial 2D observations.
We call the proposed model ROOTS (Representation of Object-Oriented Three-dimensional Scenes).
We base our model on the framework of Generative Query Networks (GQN) (Eslami et al., 2018; Kumar et al., 2018).
However, unlike GQN which provides only a scene-level representation that encodes the whole 3D scene into a single continuous vector, the scene representation of ROOTS is decomposed into objectwise representations each of which is also an independent, modular, and 3D representation.
Further,
ROOTS learns to model a background representation separately for the non-object part of the scene.
The object-oriented representation of ROOTS is more interpretable, composible, and transferable.
Besides, ROOTS provides the two-level hierarchy of the object-oriented representation: one for a global 3D scene and another for local 2D images.
This makes the model more interpretable and provides more useful structure for downstream tasks.
In experiments, we show the above abilities of ROOTS on the 3D-Room dataset containing images of 3D rooms with several objects of different colors and shapes.
We also show that these new abilities are achieved without sacriﬁcing generation quality compared to GQN.
Our proposed problem and method are signiﬁcantly different from existing works on visual 3D learning although some of those partly tackle some of our challenges.
First, our model learns factorized object-oriented 3D representations which are independent and modular, from a scene containing multiple objects with occlusion and partial observability rather than a single object.
Second, our method is unsupervised, not using any 3D structure annotation such as voxels, cloud points, or meshes as well as bounding boxes or segmentation annotation.
Third, our model is a probabilistic generative model learning both representation and rendering with uncertainty modeling.
Lastly, it is trained end-to-end.
In Section 4, we provide more discussion on the related works.
The main contributions are: (i)
We propose, in the GQN framework, a new problem of learning object-oriented 3D representations of a 3D scene containing multiple objects with occlusion and partial observability in the challenging setting described above.
(ii) We achieve this by proposing a new probabilistic model and neural architecture.
(iii)
We demonstrate that our model enables various new abilities such as compositionality and transferability while not losing generation quality.
P RELIMINARY : G ENERATIVE Q UERY N ETWORKS
The generative query networks (GQN) (Eslami et al., 2018) is a probabilistic generative latentvariable model providing a framework to learn a 3D representation of a 3D scene.
In this framework, an agent navigating a scene i collects K images x ki from 2D viewpoint v ki .
We refer this collection to context observations
C i = { ( x ki , v ki ) } k
=1 ,...,K .
While GQN is trained on a set of scenes, in the following, we omit the scene index i for brevity and discuss a single scene without loss of generality.
GQN learns scene representation z from context C .
The learned representation z of GQN is a 3Dviewpoint invariant representation of the scene in the sense that, given an arbitrary query viewpoint v q , its corresponding 2D image x q can be generated from the representation.
In the GQN framework, there are two versions.
The standard GQN model (Eslami et al., 2018) uses the query viewpoint to generate representation whereas the Consistent GQN (CGQN) (Kumar et al., 2018) uses the query after generating the scene representation in order to obtain queryindependent scene-level representation.
Although we use CGQN as our base framework to obtain query-independent scene-level representation, in the rest of the paper we use the abbreviation GQN instead of CGQN to indicate the general GQN framework embracing both GQN and CGQN.
The generative process of GQN is written as follows:
p ( x q | v q , C ) = (cid:82) p ( x q | v q , z ) p ( z | C ) d z .
As shown, GQN uses a conditional prior p ( z | C ) to learn scene representation z from context.
To do this, it ﬁrst obtains a neural scene representation r from the representation network r = f repr-gqn ( C ) which combines the encodings of ( v k , x k ) ∈ C in an order-invariant way such as sum or mean.
It then uses ConvDRAW (Gregor et al., 2016) to generate the scene latent variable z from scene representation r by p ( z | C ) =
(cid:81) l =1: L p ( z l | z <l , r ) = ConvDRAW ( r ) with L autoregressive rollout steps.
Due to intractability of the posterior distribution p ( z | C, v q , x q ) , GQN uses variational inference for posterior approximation and the reparameterization trick (Kingma & Welling, 2013) for backpropagation through stochastic variables.
The objective is to maximize the following evidence lower bound (ELBO) via gradient-based optimization.
[Equation]
, v q , x q ) (cid:107) p θ ( z | C )) .
Note that although in this paper we use a single target observation D = ( x q , v q ) for brevity, the model is in general trained on a set of target observations
D = { ( x qj , v qj ) } j .
ROOTS: R EPRESENTATION OF O BJECT -O RIENTED 3D S CENES 3. G ENERATIVE P ROCESS
The main difference of our model from GQN is that we have a 3D representation per object present in the target 3D space while GQN has a single 3D representation compressing the whole 3D space into a vector without object-level decomposition.
We begin this modeling by introducing the number of objects M in the target space as a random variable.
Then, we can write the representation
[Table]
[Figure]
[Table]
Rendering
[Table]
[Figure]
[Table]
[Figure]
prior of ROOTS as p ( z , M | C ) =
p ( M | C ) (cid:81)
Mm =1 p ( z ( m ) | C ) .
To implement such a model with a variable number of objects, in AIR (Eslami et al., 2016), the authors proposed to use an RNN that rolls out M steps, processing one object per step.
However, according to our preliminary investigation (under review) and other works (Crawford & Pineau, 2019), it turned out that this approach is computationally inefﬁcient and shows severe performance degradation with growing M .
Object-Factorized Conditional Prior.
To resolve this problem, in ROOTS we propose to process objects in a spatially local and parallel way instead of sequential processing.
This is done by ﬁrst introducing the scene-volume feature-map .
The scene-volume feature-map is obtained by encoding context C into a 3D tensor of N =
( H × W × L ) cells.
Each cell n ∈ { 1 , . . .
, N } is then associated to
D -dimensional volume feature r n ∈ R D .
Thus, the actual output of the encoder is a 4-dimensional tensor r = f repr-scene ( C ) .
Each volume feature r n represents a local 3D space in the target 3D space in a similar way that a feature vector in a 2D feature-map of CNN models a local area of an input image.
Note, however, that the introduction of the scene-volume feature-map is not the same as the feature-map of 2D images because, unlike the CNN feature-map for images, the actual 3D target space is not directly observable—it is only observed through a proxy of 2D images.
For the detail implementation of the encoder f repr-scene , refer to the Appendix A.3.
Given the scene-volume feature-map, for each volume cell n = 1 , . . .
, N but in parallel, we obtain three latent variables ( z presn , z posn , z whatn ) =
z n from the 3D-object prior model p ( z n | r n ) .
We refer this collection of object latent variables z
= { z n } N n =1 to the scene-object latent-map as z is generated from the scene-volume feature-map r .
Here, z presn is a Bernoulli random variable indicating whether an object is associated (present) to the volume cell or not, z posn is a 3-dimensional coordinate indicating the position of an object in the target 3D space, and z whatn is a representation vector for the appearance of the object.
We defer a more detail description of the 3D-object prior model p ( z n | r n ) to the next section.
Note that in ROOTS we obtain z whatn as a 3D representation which is invariant to 3D viewpoints.
The position and appearance latents for cell n are deﬁned only when the cell has an associated object to represent, i.e., z presn = 1 .
From this modeling using scenevolume feature-map, we can obtain M = (cid:80) n z presn and the previous prior model can be written as follows:
[Equation]
In addition to allowing spatially parallel and local object processing, another key idea behind introducing a presence variable per volume cell is to reﬂect the inductive bias of physics: two objects cannot co-exist at the same position.
This helps remove the sequential object processing because dealing with an object does not need to consider other objects if their features are from spatially distant areas.
Note also that the scene-volume feature-map is not to strictly partition the target 3D space and that the presence variable represents the existence of the center position of an object not the full volume of an object.
Thus, information about an object can exist across neighboring cells.
Hierarchical Object-Oriented Representation.
The object-oriented representations z = { z n } provided by the above prior model is global in the sense that it contains all objects in the whole target 3D space, independently to a query viewpoint.
From this global representation and given a query viewpoint, ROOTS generates a 2D image corresponding to a query viewpoint.
This is done ﬁrst by learning the view-dependent representation of the target image.
In a naive approach, this may be done simply by learning a single vector representation p ( z q | z , v q ) but in this case, we lose important information: the correspondence between a rendered object in the image and a global object representation z n .
That is, we cannot track from which object representation z n an object in the image is rendered.
In ROOTS, we resolve this problem by introducing local 2D-level objectoriented representation layer.
This local object-oriented and view-dependent representation allows additional useful structure and more interpretability.
This 2D local representation is similar to those in AIR (Eslami et al., 2016) and SPAIR (Crawford & Pineau, 2019).
Speciﬁcally, for n for which z presn = 1 , a local object representation s n is generated by conditioning on the global representation set z and the query v q .
Our local object representation model is written as: p ( s | z , v q ) =
(cid:81)
Nn =1 p ( s n | z , v q ) .
Similar to the decomposition of z n , local object representation s n consists of ( s presn , s posn , s scalen , s whatn ) .
Here, s presn indicates whether an object n should be rendered in the target image from the perspective of the query.
Thus, even if an object exists in the target 3D space, i.e., z presn = 1 , s presn can be set to zero if that object should be invisible from the query viewpoint.
Similarly, s posn and s scalen represent respectively the position and scale of object n in the image not in the 3D space, and s whatn represents the appearance to be rendered into the image (thus not 3D invariant).
For more details about how to obtain ( s presn , s posn , s scalen , s whatn ) from z and v q , we describe in the next section.
Given s = { s n } , we then render to the canvas to obtain the target image p ( x q | s ) .
Combining all, the generative process of ROOTS is written as follows:
[Equation]
See Figure 1 for the overview of the generation process.
3. I MPLEMENTATION D ETAILS
Global 3D-object prior.
The 3D-object prior p ( z n | r n ) generates three latents ( z presn , z posn , z whatn ) as follows.
It ﬁrst obtains the presence latent from z presn ∼ Bernoulli ( f pres ( r n )) and the 3-dimension position latent from z posn ∼ N ( f µpos ( r n ) , f σpos ( r n )) .
Using these two latents, we then obtain the appearance latent z whatn .
This process is divided into object gathering and object encoding.
For object gathering, for each context image x k we attend and crop a patch that corresponds to object n .
Speciﬁcally, we ﬁrst notice that using the deterministic camera-coordinate projection function f 3D → 2D (which we do not learn), we can project from the perspective of a context viewpoint v k , a 3D position z posn in the global 3D coordinate system into a 2D position u posk,n in context image x k , i.e., u posk,n = f 3D → 2D ( z posn , v k ) .
If object n should be invisible from the viewpoint v k , its projected 2D position u posk,n is out of the image x k
and we do not crop a patch.
For more details about the projection function f 3D → 2D , refer to Wikipedia (2019).
We also predict the bounding box scale u scalek,n = f scale ( u posk,n , r n , v k ) .
Given the center position and scale, we can crop a patch x kn ⊂ x k using the spatial transformer (Jaderberg et al., 2015).
Applying this cropping to all context pairs ( v k , x k ) ∈ C , we gather a set of object image-patches X n = { x kn } Kk =1 for all n with z presn = 1 .
Given object image-patches X n , obtaining object 3D-representation z whatn can be converted to a
GQN encoding problem.
That is, we can simply consider X n and its corresponding viewpoints as a new object-level context C n , and can run an order-invariant GQN representation network r whatn =
f repr-obj ( C n ) and then run ConvDRAW to obtain z whatn from r whatn .
Local 2D-object prior.
The intermediate prior p ( s n | z , v q ) generates ( s presn , s posn , s scalen , s whatn ) .
This is done as follows.
Similar to what we described in the above to obtain u posk,n , we use the coordinate projection function f 3D → 2D to ﬁnd the position of a global object z n in the target image from the perspective of query v q , i.e., s posq,n = f 3D → 2D ( z posn , v q ) .
Thus, we model the position as a deterministic variable.
The scale s scaleq,n is also obtained similarly as described in the above for u scalek,n , but with random sampling.
Then, we predict s presn ∼
Bern ( f pres ( s posq,n , s scaleq,n , z n , v q )) .
If object n should be visible in the 2D target image, i.e., s presn = 1 , we generate the 2D appearance representation s whatn by using an object-level GQN decoder based on ConvDRAW, i.e., s whatn =
ConvDRAW ( z whatn , v q ) .
Finally, we have p ( s n | z , v q ) =
p ( s posn | z
posn , v q ) p ( s scalen | s posn , z whatn , v q ) p ( s presn | s posn , s scalen , z n , v q ) .
Rendering to 2D Canvas.
A main challenge in rendering the local representation s = { s n } into the image canvas, i.e., p ( x q | s ) , is to deal with occlusion.
In ROOTS, this can easily be achieved by noticing (i) that the coordinate conversion f 3D → 2D ( z posn , v q ) actually converts a 3D-coordinate to another 3D-coordinate and (ii) that then the last dimension of the converted coordinate system can be interpreted as the orthogonal projection distance from the viewpoint v q to the object’s position z posn .
We can use this distance as object depth from the observer’s perspective.
This allows us to sort objects according to their depths and render each object accordingly without occlusion.
To handle background, ROOTS has an independent module to infer the background separately at imagelevel.
We also found that learning an object-level mask along with the appearance latent s whatn helps segmentation between foreground and background as well as generating less blurry images.
More details on related implementation are provided in Appendix A.4.
3. L EARNING AND I NFERENCE
Due to the intractability of the posterior p ( z , s | C, D ) with D = { ( v qj , x qj ) }
j the target viewpointimage pairs, we train ROOTS using variational inference with the following posterior approximation q φ ( z , s | C, D ) =
q φ ( z | C, D ) q φ ( s | z , C, D ) .
To compute the gradient w.r.t.
the continuous latent variables such as the position and appearance, we use reparameterization trick and for the discrete variables on the presence, we use a continuous relaxation using Gumbel-Softmax trick (Jang et al., 2016).
Other methods based on the REINFORCE algorithm (Williams, 1992; Tucker et al., 2017;
Grathwohl et al., 2017) can also be used.
Implementation of the approximate posterior q ( z | C, D ) and q ( s | z , C, D ) is made easier by sharing the parameters of q with that of conditional prior p ( z | C ) : we only need to provide additional data D .
With D =
( v q , x q ) for simplicity, the objective is to maximize the following evidence lower bound (ELBO):
L ( θ, φ ; C, D ) =
[Equation]
Combining with Unconditioned Prior.
One difﬁculty in using the conditional prior is the fact that, because the prior is learned, we have less control in reﬂecting our prior knowledge into the prior distribution.
In our experiments, it turns out that biasing the posteriors of some variables towards the values of our prior preference is helpful in stabilizing the model.
To implement this, in our training, we use the following objective that has additional KL terms between the posterior and unconditioned prior.
[Equation]
where z pres = { z presn } n and z pos , z scale , s pres are deﬁned similarly.
The ρ and γ are hyperparameters weighting the auxiliary loss terms.
We set them to 0 .
999 and 7 during training, respectively.
This auxiliary loss can be derived if we replace the conditional prior by the product of expert prior p ( z | C ) p ( z ) divided by the posterior q ( z | C ) .
R ELATED W ORKS
Although to our knowledge there has been no previous work on unsupervised and probabilistic generative object-oriented representation learning for 3D scenes containing multiple objects, there has been literature on its 2D problems.
The ﬁrst is the Attend, Infer, Repeat (AIR) model (Eslami et al., 2016).
AIR uses spatial transformer (Jaderberg et al., 2015) to crop an object patch and uses an RNN to sequentially generate next object conditioning on the previous objects.
In Crawford &
Pineau (2019), the authors showed that this RNN-based rollout is inefﬁcient and can signiﬁcantly degrade performance as the number of objects increases.
SPAIR is inspired by YOLO (Redmon et al., 2016), but unlike YOLO, it does not require bounding box labels.
The main limitation of
SPAIR is to infer the latents sequentially.
Neural Expectation Maximization (NEM) (Greff et al., 2017) considers the observed image as a pixel-level mixture of K objects and an image per object is generated and combined according to the mixture probability.
In Greff et al.
(2019), the authors proposed a more efﬁcient version of NEM, called IODINE, using iterative inference (Marino et al., 2018).
In MONET (Burgess et al., 2019), an RNN drawing a scene component at each time step is used.
Unlike AIR and SPAIR, the representations in NEM, IODINE, and MONET do not explicitly provide natural disentanglement like presence and pose per object.
There have been plenty amount of remarkable works about visual 3D learning from the computer vision community.
However, as pointed in Section 1, the problem setting and proposed model of these works are different from ours in the sense that they are either (i) not decomposing object-wise representations from a scene containing multiple objects (but working mostly on single object cases) (Wu et al., 2016; Yan et al., 2016; Choy et al., 2016; Kar et al., 2017;
Nguyen-Phuoc et al., 2019), (ii) supervised approach (Huang et al., 2018; Tulsiani et al., 2018; Cheng et al., 2018; Shin et al., 2019;
Du et al., 2018), (iii) learn image generation (synthesis) of a 3D scene without scene-representation (Sitzmann et al., 2019;
Kato & Harada, 2019; Pinheiro et al., 2019; Tulsiani et al., 2017) or learn scene-representation without learning rendering ability (Zhou et al., 2017; Yu & Wang, 2018), or (iv) not end-to-end.
Many of the more traditional works from 3D computer vision also relevant to our work but many of them are not based on neural networks and not end-to-end trainable.
E XPERIMENTS
We evaluate ROOTS quantitatively and qualitatively.
We train ROOTS with the same hyperparameters on all datasets.
We ﬁrst brieﬂy describe the datasets.
For more details of the dataset generation and network architecture, refer to Appendix A.5 and Appendix A.4.
We use MuJoCo, (Todorov et al., 2012), to simulate 3D scenes.
Speciﬁcally, we generate three different datasets of scenes with 1 - 3 objects, 2 - 4 objects and 3 - 5 objects, respectively.
We set the image size to 64 × 64 × 3 pixels.
For each object, we randomly choose its position, shape, size, and color.
Although we put all objects on the ﬂoor, we still predict the 3-dimension coordinate values for z posn because objects have different sizes.
We generate 60K different scenes for each dataset and split them into 50k for training, 5k for validation, and 5k for testing.
We use CGQN as the baseline implementation but in the rest of the section use the abbreviation ‘GQN’ instead of ‘CGQN’ to indicate the general GQN framework.
5. Q UALITATIVE E VALUATION
First, we compare the generations between ROOTS with GQN by visualizing several generated samples from the same scene under the same set of query viewpoints.
Note that the goal of this experiment is to show that achieving the object-wise representation in ROOTS does not deteriorate its generation quality in comparison to GQN.
As seen in Figure 2, ROOTS generates slightly sharper object boundaries while GQN generates more blurry images.
We believe that this is due to the object-wise generation using segmentation masks.
Then, to provide a further understanding of the advantages of object-orientated representation, we visualize decomposed generations from
ROOTS, as shown in Figure 3.A. We can see that ROOTS can generate clean background, complete objects, precise foreground, and detailed occlusion mask separately.
Furthermore, to demonstrate the viewpoint-invariant property of the global object representation z whatn , in Figure 3.B, we provide generations of two objects under different query viewpoints.
We can see that ROOTS recovers the object with pose corresponding to query viewpoints.
GQN cannot provide such decomposed generations because its representation is scene-level where objects are not decomposed.
Object-wise disentanglement.
Generations from random viewpoints after changing object positions.
In this section and the following sections, we use ROOTS trained on the 2 - 4 object dataset for demonstration unless otherwise stated.
To verify the disentanglement property of our objectoriented representation learned by ROOTS, we carry out the experiment of arbitrarily modifying z posn of one object in a scene.
As a good disentangled latent representation, this modiﬁcation should not affect the position, existence, and appearance of other objects concurrent in the scene while handling occlusion properly.
More importantly, we should be able to change either the x or y coordinate independently and this change should be consistent across viewpoints.
To demonstrate this,
[Figure] [Figure]
after changing one dimension of z posn of the red cylinder as shown in Figure we feed this modiﬁed ˆ z posn back to the model and follow the ROOTS generation model in the same way as we do during testing.
Generations under 4 different viewpoints are shown in Figure 4.
We can see that ROOTS has a strong ability to learn disentangled representation and occlusions have been handled well due to this advantage.
Compositionality.
As stated in earlier sections that a 3D scene can be decomposed into several independent objects, another advantage coming with object-orientated representations is that a new scene can be easily built up with selected object components.
Thus, by simple combination, we can build novel scenes.
To demonstrate this, we ﬁrst provide ROOTS with three sets of context images from three different scenes and save the learned object representations z for each scene,
z = { z posn , z whatn } for all n with z posn
= 1 .
Then we swap one object between the ﬁrst two scenes and add one additional object to the third scene, as shown in Figure 5.
We see that the object component learned by ROOTS is fully disentangled and can be reused to create new scenes.
Also, by adding one new object, we make a scene with 5 objects, which does not exist in the training dataset.
Another example shown a scene 9 objects is visualized in Figure 6, where we use ROOTS trained on the 1-3 objects dataset.
More details about this experiment can be found in Appendix
A.2.
Partial Observability.
We know that the successful generation of ROOTS, even with only partial observations provided, is coming from object gathering across viewpoints.
This is because if an object is invisible for one viewpoint, ROOTS can learn it from other viewpoints.
Here, we want y x yx -  yx -
+ to push the partial observability to the extreme case, where one object is totally invisible for all context viewpoints.
In this case, ROOTS should not be able to correctly predict its existence, just like humans observing the true physical world.
To show this, we manually select some images from a scene that has one object missing served as context and the rest served as targets for ROOTS to generate.
We show the results in Figure 7.
5. Q UANTITATIVE E VALUATION
NLL and MSE.
In this section, we compare the quantitative results of ROOTS and GQN on the negative log-likelihood (NLL) and the mean squared error (MSE) on the test dataset.
The goal is to show quantitatively that achieving the object-wise representation in ROOTS does not deteriorate its generation quality in comparison to GQN.
We approximate NLL using importance sampling with
K = 50 samples and report the image NLL normalized by the number of pixels in Table ?? .
Both
GQN and ROOTS are trained for 120 epochs on each dataset, making sure both ROOTS and GQN have converged.
We see that, although learning object-factorized representations, ROOTS achieves
NLL comparable to GQN.
Due to object-wise representation, ROOTS recovers objects separately, together with an independent background module, ROOTS performs better on MSE than GQN.
This beneﬁt becomes clearer as the number of objects in the scene grows.
[Table]
Object Detection Quality.
In this section
[Table]
we only provide precision and recall results as object detection evaluation of ROOTS as GQN cannot detect objects.
To estimate the true positive prediction
[Table]
we ﬁrst ﬁlter out predictions that are too far away from any ground truth by applying a
[Figure]
radius threshold.
The distance is calculated as the euclidean distance between two center points.
Then we assign the predicted object to the ground truth object to which it has the nearest distance.
Multi-association is not allowed.
We normalize the coordinate value to be in the range of [ − 1 , 1] for better interpretability of the applied radius threshold.
Accordingly, the average object size is 0 . 3 .
We provide the precision and recall result under different thresholds in Table ?? .
Besides precision and recall, we also provide the counting accuracy of ROOTS.
[Table]
C ONCLUSION
We proposed ROOTS, a probabilistic generative model for unsupervised learning of 3D scene representation and rendering.
ROOTS can learn object-oriented interpretable and hierarchical 3D scene representation.
In experiments, we showed the generation, decomposition, and detection ability of
ROOTS.
For compositionality and transferability, we also showed that, due to the factorization of structured representation, new scenes can be easily built up by reusing components from 3D scenes.
Interesting future directions would be to learn the knowledge of the 3D world in a sequential manner as we humans keep updating and inferring knowledge through time.
R
A A PPENDIX A. G ENERALIZATION
In this section, we quantitatively analyze ROOTS’s generalization performance on tasks of generation and detection.
For the task of generation, we provide a comparison between ROOTS and GQN.
Speciﬁcally, we train both ROOTS and GQN on our three datasets until fully converged.
For each model trained on one dataset, e.g 1-3 objects dataset, we test it on the other two datasets, e.g 2objects dataset, and 3-5 objects dataset.
For the task of detection, due to the scene-level 3D representation of GQN (detection is not possible), we only report detection performance from ROOTS.
Generation and detection results are shown in Table 3 and Table 4, respectively.
Precision and recall are reported with the radius threshold set to 0 .
When training on small (e.g., 1-3) number of objects while testing on large (e.g., 3-5) number of objects , both ROOTS and GQN generalize not well.
This is because, during testing, latent variables are sampled from prior trained during the training stage.
If during training, the model isn’t provided a chance to see scenes with more objects, it is hard to generalize well.
Also, we note that, compared with GQN, ROOTS obtains larger MSE error.
Together with observations in the following case, we hypothesis this sensitivity comes from the discrete variable, presenting the existence of an object.
When training on a large number of objects while testing on a small number of objects , ROOTS and GQN show a similar property in their performance.
They both perform well.
But, in this case, ROOTS has better generations than
GQN, which, we believe, is beneﬁted from the object-wise generation and a separate background module.
As, in GQN, all information are compressed into one vector.
The above conclusion of
ROOTS is consistent with ROOTS’s performance on detection task.
One interesting ﬁnding during this experiment is that ROOTS trained on 3-5 objects dataset achieves better results (lower MSE, higher precision, and recall) than trained on 1-3 objects dataset when testing on 1-3 objects dataset.
A similar phenomenon is found when testing on 2-4 objects dataset.
This is an interesting ﬁnding that is worth more investigation.
[Table] [Table]
A. C OMPOSING WITH MORE OBJECTS
In this section
[Table]
we provide one more example of compositionality.
We ﬁrst train ROOTS on 1objects dataset.
As in Figure 8, during testing, we collect object representations from 7 different scenes and reuse them to composite a new scene with 9 objects.
Note that the sampled center position of each object and query viewpoints (represented as camera position, pitch, and roll) fed into ROOTS are with respect to canonical coordinates.
Note that, to predict the scale of objects in 2D projection correctly (e.g., for the same object, the larger the distance between it and viewpoint is, the smaller it should be in the 2D projection), ROOTS learns to infer local depth (position translation) for each object given speciﬁc query viewpoint.
This can be observed on the yellow sphere, green cylinder and blue cube in the bottom row in Figure 8.
A. D ETAILS OF COMPONENT MODULES
We ﬁrst introduce some important building-blocks for implementing our model in this section, then sketch the implementation steps using modules described here in the following section.
Scene Representation Network: We use Scene Representation Network to implement the order invariant encoder at scene level, f repr-scene ( · ) .
This network is modiﬁed based on Representation
Network in GQN.
We adjust kernel size and add a CNN3D layer to make sure the output convfeatures ﬁt our needs, shown as in Figure 9.
[Figure]
Con(cid:89)T(cid:85)an(cid:86)3Dk
[Figure]
[Table]
/eb1EB+deykScaCrI7FCQcKQjlBWABkxSovnEEEwkM1kRGWGJiTY1lfMSLjKcfn95nrSPa85JrX5TrzYuizpKsA8HcAQOnEEDrqEJLSAQwyM8w4uVWE/Wq/U2G12wip09+AXr/QuZ3ZIx</latexit> v k <latexit sha1_base64="F372poshxzIl8YmKllM2bdB1tx0=">AAAB83icbVDLSsNAFL3xWeur6tLNYBFclUSLj13RjcsK9gFNLJPppB06mYSZSaGE/oYbF4q49Wfc+TdO0iBqPTBwOOde7pnjx5wpbduf1tLyyuraemmjvLm1vbNb2dtvqyiRhLZIxCPZ9bGinAna0kxz2o0lxaHPaccf32R+Z0KlYpG419OYeiEeChYwgrWRXDfEeuQH6WT2MO5XqnbNzoEWiVOQKhRo9isf7iAiSUiFJhwr1XPsWHsplpoRTmdlN1E0xmSMh7RnqMAhVV6aZ56hY6MMUBBJ84RGufpzI8WhUtPQN5NZRvXXy8T/vF6ig0svZSJONBVkfihIONIRygpAAyYp0XxqCCaSmayIjLDERJuaynkJVxnOv7+8SNqnNeesVr+rVxvXRR0lOIQjOAEHLqABt9CEFhCI4RGe4cVKrCfr1Xqbjy5Zxc4B/IL1/gWWz5Iv</latexit>
Object Representation Network: We use object Representation Network to implement order invariant encoder at object level, f repr-obj ( · ) .
As visualized in Figure 10, we design a branch to pass low level feature to provide richer conv-features.
The dimension d in the second input, which is concatenation of { z pres , s scale , s pos } and parameters of STN, will be clariﬁed in Table 5.
Convolutional LSTM Cell: We use ConvLSTM as the RNN module in the ConvDRAW module.
In ConvLSTM, all fully-connected layers are substituted with convolutional layers.
Its one updating step is described as follows:
[Equation]
where h i is the output of the cell and c i is the recurrent state of the
ConvLSTM, x i is the input.
Both h
i and c i are initialized with zeros at the i = 0 step.
ConvDRAW: We highlight one step (denoted as l ) ConvDRAW used in ROOTS here for generative process and inference process separately.
[Table]
[Figure]
• Generative Process:
[Figure]
, h ( l ) p , c ( l ) p ) z
( l +1) ∼ StatisNet ( h ( l +1) p ) • Inference Process: ( h ( l +1) q , c ( l +1) q ) ← ConvLSTM φ ( y , x ( l ) , h ( l ) p , h ( l ) q , c ( l ) q ) z ( l +1) ∼ StatisNet ( h ( l +1) q ) ( h
( l +1) p , c ( l +1) p ) ← ConvLSTM θ ( e ( l ) , z ( l ) , h ( l ) p , c ( l ) p )
where x l is the input at the l th step, y is target reconstruction, z ( l +1) is the sampled latent at the l th step.
We denote the prior module and posterior module with subscript p and q , respectively.
θ and φ are neural network parameters.
The StatisNet is described in the following paragraph.
In the
Renderer network, we replace the StatisNet with a deterministic Decoder.
Sufﬁcient Statistic Networks:
The Sufﬁcient Statistic Networks will output sufﬁcient statistics for pre-deﬁned distribution, e.g. µ and σ for Gaussian distribution, given inputs.
We list the conﬁguration of all the Sufﬁcient Statistics Networks in Table 5 used during generation.
For z posn , z whatn and s scalen,i , we use auto-regressive scheme (ConvDraw) to learn the sufﬁcient statistics.
For z presn and s presq,n , we use regular convolutional layers.
In the third column, we give the kernel size of the ﬁrst convolutional layer, the remaining are Conv3D1X1 layers.
If the kernel size is 3 , we have one zero paddings, the stride is 1 to keep the spatial size unchanged.
The ”Draw Rollouts” column shows the number of DRAW steps we apply.
The ”Concat” column speciﬁes how we use sampled latent value for the subsequent process, for example, concatenating z l , for l < L or only taking z L , where L is the rollout steps.
[Table]
GQN Implementation:
We strictly follow the paper for implementation details.
The only difference is we enhance the decoder in the renderer by adding two more convolutional layers.
A. I MPLEMENTATION D ETAILS
Here, we give the details of the implementation of ROOTS.
We ﬁrst outline the generation and inference process for one object, indexed with n .
Parallelizing it to multiple objects is straightforward.
Generation Process Superscript on ConvDRAW is used to indicate which variable the ConvDRAW module is responsible for.
The Renderer module is implemented in the same way as
ConvDRAW with one difference that we do not model any variable in Renderer, making it a deterministic decoder.
All the ConvDRAW modules have a hidden state size of 128.
We use ST denote spatial transformation process and ST − 1 denote the reversed process.
K denote the index set of context C and θ and φ are neural network parameters.
[Equation]
(Obtain scene-volume feature-map) (5)
[Equation]
(Sample 3D position) (6)
[Equation]
(Sample global presence) (7)
[Equation]
(Perspective projection) (8)
[Equation]
(Sample object scale) (9)
[Equation]
(Crop object patch) (10)
[Equation]
(Object level encoding) (11)
[Equation]
(Sample global what) (12)
[Equation]
(Perspective projection) (13)
[Equation]
q ) (Sample object scale) (14)
[Table]
[Table] [Table]
, s
[Table]
,n , s scaleq,n ) (Sample local presence) (15)
[Table]
whatn ,
[Table]
(Decode the object patch and object mask) (16)
[Table]
[Table]
qn , [ s posq,n , s sacleq,n ]) (Generate object patch) (17)
[Table] [Table]
[Table] [Table]
n ×
[Table]
qn
[Equation]
( depth
[Equation]
× α
[Equation]
) (Obtain occlusion mask) (19)
[Table]
n
[Equation]
(Generate foreground) (20)
[Table]
[Equation]
(Generate foreground mask) (21)
[Table]
[Equation]
r
[Equation]
(Sample background) (22)
[Table] [Table]
bg
[Equation]
v q ) (Decode background) (23)
[Table]
[Equation]
(Render generations) (24)
Inference Process
The inference model is paralleled with generative process.
We only highlight the different part below
[Table]
[Equation] [Equation]
[Equation] [Equation]
[Equation]
for
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
(Encode
[Equation] [Equation]
[Equation] [Equation]
[Equation]
A. D ATASET D ETAILS
The sizes of objects are randomly chosen from 0 .
56 to 0 . 66 .
Each object is put on the ﬂoor of 3D space with a range of [ − 2 , 2] along both x-axis and y-axis.
We have three different types of object, cube, sphere
[Equation]
and cylinder with 6 different colors.
For each dataset, we ﬁrst randomly choose the number of objects in a scene, then randomly choose object type, color and their positions ( x and y coordinates).
For each scene, we have 30 cameras put at a radius of 3 , pointing at a squared area located at the center, thus, the camera would