P ROBABILISTIC I MPLICIT S CENE C OMPLETION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a probabilistic shape completion method extended to the continuous geometry of large-scale 3D scenes.
Real-world scans of 3D scenes suffer from a considerable amount of missing data cluttered with unsegmented objects.
The problem of shape completion is inherently ill-posed, and high-quality result requires scalable solutions that consider multiple possible outcomes.
We employ the Generative Cellular Automata that learns the multi-modal distribution and transform the formulation to process large-scale continuous geometry.
The local continuous shape is incrementally generated as a sparse voxel embedding, which contains the latent code for each occupied cell.
We formally derive that our training objective for the sparse voxel embedding maximizes the variational lower bound of the complete shape distribution and therefore our progressive generation constitutes a valid generative model.
Experiments show that our model successfully generates diverse plausible scenes faithful to the input, especially when the input data suffers from a signiﬁcant amount of missing data and previous approaches fail.
We also demonstrate that our approach outperforms deterministic models even for input data with a relatively small level of incompleteness, which veriﬁes that probabilistic formulation is crucial for high-quality geometry completion.
I NTRODUCTION
High-quality 3D data can create realistic virtual 3D environments or provide crucial information to interact with the environment for robots or human users (Varley et al.
However, 3D data acquired from a real-world scan is often noisy and incomplete with irregular samples.
The task of 3D shape completion aims to recover the complete surface geometry from the raw 3D scans.
Shape completion is often formulated in a data-driven way using the prior distribution of 3D geometry, which often results in multiple plausible outcomes given incomplete and noisy observation.
If one learns to regress a single shape out of multi-modal shape distribution, one is bound to lose ﬁne details of the geometry and produce blurry outputs as noticed with general generative models (Goodfellow (2017)).
If we extend the range of completion to the scale of scenes with multiple objects, the task becomes even more challenging with the memory and computation requirements for representing large-scale high resolution 3D shapes.
In this work, we present continuous Generative Cellular Automata (cGCA), which generates multiple continuous surfaces for 3D reconstruction.
Our work builds on Generative Cellular Automata (GCA) (Zhang et al.
(2021)), which produces diverse shapes by progressively growing the object surface from the immediate neighbors of the input shape.
cGCA inherits the multi-modal and scalable generation of GCA, but overcomes the limitation of discrete voxel resolution producing high-quality continuous surfaces.
Speciﬁcally, our model learns to generate diverse sparse voxels associated with their local latent codes, namely sparse voxel embedding, where each latent code encodes the deep implicit ﬁelds of continuous geometry near each of the occupied voxels (Chabra et al.
; Jiang et al.
Our training objective maximizes the variational lower bound for the log-likelihood of the surface distribution represented with sparse voxel embedding.
The stochastic formulation is modiﬁed from the original GCA, and theoretically justiﬁed as a sound generative model.
We demonstrate that cGCA can faithfully generate multiple plausible solutions of shape completion even for large-scale scenes with a signiﬁcant amount of missing data as shown in Figure 1.
To the best of our knowledge, we are the ﬁrst to tackle the challenging task of probabilistic scene completion, which requires not only the model to generate multiple plausible outcomes but also be scalable enough to capture the wide-range context of multiple objects.
[Figure]
Even when the raw input is severely damaged (left), cGCA can generate plausible yet diverse complete continuous shapes.
We summarize the key contributions as follows: (1) We are the ﬁrst to tackle the problem of probabilistic scene completion with partial scans, and provide a scalable model that can capture the large-scale context of scenes.
(2) We present continuous Generative Cellular Automata, a generative model that produces diverse continuous surfaces from a partial observation.
(3) We modify infusion training (Bordes et al.
and prove that the formulation indeed increases the variational lower bound of data distribution, which veriﬁes that the proposed progressive generation is a valid generative model.
P RELIMINARIES : G ENERATIVE C ELLULAR A UTOMATA
Continuous Generative Cellular Automata (cGCA) extends the idea of Generative Cellular Automata (GCA) by Zhang et al.
but generates continuous surface with implicit representation instead of discrete voxel grid.
For the completeness of discussion, we brieﬂy review the formulation of GCA.
Starting from an incomplete voxelized shape, GCA progressively updates the local neighborhood of current occupied voxels to eventually generate a complete shape.
In GCA, a shape is represented as a state s =
( c, o c ) | o c ∈ { 0 , 1 } , c ∈ Z 3 } , a set of binary occupancy
o c for every cell c ∈ Z 3 , where the occupancy grid stores only the sparse cells on the surface.
Given a state of an incomplete shape s 0 , GCA evolves to the state of a complete shape s T by sampling s 1: T from the Markov chain
[Equation]
[Equation]
where T is a ﬁxed number of transitions and p θ is a homogeneous transition kernel parameterized by neural network parameters θ .
The transition kernel p θ is implemented with sparse CNN (Graham et al.
; Choy et al.
(2019)), which is a highly efﬁcient neural network architecture that computes the convolution operation only on the occupied voxels.
The progressive generation of GCA conﬁnes the search space of each transition kernel at the immediate neighborhood of the current state.
The occupancy probability within the neighborhood is regressed following Bernoulli distribution, and then the subsequent state is independently sampled for individual cells.
With the restricted domain for probability estimation, the model is scalable to high resolution 3D voxel space.
GCA shows that the series of local growth near sparse occupied cells can eventually complete the shape as a uniﬁed structure since the shapes are continuous and connected.
While GCA is a scalable solution for generating diverse shapes, the grid representation for the 3D geometry inherently limits the resolution of the ﬁnal shape.
C ONTINUOUS G ENERATIVE C ELLULAR A UTOMATA
In Sec. 3.1, we formally introduce an extension of sparse occupancy voxels to represent continuous geometry named sparse voxel embedding, where each occupied voxel contains latent code representing local implicit ﬁelds.
We train an autoencoder that can compress the implicit ﬁelds into the embeddings and vice versa.
Then we present the sampling procedure of cGCA that generates 3D shape in Sec. 3.2, which is the inference step for shape completion.
Sec.
3.3 shows the training objective of cGCA, which approximately maximizes the variational lower bound for the distribution of the complete continuous geometry.
[Figure]
3. S PARSE V OXEL E MBEDDING
In addition to the sparse occupied voxels of GCA, the state of cGCA, named sparse voxel embedding, contains the associated latent code, which can be decoded into continuous surface.
Formally, the state s of cGCA is deﬁned as a set of pair of binary occupancy o c and the latent code z c , for the cells c in a three-dimensional grid Z
[Equation]
Similar to GCA, cGCA maintains the representation sparse by storing only the set of occupied voxels and their latent codes, and sets z c = 0 if o c = 0 .
The sparse voxel embedding s can be converted to and from the implicit representation of local geometry with neural networks, inspired by the work of Peng et al.
(2020), Chabra et al.
(2020), and
Chibane et al.
(2020a).
We utilize the (signed) distance to the surface as the implicit representation, and use autoencoder for the conversion.
The encoder g φ produces the sparse voxel embedding s from the coordinate-distance pairs P = { ( p, d p ) | p ∈ R 3 , d p ∈ R } , where p is a 3D coordinate and d p is the distance to the surface, s = g φ ( P ) .
The decoder f ω , on the other hand, regresses the local implicit ﬁeld value d q at the 3D position q ∈ R 3 given the sparse voxel embedding s , ˆ d q = f ω ( s, q ) for given coordinate-distance pairs Q = { ( q, d q ) | q ∈ R 3 , d q ∈ R } .
The detailed architecture of the autoencoder is described in Appendix A, where the decoder f ω generates continuous geometry by interpolating hierarchical features extracted from sparse voxel embedding.
An example of the conversion is presented on the left side of Fig.
2, where the color of the sparse voxel embedding represents clustered labels of the latent codes with k-means clustering (Hartigan & Wong (1979)).
Note that the embedding of a similar local geometry, such as the seat of the chair, exhibits similar values of latent codes.
The parameters φ, ω of the autoencoder are jointly optimized by minimizing the following loss function:
[Equation]
where s = g φ ( P ) and (cid:15) is the size of a single voxel.
The ﬁrst term in Eq.
(3) corresponds to minimizing the normalized distance and the second is the regularization term for the latent code weighted by hyperparameter β .
Clamping the maximum distance makes the network focus on predicting accurate values at the vicinity of the surface (Park et al.
; Chibane et al.
(2020b)).
3. S AMPLING FROM CONTINUOUS G ENERATIVE C ELLULAR A UTOMATA
The generation process of cGCA echos the formulation of GCA (Zhang et al.
(2021)), and repeats T steps of sampling from the transition kernel to progressively grow the shape.
Each transition kernel p ( s t +1 | s t ) is factorized into cells within the local neighborhood of the occupied cells of the current state, N ( s t ) =
{ c (cid:48)
∈
Z 3 | d ( c, c (cid:48) ) ≤ r, c ∈ s t } 1 given a distance metric d and the radius r : p
[Equation]
Note that the distribution is further decomposed into the occupancy o c and the latent code z c , where we denote o c and z c as the random variable of occupancy and latent code for cell c in state s t +1 .
Therefore the shape is generated by progressively sampling the occupancy and the latent codes for the occupied voxels which are decoded and fused into a continuous geometry.
The binary occupancy is represented with the Bernoulli distribution
[Equation]
where λ θ,c ∈ [0 , 1] is the estimated occupancy probability at the corresponding cell c .
With our sparse representation, the distribution of the latent codes is
[Equation]
δ 0 is a Dirac delta distribution at 0 indicating that z c = 0 when o c = 0 .
For the occupied voxels ( o c = 1 ), z c follows the normal distribution with the estimated mean of the latent code µ θ,c ∈ R K and the predeﬁned standard distribution σ t
I decreasing with respect to t .
Initial State.
Given an incomplete point cloud, we set the initial state s 0 of the sampling chain by setting the occupancy o
c to be 1 for the cells that contain point cloud and associating the occupied cells with a latent code sampled from the isotropic normal distribution.
However, the input can better describe the provided partial geometry if we encode the latent code z c of the occupied cells with the encoder g φ .
The ﬁnal completion is more precise when all the transitions p θ are conditioned with the initial state containing the encoded latent code.
Further details are described in Appendix A.
Mode Seeking.
While we effectively model the probabilistic distribution of multi-modal shapes, the ﬁnal reconstruction needs to converge to a single coherent shape.
Naïve sampling of the stochastic transition kernel in Eq.
(4) can include noisy voxels with low-occupancy probability.
As a simple trick, we augment mode seeking steps that determine the most probable mode of the current result instead of probabilistic sampling.
Speciﬁcally, we run additional T (cid:48) steps of the transition kernel but we select the cells with probability higher than 0.5 and set the latent code as the mean of the distribution µ θ,c .
The mode seeking steps ensure that the ﬁnal shape discovers the dominant mode that is closest to s T as depicted in Fig.
2, where it can be transformed into implicit function with the pretrained decoder f w .
3. T RAINING C ONTINUOUS G ENERATIVE C ELLULAR A UTOMATA
We train a homogeneous transition kernel p θ ( s t +1 | s t ) , whose repetitive applications eventually yield the samples that follow the learned distribution.
However, the data contains only the initial s 0 and the ground truth state x , and we need to emulate the sequence for training.
We adapt infusion training (Bordes et al.
(2017)), which induces the intermediate transitions to converge to the desired complete state.
To this end, we deﬁne a function G x ( s ) that ﬁnds the valid cells that are closest to the complete shape x within the neighborhood of the current state N ( s ) :
[Equation]
Then, we deﬁne the infusion kernel q t factorized similarly as the sampling kernel in Eq
[Equation]
The distributions for both o c and z c are gradually biased towards the ground truth ﬁnal shape x with the infusion rate α t , which increases linearly with respect to time step, i.e., α t = max( α 1 t
+ α 0 , 1) , with α 1 > 0 :
[Equation]
[Equation]
Here 1 is an indicator function, and we will denote o xc , z xc as the occupancy and latent code of the ground truth complete shape x at coordinate c .
cGCA aims to optimize the log-likelihood of the ground truth sparse voxel embedding log p θ ( x ) .
However, since the direct optimization of the exact log-likelihood is intractable, we modify the variational lower bound using the derivation of diffusion-based models (Sohl-Dickstein et al.
[Equation]
where the full derivation is in Appendix B.1.
We now analyze L init , L t , L ﬁnal separately.
We ignore the term L init during optimization since it contains no trainable parameters.
L t for 0 ≤ t < T −can be decomposed as the following :
[Equation]
where the full derivation is in Appendix B.2.
Since L o and L z are the KL divergence between
Bernoulli and normal distributions, respectively, L t can be written in a closed-form.
In practice, the scale of L z can be much larger than that of L o .
This results in local minima in the gradient-based optimization and reduces the occupancy probability q θ ( o c = 1 | s t , x ) for every cell.
So we balance the two losses by multiplying a hyperparameter γ at L z , which is ﬁxed as γ = 0 .
01 for all experiments.
The last term L ﬁnal can be written as following:
[Equation]
A problem rises when computing L ﬁnal , since the usage of Dirac distribution makes L ﬁnal → ∞ if o xc = 0 , which does not produce a valid gradient for optimization.
However, we can replace the loss
L ﬁnal with a well-behaved loss L t for t =
T − 1 , by using the following proposition:
Proposition 1.
Assume δ 0 ( z xc ) is replaced by the indicator function 1 [ z xc = 0] when computing
L ﬁnal .
Then ∇L T − 1 = ∇L ﬁnal , for T (cid:29) 1 .
Proof.
The proof is found in Appendix B.3.
The proposition above serves as a justiﬁcation for approximating ∇L ﬁnal as ∇L T − 1 , with the beneﬁts of having a simpler training procedure and easier implementation.
Replacing δ
( z Tc ) with 1 [ z xc = 0] can be regarded as a reweighting technique that naturally avoids divergence in the lower bound, since both functions output a non-zero value only at 0 .
Further discussions about the replacement are in
Appendix B.3.
In conclusion, the training procedure is outlined as follows:
Sample s 0: T by s 0 ∼ q 0 , s t +1 ∼ q tθ ( ·| s t , x ) .
For t < T , update θ with θ ← θ + η ∇ θ L t , where η is the learning rate.
Note that the original infusion training (Bordes et al.
also attempts to minimize the variational lower bound, employing the Monte Carlo approximation with reparameterization trick (Kingma &
Welling (2014)) to compute the gradients.
However, our objective avoids the approximations and can compute the exact lower bound for a single training step.
The proposed simpliﬁcation can be applied to infusion training with any data structure including images.
We also provide the summary on the difference of our formulation compared to GCA in Appendix C.
[Table]
[Figure]
[Table]
E XPERIMENTS
We test the probabilistic shape completion of cGCA for scenes (Section 4.1) and single object (Section 4.2).
For all the experiments, we use latent code dimension K = 32 , trained with regularization parameter β = 0 .
All the results are re-trained for each dataset.
Further implementation details are in the Appendix A.2 and effects of mode seeking steps are discussed in Appendix E.
4. S CENE C OMPLETION
In this section, we evaluate our method on two datasets: ShapeNet scene (Peng et al.
and 3DFront (Fu et al.
(2021)) dataset.
The input incomplete scans are created by iteratively removing points within a ﬁxed distance from a random surface point.
We control the minimum preserved ratio of the original complete surface points for each object in the scene to test varying levels of completeness.
The levels of completeness tested are 0.2, 0.5, and 0.8 with each dataset trained/tested separately.
We evaluate the quality and diversity of the completion results by measuring the Chamfer-L1 distance (CD), total mutual distance (TMD), respectively, as in the previous methods (Peng et al.
Zhang et al.
For probabilistic methods, ﬁve completion results are generated and we report the minimum and average of Chamfer-L1 distance (min.
CD, avg. CD).
Note that if the input is severely incomplete, there exist various modes of completion that might be feasible but deviate from its ground truth geometry.
Nonetheless, we still compare CD assuming that plausible reconstructions are likely to be similar to the ground truth.
ShapeNet Scene.
ShapeNet scene contains synthetic rooms that contain multiple ShapeNet (Chang et al.
(2015)) objects, which have been randomly scaled with randomly scaled ﬂoor and random
[Table]
[Figure]
walls.
We compare the performance of cGCA with two deterministic scene completion models that utilize occupancy as the implicit representation:
ConvOcc (Peng et al.
and IFNet (Chibane et al.
(2020a)).
We additionally test the quality of our completion against a probabilistic method of
GCA (Zhang et al.
Both GCA and cGCA use 64 3 voxel resolution with T = 15 transitions.
cGCA (w/ cond.) indicates a variant of our model, where each transition is conditioned on the initial state s 0 obtained by the encoder g φ as discussed in Sec. 3.2.
Table 1 contains the quantitative comparison on ShapeNet scene.
Both versions of cGCA outperform all other methods on min.
CD for all level of completeness.
The performance gap is especially prominent for highly incomplete input, which can be visually veriﬁed from Fig.
The deterministic models generate blurry objects given high uncertainty, while our method consistently generates detailed reconstructions for inputs with different levels of completeness.
Our result coincides with the well-known phenomena of generative models, where the deterministic models fail to generate crisp outputs of multi-modal distribution (Goodfellow (2017)).
Considering practical scenarios with irregular real-world scans, our probabilistic formulation is crucial for accurate 3D scene completion.
When conditioned with the initial state, the completion results of cGCA stay faithful to the input data, achieving lower CDs.
Note that the high diversity for GCA are partially due to the discretization artifacts.
Further analysis on varying completeness of input is discussed in Appendix F.
3DFront.
3DFront is a large-scale indoor synthetic scene dataset with professionally designed layouts and contains high-quality 3D objects, in contrast to random object placement for ShapeNet scene.
3DFront dataset represents the realistic scenario where the objects are composed of multiple meshes without clear boundaries for inside or outside.
Unless the input is carefully processed to be converted
[Table]
[Figure]
[Table]
into a watertight mesh, the set-up excludes many of the common choices for implicit representation, such as occupancy or signed distance ﬁelds.
However, the formulation of cGCA can be easily adapted for different implicit representation, and we employ unsigned distance ﬁelds (Chibane et al.
(2020b))
to create the sparse voxel embedding for 3DFront dataset.
We compare the performance of cGCA against GCA, both with voxel resolution of 5cm and T = 20 transitions.
Table 2 shows that cGCA outperforms GCA by a large margin in CD, generating high-ﬁdelity completions with unsigned distance ﬁelds.
While both GCA and cGCA are capable of generating multiple plausible results, GCA suffers from discretization artifacts due to voxelized representation, as shown in Fig. 4.
cGCA not only overcomes the limitation of the resolution, but also is scalable to process the entire rooms at once during both training and test time.
In contrast, previous methods for scene completion (Siddiqui et al.
; Peng et al.
(2020)) divide the scene into small sections and separately complete them.
We analyze the scalability in terms of the network parameters and the
GPU usage in Appendix D. In Appendix G, we also provide results on ScanNet (Dai et al.
(2017a))
dataset, which is one of the widely used datasets for real-world indoor environments.
4. S INGLE O BJECT C OMPLETION
We analyze various performance metrics of cGCA for a single object completion with chair/sofa/table classes of ShapeNet (Chang et al.
dataset.
Given densely sampled points of a normalized object in [ − 1 , 1] 3 , the incomplete observation is generated by selecting points within the sphere of radius 0.5 centered at one of the surface points.
From the partial observation, we sample 1,024 points which serve as a sparse input, and sample 2,048 points from completion results for testing.
Following the previous method (Wu et al.
, we generate ten completions and compare MMD (quality),
UHD (ﬁdelity), and TMD (diversity).
Our method is compared against other probabilistic shape completion methods: cGAN
(Wu et al.
(2020)) which is based on point cloud, and GCA (Zhang et al.
(2021)) which uses voxel representation.
We use T = 30 transitions.
Quantitative and qualitative results are shown in Table 3 and Fig.
Our approach exceeds other baselines in all metrics, indicating that cGCA can generate high-quality completions (MMD) that are faithful to input (UHD) while being diverse (TMD).
By using latent codes, the completed continuous surface of cGCA can capture geometry beyond its voxel resolution.
The quality of completed shape in 32 3 voxel resolution therefore even outperforms in MMD for discrete GCA in higher 64 3 voxel resolution.
Also, the UHD score of cGCA (w/ cond.) exceeds that of GCA and vanilla cGCA indicating that conditioning latent codes from the input indeed preserves the input partial geometry.
R ELATED W ORKS
3D Shape Completion.
The data-driven completion of 3D shapes demands a large amount of memory and computation.
The memory requirement for voxel-based methods increases cubically to the resolution (Dai et al.
(2017b))
while the counterpart of point cloud based representations (Yuan et al.
(2018)) roughly increases linearly with the number of points.
Extensions of scene completion in voxel space utilize hierarchical representation (Dai et al.
or subdivided scenes (Dai et al.
(2018; 2020)) with sparse voxel representations.
Recently, deep implicit representations (Park et al.
; Chen & Zhang (2019); Mescheder et al.
(2019)) suggest a way to overcome the limitation of resolution.
Subsequent works (Chabra et al.
; Chibane et al.
(2020a)
; Jiang et al.
Peng et al.
demonstrate methods to extend the representation to large-scale scenes.
However, most works are limited to regressing a single surface from a given observation.
Only a few recent works (Wu et al.
; Zhang et al.
(2021); Smith & Meger (2017)) generate multiple plausible outcomes by modeling the distribution of surface conditioned on the observation.
cGCA suggests a scalable solution for multi-modal continuous shape completion by employing progressive generation with continuous shape representations.
Diffusion Probabilistic Models.
One way to capture the complex data distribution by the generative model is to use a diffusion process inspired by nonequilibrium thermodynamics such as SohlDickstein et al.
; Ho et al.
; Luo & Hu (2021).
The diffusion process incrementally destroys the data distribution by adding noise, whereas the transition kernel learns to revert the process that restores the data structure.
The learned distribution is ﬂexible and easy to sample from, but it is designed to evolve from a random distribution.
On the other hand, the infusion training by Bordes et al.
(2017) applies a similar technique but creates a forward chain instead of reverting the diffusion process.
Since the infusion training can start from a structured input distribution, it is more suitable to a shape completion that starts from a partial data input.
However, the infusion training approximates the lower bound of variational distribution with Monte Carlo estimates using the reprameterization trick (Kingma & Welling (2014)).
We modify the training objective and introduce a simple variant of infusion training that can maximize the variational lower bound of the log-likelihood of the data distribution without using Monte Carlo approximation.
C ONCLUSION
We are the ﬁrst to tackle the challenging task of probabilistic scene completion, which requires not only the model to generate multiple plausible outcomes but also be scalable to capture the wide-range context of multiple objects.
To this end, we propose continuous Generative Cellular Automata, a scalable generative model for completing multiple plausible continuous surfaces from an incomplete point cloud.
cGCA compresses the implicit ﬁeld into sparse voxels associated with their latent code named sparse voxel embedding, and incrementally generates diverse implicit surfaces.
The training objective is proven to maximize the variational lower bound of the likelihood of sparse voxel embeddings, indicating that cGCA is a theoretically valid generative model.
Extensive experiments show that our model is able to faithfully generate multiple plausible surfaces from partial observation.
There are a few interesting future directions.
Our results are trained with synthetic scene datasets where the ground truth data is available.
It would be interesting to see how well the data performs in real data with self-supervised learning.
For example, we can extend our method to real scenes such as
ScanNet Dai et al. (2017a) or Matterport 3D (Chang et al.
by training the infusion chain with data altered to have different levels of completeness as suggested by Dai et al.
Also, our work requires two-stage training, where the transition kernel is trained with the ground truth latent codes generated from the pre-trained autoencoder.
It would be less cumbersome if the training could be done in an end-to-end fashion.
Lastly, our work takes a longer inference time compared to previous methods (Peng et al.
since a single completion requires multiple transitions.
Reducing the number of transitions by using a technique similar to Anonymous (2022) can accelerate the runtime.
E THICS S TATEMENT
The goal of our model is to generate diverse plausible scenes given an observation obtained by sensors.
While recovering the detailed geometry of real scenes is crucial for many VR/AR and robotics applications, it might violate proprietary or individual privacy rights when abused.
Generating the unseen part can also be regarded as creating fake information that can deceive people as real.
R EPRODUCIBILITY
Code to run the experiments is available at https://github.com/iclr2022submission4/cgca.
Appendix A contains the implementation details including the network architecture, hyperparameter settings and dataset processing.
Proofs and derivations are described in Appendix B.
A I MPLEMENTATION D ETAILS
We use Pytorch (Paszke et al.
and the sparse convolution library MinkowskiEngine (Choy et al.
for all implementations.
A. A UTOENCODER
Network Architecture.
The autoencoder is composed of two modules, encoder g φ and decoder f ω to convert between the sparse voxel embedding and implicit ﬁelds, which is depicted in Fig.
The encoder g φ generates the sparse voxel embedding s from the coordinate-distance pairs P = { ( p, d p ) | p ∈ R 3 , d p ∈ R } , where p is a 3D coordinate and d p is the (signed) distance to the surface, s = g φ ( P ) .
The decoder f ω regresses the implicit ﬁeld value
d q at the 3D position q ∈ R 3 given the sparse voxel embedding s , d q = f ω ( q, s ) .
The encoder g φ is a a local PointNet (Qi et al.
(2016)) implemented with MLP of 4 blocks with hidden dimension 128, as in ConvOcc (Peng et al.
Given a set of coordinate-distance pair P , we ﬁnd the nearest cell c p for each coordinate-distance pair ( p, d p ) .
Then we make the representation sparse by removing the coordinate-distance pair if the nearest cell c p do not contain the surface.
For the remaining pairs, we normalize the coordinate p to the local coordinate p (cid:48) ∈ [ − 1 , 1] 3 centered at the nearest voxel.
Then we use the local coordinate to build a vector in R 4 for coordinate-distance pair ( p (cid:48) , d p ) which serves as the input to the PointNet for the encoder g φ .
The local PointNet architecture employs average pooling to the point-coordinate pairs that belong to the same cell.
The architecture of decoder f ω , inspired by IFNet (Chibane et al.
(2020a)) is depicted in Fig.
As shown on the left, the decoder f ω can be decomposed into the sparse convolution f ω 1 that extracts hierarchical feature, followed by trilinear interpolation of the extracted feature Ψ , and ﬁnal MLP layers f ω 2 .
The sparse convolution layer f ω 1 aggregates the information into multi-level grids in n different resolutions, i.e., f ω 1 ( s )
= F 1 , F 2 , ..., F n , Starting from the original resolution F 1 , F k contains a grid that is downsampled k times.
However, in contrast to the original IFNet, our grid features are sparse, since they only store the downsampled occupied cells of sparse voxel embedding s .
The sparse representation F k = { ( c, o c , e c ) | o c ∈ { 0 , 1 } , e c ∈ R L } is composed of occupancy o c and L -dimensional feature e c for each cell, like sparse voxel embedding.
The grid points c that are not occupied are considered as having zero features.
We use n = 3 levels, with feature dimension
L = 32 for all experiments.
The multi-scale features are then aggregated to deﬁne the feature for a query point q : Ψ( f ω 1 ( s ) , q )
= (cid:80) k
ψ k ( F k , q ) .
Here ψ k ( F k , q ) ∈ R L is the trilinear interpolation of features in discrete grid F k to deﬁne the feature at a particular position q , similar to IFNet.
We apply trilinear interpolation at each resolution k then combine the features.
Lastly, the MLP layer maps the feature at the query point q to the implicit function value ˆ d q , f ω 2 : R L → R .
The MLP is composed of 4 ResNet blocks with hidden dimension 128 with tanh as the ﬁnal activation function.
The ﬁnal value ˆ d q is the truncated signed distance within the range of [ − 1 , 1] except for 3DFront (Fu et al.
(2021)) which does not have a watertight mesh.
We use unsigned distance function for 3DFront as in NDF (Chibane et al.
(2020b))
by changing the codomain of the
[Figure]
decoder to [0 , 1] with sigmoid as the ﬁnal activation function.
Thus, the decoder can be summarized as f ω ( s, q ) =
f ω 2 (Ψ( f ω 1 ( s ) , q )) .
We would like to emphasize that the autoencoder does not use any dense 3D convolutions, and thus the implementation is efﬁcient, especially when reconstructing the sparse voxel embedding.
Since the decoding architecture only consists of sparse convolutions and MLPs, the memory requirement increases linearly with respect to the number of occupied voxels.
In contrast, the requirement for dense 3D convolution increases cubic with respect to the maximum size of the scene.
Our sparse implementation is efﬁcient enough to eliminate any preprocessing for 3D scene completion.
While Jiang et al.
(2020) also employ an autoencoder to obtain local implicit ﬁelds, the method uses dense convolutions and need to divide a single object into pieces which are processed separately.
On the other hand, our implementation can train the autoencoder for the whole continuous 3D geometry without preprocessing.
Reconstruction Results.
Fig.
7 shows a reconstruction of a chair (left) using signed distance ﬁelds and a scene in 3DFront (right) using unsigned distance ﬁelds.
The ﬁgure shows that our autoencoder can faithfully reconstruct a continuous surface using the sparse voxel embedding.
However, the usage of unsigned distance ﬁelds (Chibane et al.
(2020b))
representation tends to generate relatively thicker reconstructions compared to that of signed distance ﬁelds.
Without a clear distinction between the interior/exterior, the unsigned representation makes it difﬁcult to scrutinize the zero-level isosurface and thus results in comparably thick reconstruction.
Thus, the ﬂexibility of being able to represent any mesh with various topology comes at a cost of expressive power, in contrast to signed distance ﬁelds which can only represent closed surfaces.
Our work can be further improved by utilizing more powerful implicit representation techniques, such as SIREN (Sitzmann et al.
, but we leave it as future work.
Hyperparameters and Implementation Details.
We train our model with Adam (Kingma & Ba (2015)) optimizer with learning rate 5e-4 and use batch size of 3, 1, and 4 for ShapeNet scene (Peng et al.
, 3DFront (Fu et al.
(2021)), and ShapeNet (Chang et al.
(2015)) dataset, respectively.
The autoencoder for scene completion is pretrained with all classes of ShapeNet assuming that the local geometric details for scenes are represented with the patches from individual objects.
A. T RANSITION M ODEL
Network Architecture.
Following GCA (Zhang et al.
, we implement the transition model by using a variant of U-Net (Ronneberger et al. (2015)).
The input to the sparse convolution network is the sparse voxel embedding s t , and the output feature size of the last convolution is K + 1 , where we use generalized sparse convolution for the last layer to obtain the occupancy (with dimension 1) and the latent code (with dimension K ) of the neighborhood.
Unlike GCA, our implementation does not need an additional aggregation step.
s 0 Conditioned Model.
The s 0 conditioned model is a variant of our model, where we employ the use of encoder g φ to estimate the initial latent codes of initial state s 0 and condition every transition kernel on s 0 .
Given incomplete point cloud R = { c 1 , c 2 , ..., c M } with c i ∈ R 3 , the variant constructs the initial s 0 by encoding a set of coordinated-distance pair P R = { ( c, 0) | c ∈ R } , where the distance s p a r s e c onv 7 x 7 x 7 ,s p a r s e c onv 5 x 5 x 5 ,( 2 ) s p a r s e c onv b l o c k x 2 ,s p a r s e c onv 5 x 5 x 5 ,( 2 ) s p a r s e c onv b l o c k x 2 ,s p a r s e c onv 3 x 3 x 3 ,( 2 ) s p a r s e c onv b l o c k x 3 , s p a r s e c onv 3 x 3 x 3 ,( 2 ) s p a r s e c onv b l o c k x 5 , s p a r s e c onv t r 3 x 3 x 3 , 256 ( 2 ) s p a r s e c onv b l o c k x 1 , s p a r s e c onv t r 3 x 3 x 3 , 128 ( 2 ) s p a r s e c onv b l o c k x 1 , s p a r s e c onv t r 3 x 3 x 3 , 96 ( 2 ) s p a r s e c onv do w n b l o c k ,s p a r s e c onv t r 5 x 5 x 5 , 96 ( 2 ) s p a r s e c onv do w n b l o c k ,s p a r s e c onv 3 x 3 x 3 , 𝐾 +sparse conv block x n, F s p a r s e c onv 3 x 3 x 3 , F s p a r s e c onv 3 x 3 x 3 , F + R e L U x n sparse conv down block, F s p a r s e c onv 3 x 3 x 3 , F s p a r s e c onv 3 x 3 x 3 , F + R e L U s p a r s e c onv 1 x 1 x 1 , F s p a r s e c onv do w n b l o c k ,s p a r s e c onv do w n b l o c k , s p a r s e c onv do w n b l o c k , s p a r s e c onv do w n b l o c k , s p a r s e c onv do w n b l o c k , s p a r s e c onv b l o c k x 1 ,s p a r s e c onv b l o c k x 1 , is set to 0 for every coordinate in R .
Since the point cloud is sampled from the surface of the original geometry, this is a reasonable estimate of the latent code.
The transition kernel p θ is conditioned on s 0 by taking a sparse voxel embedding ˆ s t of latent code dimension 2 K that concatenates the current state s t and s 0 as the input.
Speciﬁcally, we deﬁne ˆ s t to be ˆ s t = { ( c, o c , z c ) | o c
= o s t c
∨ o s 0 c , z c
=
[ z s 0 c , z s t c ] ∈ R 2 K , c ∈ Z 3 } , where the o s t c , o s 0 c , z s t c , z s
c are the occupancies and latent codes of s t and s 0 .
The occupancy is set to one if either s t or s 0 is occupied and the latent code is set to be the concatenation of latent codes of s 0 and s t .
Thus, the only modiﬁcation to the neural network p θ is the size of input feature dimension, which is increased from
K to 2 K .
Hyperparameters and Implementation Details We train our model with Adam (Kingma & Ba (2015)) optimizer with learning rate 5e-4.
The transition kernel is trained using infusion schedule α t = 0 .
1 + 0 .
005 t and standard deviation schedule σ t = 10 − 1 − 0 .
01 t .
We use the accelerated training scheme introduced in GCA (Zhang et al.
by managing the time steps based on the current step.
We use neighborhood size r = 2 , with metric d induced by L 1 norm, except for the ShapeNet (Chang et al.
experiment on 64 3 voxel resolution which uses r = 3 .
For transition model training, we utilize maximum batch size of 32, 16 and 6, for ShapeNet scene, 3DFront, ShapeNet dataset, respectively.
We use adaptive batch size for the transition model, which is determined by the total number of cells in the minibatch for efﬁcient GPU memory usage.
Lastly, we use mode seeking steps
T (cid:48)
= 5 for all experiments.
Visualizations and Testing.
All surface visualizations are made by upsampling the original voxel resolution 4 times, where we efﬁciently query from the upsampled grid only near the occupied cells of the ﬁnal state s T + T (cid:48) .
Then, the mesh is extracted with marching cubes (Lorensen & Cline (1987)) with no other postprocessing applied.
From the 4 × upsampled high resolution grid, we extract cells with the distance value below 0.5 which serve as the point cloud to compute the metrics for testing.
A. B ASELINES
We use the ofﬁcial implementation or the code provided by the authors by contact with default hyperparameters unless stated otherwise.
All results are obtained by training on our dataset.
The output point cloud used for testing, is obtained by sampling from mesh with the same resolution if possible, unless stated otherwise.
For GCA (Zhang et al.
we use the same hyperparameters as ours for fair comparison.
The mesh is created by using marching cubes (Lorensen & Cline (1987)) with the voxel resolution.
We use the volume 64 3 encoder for ConvOcc (Peng et al.
(2020)) which achieves the best results in the original ShapeNet scene dataset (Peng et al.
For
IFNet (Chibane et al.
(2020a))
, we use the ShapeNet128Vox model as used in the work of Siddiqui et al.
with occupancy implicit representation.
cGAN
(Wu et al.
(2020)) is tested by 2,048 generated point cloud.
A. D ATASETS
ShapeNet Scene.
ShapeNet scene (Peng et al.
(2020)) contains synthetic rooms that contain multiple ShapeNet (Chang et al.
(2015)) objects, which has randomly scaled ﬂoor and random walls, normalized so that the longest length of the bounding box has length of 1.
The number of rooms for train/val/test split are 3750/250/995.
We sample SDF points for training our model.
The input is created by iteratively sampling points from the surface and removing points within distance 0.1 for each instance.
We run 20 iterations and each removal is revoked if the aggregated removal exceeds that of the guaranteed rate.
Since we are more interested in the reconstruction of objects, the walls and ﬂoors are guaranteed to have minimum rate of 0.8, regardless of the deﬁned minimum rate.
Lastly, we sample 10,000 points and add normal noise with standard deviation 0.005.
For computing the metrics at test time, we sample 100,000 points.
3DFront.
3DFront (Fu et al.
(2021)) is a large-scale indoor synthetic scene dataset with professionally designed layouts and contains high quality 3D objects.
We collect rooms that contain multiple objects, where 10% of the biggest rooms are removed from the data to discard rooms with immense size for stable training.
Thus, 7044/904/887 rooms are collected as train/val/test rooms.
The incomplete point cloud is obtained by randomly sampling with density 219 points/m
and then removed and noised as in ShapeNet scene, but with removal sphere distance proportional to the object size and noise of standard deviation of 0.01.
During evaluation, we sample points proportionally to the area for each ground truth surface with density of 875 points/m 2 to compute the metrics.
ShapeNet.
We use chair, sofa, table classes of ShapeNet (Chang et al.
(2015)), where the signed distance values of the query points are obtained by using the code of DeepSDF (Park et al.
The shapes are normalized to ﬁt in a bounding box of [ − 1 , 1] 3 .
The partial observation is generated by extracting points within a sphere of radius 0.5, centered at one of the surface points.
Then we sample 1,024 points to create a sparse input and compare the metrics by sampling 2,048 points for a fair comparison against cGAN (Wu et al.
B P ROOFS AND D ETAILED D ERIVATIONS B. V ARIATIONAL B OUND D ERIVATION
We derive the variational lower bound for cGCA (Eq.
(11)) below.
The lower bound is derived by the work of Sohl-Dickstein et al.
and we include it for completeness.
[Equation]
The second term of the right hand side can be converted into KL divergence as following:
[Equation]
s −
[Equation]
t +1) denotes variables s 0: T − 1 except s t +1 )
[Equation]
Thus,
[Equation]
is derived
[Equation]
B. KL D IVERGENCE D ECOMPOSITION
We derive the decomposition of KL divergence (Eq.
(12) ) below.
[Equation] [Equation]
chain rule for conditional KL divergence )
[Equation]
Thus, Eq.
(12) is derived.
B. A PPROXIMATING L ﬁnal BY L T −
In this section
[Equation]
we claim that L ﬁnal can be replaced by L T − 1 , which allows us to train using a simpliﬁed objective.
First, we show that L ﬁnal =
E q θ [log p θ ( x | s T − 1 )]
= ∞ due to the usage of
Dirac distribution and introduce a non-diverging likelihood by replacing the Dirac function δ 0 ( z c ) with indicator function 1 [ z c = 0] .
Recall,
[Equation]
Thus, if any cell c ∈ N ( s T − 1 ) is unoccupied, the likelihood diverges to ∞ since z xc = 0 .
This disables us to use negative log-likelihood as a loss function, since all the gradients will diverge.
This problem can be easily resolved by substituting δ 0 ( z xc ) with indicator function 1 [ z xc = 0] .
Both functions have non-zero value only at z c (cid:54) = 0 , but the former has value of ∞ while the latter has value 1 .
While 1 [ z xc = 0] is not a probability measure, i.e. (cid:82) z xc 1 [ z xc = 0] (cid:54) = 1 , replacing δ
( z xc ) will have the effect of reweighting the likelihood at z xc = 0 from ∞ to 1 , with the likelihood at other values z xc (cid:54) = 0 unchanged.
Thus, 1 [ z xc = 0] is a natural replacement of δ 0 ( z xc ) for computing a valid likelihood.
Proposition 1.
Assume δ 0 ( z xc ) is replaced by the indicator function 1 [ z xc = 0] when computing
L ﬁnal .
Then ∇L T − 1 = ∇L ﬁnal , for T (cid:29) 1 .
Proof.
For brevity of notations, we deﬁne
[Equation]
where λ q,c , µ q,c is the occupancy probability, and mean of the infusion kernel q at cell c .
With
T (cid:29) 1 , there exists T 1 < T , such that α T 1 = 1 , since α t = max( α 1 t + α 0 , 1) .
Since d ( c, c (cid:48) )
=
if and only if c = c (cid:48) , G x ( s t ) = { argmin c ∈N ( s t ) d
( c, c (cid:48) )
| c (cid:48)
∈ x } must converge to a set of occupied coordinates of x since the distance decreases for each cell c ∈ x at every step.
This indicates that λ q,c = 1 [ c ∈ G x
( s T − 1 )]
= o xc .
Also, µ q,c
=
z xc holds for all t > T 1 .
Then, L T − 1 and L ﬁnal can be expressed as the following:
[Equation]
We divide cases for o xc =
and o xc = 1 .
When o xc = 0 ,
[Equation]
where we set y log y = 0 for y = 0 , since lim y
→
y log y = 0 .
Analogously, when o xc = 1 ,
[Equation]
Thus, if o xc = 1 , L T − 1 =
L ﬁnal − K 2 log(2 πσ ) , where − K 2 log(2 πσ ) is a constant.
We can conclude that ∇ θ L T − 1 = ∇ θ L ﬁnal for all o x c .
[Table]
C D IFFERENCE OF F ORMULATION COMPARED TO GCA
We elaborate the difference of formulations compared to GCA (Zhang et al.
Usage of Latent Code.
The biggest difference between GCA and our model is the usage of local latent codes (Jiang et al.
(2020); Chabra et al.
to generate continuous surface.
Although the formulation of GCA allows scalable sampling of high resolution voxel, it cannot produce continuous surface due to predeﬁned voxel resolution.
However, we deﬁne a new state named sparse voxel embedding by appending local latent codes to the voxel occupancy.
Sparse voxel embedding can be decoded to produce continuous geometry, outperforming GCA in accuracy in all the evaluated datasets in Sec. 4.
Training Disconnected Objects.
During training, GCA assumes a property named partial connectivity between an intermediate state s t and ground truth state x to guarantee the convergence of infusion sequences to ground truth x (Property 1 and Proposition 1 of Zhang et al.
The connectivity means that any cell in s t can reach any cell in x following the path of cells in x with local transitions bounded by neighborhood size r .
The assumption is required since the infusion kernel of single cell
q t ( o c | s t ) for GCA, deﬁned as
[Equation]
14) inserts the ground truth cell c ∈ x if the cell is within the neighborhood of current state
N ( s t ) , i.e. c ∈ N ( s t ) ∩ x .
However, we eliminate the assumption by deﬁning an auxiliary function
[Equation]
and modifying the formulation as
[Equation]
by switching x to G x ( s t ) from the original infusion kernel.
The usage of G x ( s t ) guarantees infusing a temporary ground truth cell c that assures reaching all cells in x regardless of the connectivity.
The assumption is no longer required and therefore our training procedure is more general than that of
GCA.
Training Procedure for Maximizing Log-likelihood.
Both GCA and our model utilizes the infusion training (Bordes et al.
(2017)), by emulating the sequence of states that we aim to learn.
The objective of GCA does not explicitly show the relationship between the training objective and maximizing the log-likelihood, which is what we aim to achieve.
In Sec. 3.3, we give a detailed explanation of how our training objective approximates maximizing the lower bound of log-likelihood of data distribution.
Thus, our training procedure completes the heuristics used in original work of GCA.
D A NALYSIS ON S CALABILITY
In this section, we investigate the scalability of our approach.
Scene completion needs to process the 3D geometry of large-scale scenes with multiple plausible outputs, and sparse representation is crucial to ﬁnd the diverse yet detailed solutions.
As the scale of the 3D geometry increases, the approaches utilizing the dense convolutional network cannot observe the entire scene at the same time.
As a simple remedy, previous methods employ sliding window techniques for 3D scene completion (Siddiqui et al.
; Peng et al.
Basically they divide the input scene into small segments, complete each segment, and then fuse them.
The formulation can only observe the information that is within the same segment for completion, and assumes that the divisions contain all the information necessary to complete the geometry.
The size of the maximum 3D volume that recent works utilize amounts to a 3D box whose longest length is 3.2m for 64 3 resolution voxel, where individual cells are about 5 cm in each dimension.
Note that the size is comparable to ordinary furniture, such as sofa or dining table, whose spatial context might not be contained within the same segment.
The relative positions between relevant objects can be ignored, and the missing portion of the same object can accidentally be separated into different segments.
With our sparse representation, the receptive ﬁeld is large enough to observe the entire room, and we can stably complete the challenging 3D scenes with multiple objects capturing the mutual context.
We further provide the numerical comparison of our sparse representation against the dense convolution of ConvOcc (Peng et al.
Table 4 shows the number of neural network parameters and the GPU usage for processing single rooms of different sizes in 3DFront (Fu et al.
We used 5 cm voxel resolution and the grid size indicates the number of cells required to cover the longest length of the bounding box for the room.
The maximum GPU usage during 25 transitions of cGCA is provided.
We did not include the memory used for the ﬁnal reconstruction which regresses the actual distance values at dense query points, since the memory usage differs depending on the number of query points.
Our neural network is powerful, employing about 10 times more parameters than ConvOcc.
With the efﬁcient sparse representation, we can design the network to capture larger receptive ﬁelds with deeper network, enjoying more expressive power.
This is reﬂected in the detailed reconstruction results of our approach.
The memory usage is similar for smaller rooms for both approaches, but cGCA becomes much more efﬁcient for larger rooms.
When the grid size for room reaches 235, our method consumes 7 times less GPU memory compared to ConvOcc.
As widely known, the memory and computation for 3D convolution increases cubic to the grid resolution with dense representation, while those for sparse representations increase with respect to the number of occupied voxels, which is much more efﬁcient.
Therefore our sparse representation is scalable and crucial for scene completion.
E E FFECTS OF M ODE S EEKING S TEPS
[Figure]
We perform an ablation study on the effects of mode seeking steps, which removes the noisy voxels with low-occupancy probability as discussed in Sec. 3.2.
We test with ShapeNet dataset (Chang et al.
with voxel size 64 3 and neighborhood radius r = 3 , which tends to show the most intense effect of mode seeking phase due to large neighborhood size.
The right side of Fig.
9 shows the graph of metrics on quality (MMD), ﬁdelity (UHD), and diversity (TMD) with respect to mode seeking steps.
There is a dramatic decrease in all metrics on the ﬁrst step of mode seeking step.
The effect is visualized in the left side of Fig.
9, where the unlikely voxels are removed in a single mode seeking step.
After the ﬁrst mode seeking step, UHD and TMD metrics increase slightly, but overall, the metrics remain stable.
We choose T (cid:48)
= 5 for all the conducted experiments to ensure that shape reaches a stable mode.
[Table]
F A NALYSIS ON V ARYING C OMPLETENESS OF I NPUT
In this section, we further investigate the effects on the level of completeness of input.
Sec. F.1 shows results of completion models on sparse inputs and Sec.
F.2 provides how our model behaves when a non-ambiguous input is given.
F. R ESULTS ON S PARSE I NPUT
For the experiments of ShapeNet scene dataset, the models are trained and evaluated on the input where the ambiguity lies on the missing parts, but the remaining parts of input pointcloud are relatively clear by sampling 10,000 points, following the experiment setting of ConvOcc (Peng et al.
In this section, we further provide ambiguity to the remaining parts by sampling less points.
For the models trained on 10,000 point samples with 0.5 completeness, we evaluate the results on sparse input by sampling 500, 1,000, 5,000 and 10,000 points after applying the same iterative removal process as done during training.
As in Sec. 4.1, we measure the accuracy (CD) and diversity (TMD) metrics compared with ConvOcc (Peng et al.
, IFNet (Chibane et al.
(2020a)), and GCA (Zhang et al.
Table 5 contains the quantitative comparison results.
For all the models, accuracy (CD) degrades as the input gets sparser.
However, cGCA (w/ cond.) achieves best accuracy at all times, indicating that our model is the most robust to the sparsity of input.
For probabilistic models, the diversity (TMD) increases as the density decreases.
This shows that the diversity of the reconstructions is dependent to the ambiguity of the input.
This is intuitively a natural response, since the multi-modality of plausible outcomes increases as the partial observations become less informative.
Fig.
10 visualizes the results for various sparsity.
All of the models produce less plausible reconstructions with 500 points (leftmost column of Fig. 10) compared to that of higher density inputs.
However, we observe a clear distinction between the completions of probabilistic models (GCA, cGCA) and deterministic models (ConvOcc, IFNet).
While probabilistic models try to generate the learned shapes from the observations, such as a shade of the lamp, deterministic methods tend to ﬁll the gaps between points.
As discussed in Sec. 4.1, we hypothesize that this consequence stems from the fact that deterministic methods produce blurry results since it is forced to generate a single result along out of multiple plausible shapes which are the modes of the multi-modal distribution (Goodfellow (2017)).
Therefore, we claim that a generative model capable of modeling multi-modal distribution should be used for shape completion task.
F. R ESULTS ON N ON - AMBIGUOUS I NPUT
We present experiments where cGCA is given a non-ambiguous input.
For the models trained on various level of completeness on ShapeNet scene dataset, we evaluate each model on an input presenting distinct geometry.
The input is generated by sampling 100,000 points from the mesh without any procedure of removal.
Table 6 shows the quantitative results of the experiments.
The ﬁrst row shows the accuracy (CD) and diversity (TMD) scores of the models operated on test set when the removal procedure is same as that of the training set.
The second row shows the metrics when the input of the test set is sampled densely (100,000 points) without any removal.
For all models trained with varying level of completeness, accuracy increases with decreasing diversity given a clear shape compared to an ambiguous shape.
The result indicates that the diversity of the trained models is dependent on the ambiguity of the input.
c G C
A
G C A
I F N e t
C onv
O cc
I npu t c G C
A ( w / c ond . )
500 Points 1,000 Points 5,000 Points 10,000 Points
[Figure]
[Table]
This coincides with the result of Sec.
F.1 that the reconstructions of our model is dependent on the multi-modality of plausible outcomes of input.
We also observe that the diversity of reconstructions is associated with the training data.
Comparing the quantitative results on non-ambiguous input, the accuracy is low while the diversity is high when the model is trained on relatively incomplete data.
We hypothesize that models trained with higher level of incompleteness are more likely to generate shapes from the input.
However, Fig.
11 shows that while the completions of cGCA trained on highly incomplete data (min.
rate 0.2) are diverse, they are quite plausible.
G P ROBABILISTIC S CENE C OMPLETION ON R EAL D ATASET
We investigate how our model behaves on indoor real-world data.
We test on the ScanNet indoor scene dataset (Dai et al.
(2017a)), which is highly incomplete compared to other datasets, such as Matterport (Chang et al.
The input is sampled by collecting 500 points/m 2 from each scene.
ScanNet dataset does not have a complete ground truth mesh, so we test our model trained on 3DFront (Fu et al.
(2021)) dataset with completness of 0.8.
We align the walls of ScanNet data to xy-axis since the scenes of 3DFront are axis aligned.
We compare our result with GCA (Zhang et al.
The results are visualized in Fig.
Our model shows diverse results (e.g. chairs, closets) and generalizes well to the real data, which has signiﬁcantly different statistics compared to the training data.
Especially, the conditioned variant of our model shows better results by generalizing well to new data (e.g. tree), not found in 3DFront training dataset.
We emphasize that our model is trained on unsigned distance ﬁelds and does not require the sliding-window technique, unlike the previous methods (Peng et al.
; Chibane et al.
(2020a)).
c G
C A ( w / c o nd . )
c G
C A
G C
A
I npu t
H
A DDITIONAL R ESULTS FOR S CENE C OMPLETION
H.
A DDITIONAL R ESULTS FOR S HAPE N ET S CENE min.
rate 0.
min.
rate 0.
min.
rate 0.
c G
C A ( w / c ond . )
c G
C A
G C
A
I F N e t
C onv
O cc
I npu t min.
rate 0.
min.
rate 0.
min.
rate 0.
c G
C A ( w / c ond . )
c G
C A
G C
A
I F N e t
C onv
O cc
I npu t
H.
A DDITIONAL R ESULTS FOR
RONT
min.
rate 0.
min.
rate 0.
min.
rate 0.
[Figure]
( w
[Figure]
ond . )
[Figure]
G C
A
I npu t c G
C A ( w / c ond . )
c G
C A
G C
A min.
rate 0.
min.
rate 0.
min.
rate 0.
I npu t
I A DDITIONAL R ESULTS FOR S INGLE O BJECT C OMPLETION