D EEP P OINT C LOUD
R ECONSTRUCTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Point cloud obtained from 3D scanning is often sparse, noisy, and irregular.
To cope with these issues, recent studies have been separately conducted to densify, denoise, and complete inaccurate point cloud.
In this paper, we advocate that jointly solving these tasks leads to signiﬁcant improvement for point cloud reconstruction .
To this end, we propose a deep point cloud reconstruction network consisting of two stages: 1) a 3D sparse stacked-hourglass network as for the initial densiﬁcation and denoising, 2) a reﬁnement via transformers converting the discrete voxels into 3D points.
In particular, we further improve the performance of transformer by a newly proposed module called ampliﬁed positional encoding .
This module has been designed to differently amplify the magnitude of positional encoding vectors based on the points’ distances for adaptive reﬁnements.
Extensive experiments demonstrate that our network achieves state-of-the-art performance among the recent studies in the ScanNet, ICL-NUIM, and ShapeNetPart datasets.
Moreover, we underline the ability of our network to generalize toward real-world and unmet scenes.
Implementation will be released after the review.
I NTRODUCTION
[Figure]
We propose a novel neural architecture that jointly solves inherent shortcomings in raw point cloud, such as noise, sparsity, and incompleteness.
3D scanning devices, such as LiDAR and RGB-D sensors, allows to quickly and accurately reconstruct a scene as a 3D point cloud.
This compact 3D representation is commonly used to achieve various tasks in autonomous driving (Yang et al., 2018;
Qi et al., 2018; Shi et al., 2020;
Lang et al., 2019), robotics (He et al., 2021; 2020; Wang et al., 2019), or 3D mapping (Choe et al., 2021b; Choy et al., 2020).
However, processing raw point cloud happens to be particularly challenging due to their sparsity, irregularity and sensitivity to noise (Xiang et al., 2019;
Guo et al., 2020).
To address these issues, recent deep learning-based studies have proposed to improve the quality of point cloud.
Speciﬁcally, prior works that are dedicated to point cloud reﬁnement can be categorized into three distinct subﬁelds: (1) point upsampling (Yu et al., 2018b), (2) point denoising (Rakotosaona et al., 2020), and (3) point completion
(Yuan et al., 2018).
Each of these tasks has its own beneﬁts and inconveniences.
For instance, the point upsampling task usually handles noisy input points but do not explicitly deal with strong outliers.
On the other hand, point denoising techniques have been designed to ﬁlter noisy data but cannot densify the point cloud.
Finally, point completion methods mainly focus on object-level full 3D completion but are known for their weak generalization to unmet environments or unknown category of objects.
Though there exist differences among the tasks, their common goal is to improve proximity-to-surface in the 3D point representation.
Thus, we believe in the complementary of these three sub-tasks and hypothesize that a joint framework is greatly beneﬁcial to deal with deteriorated and incomplete 3D point cloud.
Accordingly, this paper challenges to resolve all these issues as a single task, called point cloud reconstruction as depicted in Fig.
To the best of our knowledge, this paper is the ﬁrst attempt to jointly resolve the inherent shortcomings of point cloud obtained from 3D scanning devices: sparsity, noise, irregular- ity, and outliers.
To this end, we propose a deep point cloud reconstruction network that consists in two stages: a voxel generation network and a point re-localization network.
In the ﬁrst stage, the voxel generation network (Sec. 3.1) aims to densify voxels and remove outliers via sparse convolution layer (Choy et al., 2019).
Rather than to use k -Nearest Neighbor that is sensitive to points’ density (Mao et al., 2019), we utilize voxel hashing with sparse convolution layer to understand absolute-scale 3D structures for densiﬁcation and denoising.
Despite its success, voxelization inevitably leads to information loss due to the discretization process.
To provide a ﬁne-grained reconstruction, in the 2 nd stage, we propose a point re-localization network that converts discrete voxels into 3D points using transformer.
Additionally, we increase the performance of transformer by the ampliﬁed positional encoding .
In our analysis, spatial frequency plays an important role in the context of voxel-to-point conversion.
Our new positional encoding strategy is useful to infer descriptive and detailed point cloud by simply changing the amplitude of the encoding vector.
Our contributions can be summarized as follows:
• New problem formulation: point cloud reconstruction.
• Novel two-stage architecture for voxel-to-point reﬁnement.
Unique embedding scheme for point cloud, ampliﬁed positional encoding.
• A large number of experiments illustrating the generalization ability of our point cloud reconstruction to real 3D scans.
R ELATED W ORKS
Point cloud obtained from 3D scanning devices are known to contain various artifacts (Berger et al., 2017), such as noise, outliers, irregularity, and sparsity.
Point cloud reconstruction aims at resolving these aforementioned issues in order to provide ﬁne-grained 3D reconstruction.
In this section, we introduce different approaches related to point cloud reﬁnement.
Point Cloud Denoising.
Without extra information, such as RGB images, point cloud denoising purely based on the input point distribution is a challenging task (Lee, 2000; Avron et al., 2010).
Recent deep learning-based strategies (Rakotosaona et al., 2020; Roveri et al., 2018; Hermosilla et al., 2019; Pistilli et al., 2020;
Luo & Hu, 2021) demonstrate promising results.
However, despite the large improvement provided by these architectures, these methods do not have the ability to densify the reconstruction.
Point Cloud Completion.
A family of solutions called point cloud completion have recently been proposed to alleviate the problem of incomplete reconstruction.
One of the pioneer papers (Yuan et al., 2018) proposes to overcome the point incompleteness problem by generating points in both non-observed and visible areas.
To improve the completion process, recent approaches (Huang et al., 2020; Zhang et al., 2020; Wen et al., 2021; Xiang et al., 2021) propose to include prior structural information of the objects.
While these object-speciﬁc point completion techniques can lead to accurate results, they suffer from a lack of versatility since the object class is assumed to be known beforehand.
Without such semantic level a priori, the entire completion in occluded surface ( e.g ., behind objects) is a doubtful task.
Thus, in this paper, we focus on a reliable and generic object-agnostic approach that is able to complete visible areas within an arbitrary point cloud and to cope with sparse, noisy, and unordered point cloud.
Point Cloud Upsampling.
Given a sparse 3D reconstruction, the goal of point cloud upsampling is to densify the point cloud distribution.
Conventional methods (Alexa et al., 2003; Huang et al., 2013) rely on point clustering techniques to re-sample points within the 3D space.
Recent deep learningbased studies (Li et al., 2021; Yu et al., 2018a; Yifan et al., 2019; Yu et al., 2018b) propose point densiﬁcation networks under the supervision of dense ground truth points.
Though some papers internally conduct point denoising (Yu et al., 2018a; Yifan et al., 2019), most of these works mainly assume that there are no strong outliers among the input points input points, such as ﬂying-point cloud (You et al., 2019).
Surface Reconstruction.
Point cloud meshing
is a common solution for surface reconstruction (Hoppe et al., 1992; Curless & Levoy, 1996).
For instance, this process can be achieved by converting point cloud to an implicit surface representation via signed distance function (SDF) and to apply Marching Cubes (Lorensen & Cline, 1987) or Dual Contouring (Ju et al., 2002) for meshing ( e.g ., Poisson surface reconstruction (Kazhdan et al., 2006)).
This strategies is known to be effec-
[Figure]
tive under favorable conditions but can hardly handle sparse and non-uniform points’ distribution.
Moreover, such technique tends to lead to over-smoothed and undetailed reconstruction (Berger et al., 2017).
Similar limitations can be observed with recent deep learning-based SDF representation (Park et al., 2019).
Many other solution have been developed for the problem of surface reconstruction, such as, Delaunay triangulation (Rakotosaona et al., 2021; Boissonnat, 1984; Kolluri et al., 2004; Boissonnat & Oudot, 2005; Gopi et al., 2000), alpha shapes (Edelsbrunner & M ¨ ucke, 1994), or ball pivoting (Bernardini et al., 1999).
Despite their compelling performances, these approaches require uniform, dense, and accurate inputs since they usually preserve the original distribution of a given point cloud.
This assumption still holds for the recent studies (Sharp & Ovsjanikov, 2020;
Rakotosaona et al., 2021).
Compared to surface reconstruction, point cloud reconstruction can be viewed as a prior-process to overcome natural artifacts: sparsity, noise, irregularity.
Furthermore, in contrast to previous studies about point cloud reconstruction, our method aims to perform object-agnostic and well generalized point cloud reconstruction that is robust to overall shortcomings in the real-world 3D point cloud.
To this end, we introduce our deep point cloud reconstruction network that consists of the voxel generation network and the point re-localization network.
D EEP P OINT C LOUD R ECONSTRUCTION
Given a noisy and sparse point cloud P in , point cloud reconstruction aims to generate a dense and accurate set of points P out .
For this purpose, we propose a deep point cloud reconstruction network composed of two stages: voxel generation network (Sec. 3.1) and point re-localization network (Sec. 3.2).
The ﬁrst network has been designed to perform a densiﬁcation and outliers removal in the sparse voxel space.
The second network complements and improves this ﬁrst stage by denoising the data further and by converting voxels back to 3D points.
The overall architecture is illustrated in Fig.
3. V OXEL G ENERATION N ETWORK
Most previous studies (Li et al., 2021; Yifan et al., 2019; Yu et al., 2018a;
Luo & Hu, 2021) attempts to resolve these problems by directly processing a raw point cloud.
However, it is known to be a challenging problem due to the unordered nature of the 3D point cloud.
To tackle this issue, we propose to convert the 3D point cloud P in into a sparse voxel representation V in through our voxel generation network.
From this data, our network generates a reﬁned and densiﬁed volume V out .
A noticeable advantage of sparse voxel representation is its ability to preserve the neighbor connectivity of point cloud in a uniform and regular 3D grids (Guo et al., 2020) via voxel hashing (Choy et al., 2019; Graham & van der Maaten, 2017).
Voxel hashing keeps tracking of the geometric positions of sparse tensors so that we can ﬁnd the voxels’ neighbors to apply sparse convolution.
Moreover, sparse convolution layer can cover the consistent and absolute-scale scope of the local region and hierarchically capture its large receptive ﬁelds.
Such property is then particularly beneﬁcial to deal with an inaccurate and unordered input point cloud P in .
In order to denoise and densify V in , our voxel generation network consists of a sparse stacked hourglass network as shown in Fig.
Our network architecture is close to the previous studies (Newell et al., 2016b; Chang & Chen, 2018; Im et al., 2019;
Choe et al., 2019) in that it is composed of several hourglass networks for cascaded reﬁnements of images or dense volumes.
In the context of this paper, scanning devices can only reconstruct the surface of the structure, thus, the resulting voxelized
[Figure]
+ + + + + + + + + + + + 𝑉 out 𝑉 in
C 𝐹 v 𝑉 mid volume is inherently sparse.
Therefore, to process this 3D volume efﬁciently, our voxel generation network takes advantage of a sparse 3D convolution strategy inspired by recent works (Choy et al., 2019; Graham & van der Maaten, 2017).
It should be noted that the speciﬁcity of sparse convolution operation is particularly desirable for the tasks at hand.
First, generative sparse transposed convolution layer (Gwak et al., 2020) has the ability to create new voxels given a pre-deﬁned upsampling rate r , which can be seen as a point upsampling methodology (Yu et al., 2018b).
Secondly, voxel pruning is widely used for ﬁne-grained volume rendering (Liu et al., 2020) and efﬁcient computation (Choy et al., 2019).
In our context, voxel pruning is closely related to outlier removal.
Last, the joint use of generation and pruning enables our network to create an arbitrary number of points.
As illustrated in Fig.
3, we ﬁrst compute local voxel features using sparse convolutional blocks in an encoder part.
The aggregated voxel features undergo the voxel generation layers and pruning layers in a decoder part.
We use two hourglass networks to densify and prune voxels.
To train the voxel generation network, we adopt a Binary Cross-Entropy loss (BCE) to classify the status of estimated voxels into an occupied/empty state as:
[Equation]
where x v is an estimated class of a generated voxel v from the network, and y v is the ground truth class at the generated voxel v .
N v is the total number of voxels that we used to compute the BCE loss.
If v is on the 3D surface, y v becomes true class ( i.e ., y v =1 ).
Furthermore, following the original loss design (Newell et al., 2016b), we compute the BCE Loss L BCE for both the ﬁnal voxel prediction V out and the intermediate results V mid .
In total, we compute the voxel loss L vox as:
[Equation]
where N v is the total number of voxels in the intermediate inference volume V mid that we used to calculate the BCE loss.
Along with the predicted voxels V out , we extract voxel features F v using sparse interpolation as illustrated in Fig.
3. P OINT R E - LOCALIZATION N ETWORK
Despite the denoising and densiﬁcation offered by the 1 st stage network, voxelization inevitably leads to information loss.
Thus, in this section, we propose a point re-localization network that converts the discrete voxels V out ∈ R N × 3 into 3D points P out ∈ R N × 3 .
This conversion requires an understanding of local geometry to describe the 3D surface as a group of point sets.
Inspired by (Zhao et al., 2021), we utilize self-attention as a point set operator.
Let us describe the detailed process in the point re-localization network, which is illustrated in Fig.
Given output voxels V out , we collect the K (=8) closest voxels to each voxel v i ∈ R 1 × 3 using the hash table that we used in the 1 st stage network.
Then, we obtain a voxel set V i
= { v k } Kk =1 that is referenced to the query voxel v i .
Similarly, we cluster a set of voxel features F i = { f k } K k =1 that correspond to P i .
As in Fig. 4, the 2 nd stage network regresses the location of a query point p
i (red voxel) using the geometric relation between the query voxel v i and its neighbor V i (white voxels) through self/cross attention layers.
While this formulation is close to PointTransformer (Zhao et al., 2021), our point re-localization network has three distinctive differences.
First, PointTransformer utilizes k -Nearest Neighbors to
[Figure]
PE
[Figure]
group 3D points but we re-use the voxel hashing that covers absolute-scale local geometry.
Second, our network consists of both self-attention and cross-attention.
In our analysis, it is better to use two different attention layer than to solely adopt self-attention.
Third, we design our own positional encoding system that controls the necessity of high-frequency signal by changing the amplitude of the encoding vector, called ampliﬁed positional encoding .
Originally, positional encoding has been introduced to help transformer to understand the relative positions between tokens ( e.g ., word embeddings) in a given sentence (Vaswani et al., 2017).
When applied to other domains of applications, positional encoding is often used to preserve high-frequency signals which tend to be disregarded by neural networks (Rahaman et al., 2019).
For example, positional encoding is proven to be essential to synthesize sharp images from an implicit 3D neural representation (NeRF (Mildenhall et al., 2020)).
In our purpose, we need to consider spatial frequency to coherently re-locate a query voxel v i by using its neighbor set V i .
Our intuition is simple.
Since positional encodings are mainly effective for the high-frequency embedding.
As in Fig.
4, we increase the amplitude of positional encoding when we need to move a query voxel by a small amount of distance ( i.e ., high-frequency in 3D spatial domain).
If a query voxel is far away from its neighbor, we decrease the amplitude of the positional encoding.
Based on the relation between high frequency signals and our point re-location scheme, we deﬁne our ampliﬁed positional encoding γ ampPE as:
[Equation]
where γ ampPE is an ampliﬁed positional encoding vectors and (cid:107) · (cid:107) 11 is the L1-norm.
Following the conventions (Vaswani et al., 2017), γ PE is 1D positional encoding vectors, pos represents position, i means dimension, and C is channel dimensions of voxel embedding.
After we encode γ ampPE into the voxel embedding, we aggregate this embedding through self-attention layers and cross-attention layers to ﬁnd relationships among the center voxel and its neighbors.
Finally, the regressed offset ∆ v is added to the original location of center voxel v c to update its location.
To train the second stage network, we calculate a chamfer distance loss L dist as:
[Equation]
where P pred is the predicted point cloud, P gt indicates ground truth point cloud, ˜ p represents a predicted point, and p means a ground truth point.
N P pred and N P gt are the number of prediction and ground truth point cloud, respectively.
Through our two-stage pipelines, our whole network performs point cloud reconstruction.
By jointly generating, removing, and reﬁning points, our network can robustly recover accurate point cloud and increase the level of proximity-to-surface.
[Table]
E XPERIMENT
In this section
[Table]
we describe the implementation details of our network and we present a large series of assessments underlying the effectiveness of our approach.
Our point reconstruction network has been solely trained on the ShapeNet-Part (Yi et al., 2016) dataset but tested on other real and synthetic datasets such as ScanNet (Dai et al., 2017) and ICL-NUIM (Handa et al., 2014).
These experiments highlight the ability of our solution to generalize well to other scenarios involving various levels of noise and different densities.
4. I MPLEMENTATION
We ﬁrst train our voxel generation network for 10 epochs using the Adam optimizer (Kingma & Ba, 2014) with initial learning of 1 e - 3 and a batch size of 4.
We decrease the learning rate by half for every 2 epochs.
After that, we freeze the voxel generation network and initiate the training of the point re-localization using the inferred voxels V out from the voxel generation network.
The point relocation network is trained using identical hyper-parameters used for training the ﬁrst stage network.
Further details regarding the training rules are provided in Sec. 4.3.
4. B ASELINE A PPROACHES
We conduct a thorough comparison against previous state-of-the-art techniques: PU for point cloud upsampling (Li et al., 2021), PD for point cloud denoising (Luo & Hu, 2021), and PC for point cloud completion (Xiang et al., 2021).
These three papers are currently positioned at the highest rank for each task and we utilize their ofﬁcial implementation for training and evaluation.
all methods are trained with the same data augmentation as competing approaches and its performances have been measured on various datasets: ShapeNet-part (Yi et al., 2016), ScanNet (Dai et al., 2017), and
ICL-NUIM (Handa et al., 2014) datasets.
In particular, we analyze the efﬁcacy of ours compared to the combination of the existing studies.
For instance in Table 1, PC( r =4 ) → PU( r =4 ) → PD represents consecutive operation of point completion, and point upsampling and point denoising under upsampling ratio r =4 .
Note that we also take into account the point completion task since this task also considers point densiﬁcation on visible surface.
4. D ATASET
ShapeNet dataset.
We utilize 13 categories of different objects categories from the ShapeNetPart dataset (Chang et al., 2015; Yi et al., 2016).
We extract dense and uniform 10 K ground truth
[Figure] [Figure]
point cloud per object by using the Poisson Disk Sampling algorithm
(Yuksel, 2015) provided by
Open3D (Zhou et al., 2018).
Among these ground truth 3D points, we randomly sample 2048 points as an input P in for the training networks.
For the sake of fairness, we adjust our hyper-parameter l vox to 0 . 0200 for training our network.
This is to produces a comparable number of points than previous studies.
We follow a data augmentation scheme proposed by the previous point upsampling study (Li et al., 2021), such as random noise addition and random re-scaling.
Additionally, we include random outliers that constitute less that 5 percent of input points.
To train and validate the networks, we carefully follow the ofﬁcial train/val/test split provided by (Yi et al., 2016).
ScanNet / ICL-NUIM dataset.
To demonstrate the generalization capabilities of our point cloud reconstruction network, we propose to test our method and previous studies in unmet datasets.
In particular, we propose to reﬂect the versatility of our work via the evaluation process proposed in image-based multi-view stereo studies (Yao et al., 2018) 1 .
In particular, we choose real-world datasets to show the applicability of the point cloud reconstruction task.
ScanNet
(Dai et al., 2017) is an indoor dataset that consists of 800 real-world scenes.
Each scene provides RGB-D sequences that are captured by 3D sensors.
Following the ofﬁcial guideline, we utilize 100 test scenes from which we randomly sample 200 frames for evaluation.
For integrity, we also propose an extensive evaluation on the ICL-NUIM dataset (Handa et al., 2014) which provides synthetic ground truth RGB-D sequences.
Since the quality of the ground truth depth maps in ScanNet dataset are measured from low-quality 3D sensors, we utilize ICL-NUIM dataset for the precise comparison.
Within ICL-NUIM dataset, we use four scenes (livingroom1, livingroom2, ofﬁce1, ofﬁce2), and totally sample 200 RGB-D pairs for evaluation.
Note that we apply identical conditions as used in ScanNet.
4. C OMPARISON
For metric computation, we Chamfer distance and percentage metrics ( e.g ., Accuracy, Completeness, and f -score)
(Yao et al., 2018; Jensen et al., 2014), as shown in Table 1.
In particular, we extend to measure the k -Nearest Chamfer distance ( i.e ., k -Chamf. 2 ) that efﬁciently computes pointwise distance than Earth Mover’s distance.
Also, this formulation considers more nearest neighbor points than the original Chamfer distance ( i.e ., k =1 ).
Based on these criteria, we calculate the metrics of the whole methods.
As shown in Table 1, our network achieves state-of-the-art performance compared to the previous studies (Li et al., 2021;
Luo & Hu, 2021; Xiang et al., 2021).
Moreover, the performance gap increases when we conduct generalization test in ScanNet dataset and ICL-NUIM dataset.
There are two dominant reasons behind the better generalization and transferability.
First, each sparse convolution layer has consistent and absolute-scale receptive ﬁelds in 3D space.
This local voxel neighborhood appears to be more robust than raw point cloud processing via k -Nearest Neighbor (Mao et al., 2019) as adopted in the three baselines (Xiang et al., 2021;
Luo & Hu, 2021; Li et al., 2021).
This is also one of the strength of using voxelization as an initial processing of point cloud reconstruction, which is conducted by our 1 st network.
Second, our network does not have task-speciﬁc header networks or task-oriented losses for completion/upsampling/denoising tasks.
Our network purely focuses on the inherent problems resided in raw point cloud.
Such joint approach trains our network to improve the overall proximity-to-surface, not the task-speciﬁc quality.
Regardless the order in which we apply the baselines, our method systematically outperforms them.
When we sequentially apply point densiﬁcation and point denoising, the performances are not constantly improved in Table 1.
This observation is supported by the qualitative results in Fig.
While point denoising can effectively remove the remaining outliers that are generated from (Xiang et al., 2021) or (Li et al., 2021), it has chance to remove the wrong points sets.
For instance in Fig.
4, the point denoising method erases part of the airplane’s wing.
In contrast, our voxel generation network conducts both voxel pruning and generation in each hourglass network so that it can effectively circumvent such case.
[Table]
4. A BLATION STUDY
In this section we propose to evaluate the contribution of each proposed component (ampliﬁed positional encoding and point re-localization network), through an extensive ablation study in Table 2.
In Table 2-(a)
[Table]
we validate our ampliﬁed positional encoding in comparison with different styles of positional encodings, γ PE (Vaswani et al., 2017) and PE* (Mildenhall et al., 2020).
Also, we include an additional experiment that does not use any positional encodings ( i
[Table]
e ., No PE in Table 2-(a)).
We measure the accuracy by calculating the Chamfer distance and f -score.
It demonstrates that our method outperforms the previous embedding schemes.
We deduce that by using the geometric position priors of point cloud, we can intentionally magnify or suppress the position encoding vectors that are usually beneﬁcial for the control of high-frequency embedding in 3D spatial domain.
In Table 2-(b), we conduct experiments on different composition of point re-localization network.
Our network involves sparse stacked hourglass network ( i.e ., S-Hourglass), self-attention layers ( i.e .,
Self-attn in Table 1), and cross-attention layers ( i.e ., Cross-attn in Table 1).
By intentionally omitting certain modules in the point re-localization network, we measure the quality of point reconstruction using the Chamfer distance and f -scores.
Despite reasonable results from the voxel generation network ( i.e . 1 st stage in Table 1), it turns out that self- and cross-attention layers further increase the accuracy of of the point cloud reconstruction.
In particular, the combined usage of self- and crosslayers outperforms other strategies.
To ensure a fair comparison, the self-attention only method and cross-attention solely network adopts more transformers to balance with our ﬁnal network design.
So far, we demonstrate the effectiveness of our network design through a fair comparison with previous studies and an extensive ablation studies.
Our network aims at point cloud reconstruction that can be applied in general cases to achieve high-quality point reﬁnement.
C ONCLUSION
Contrary to previous approaches – individually addressing the problems of densiﬁcation and denoising, we propose an elegant two-stage pipeline to jointly resolves these tasks.
As a result of this uniﬁed framework, the quality of sparse, noisy and irregular point cloud can be drastically improved.
This claim has been validated through a large series of experiments underlying the relevance and efﬁciency of this joint reﬁnement strategy.
Speciﬁcally, qualitative and quantitative results demonstrate signiﬁcant improvements in terms of reconstruction’s accuracy and generalization when compared with prior techniques.
Our point cloud reconstruction strategy paves the way towards more ﬂexible, multi-tasks and effective architectures to reﬁne point cloud’s quality.
Despite our success, there exist remaining issues, such as noise modeling for better denoising and point-voxel fusion design
A PPENDIX
In this appendix
[Table]
we describe further details of the proposed methodology, deep point cloud reconstruction.
First, we provide the additional quantitative results (Sec. A).
Second, we describe precise equations of the metrics that we used for evaluation (Sec. B).
Third, we present qualitative comparisons of ours and baseline methods (Sec. C).
Last, we include our response letter for the reviewers’ convenience (Sec. D).
A Q UANTITATIVE RESULTS
Along with Table 1 of the manuscript
[Table]
we provide additional results that we did not include in the manuscript.
[Table]
B M ETRICS B. P ERCENTAGE METRICS
Let us describe the precise equation of the metrics that we used in Table 1 of the manuscript and
Table 3 of the appendix.
Let P GT be ground truth point cloud, P pred a reconstructed points.
For each inferred point p ∈P pred , its distance to ground truth is deﬁned as:
[Equation]
where g is a single point included in the ground truth point cloud P GT .
These distances are gathered to calculate Accuracy ( i.e ., Acc.) as:
[Equation]
where N P pred is the number of reconstructed points P pred , [ · ] is the Iverson bracket, and d thresh is a threshold distance.
In a similar manner, we calculate the distance from ground truth point g ∈P GT to the reconstructed points P pred as:
[Equation]
These distances are gathered to calculate Completeness ( i.e ., Comp.)
as:
[Equation]
where N P GT is the number of ground truth point clouds.
To calculate f1-score, we use Accuracy and
Completeness as below:
[Equation]
Note that in ShapeNet dataset, we set d thresh =0 . 0200 .
For ScanNet and ICL-NUIM dataset, we set d thresh =0 .
0050 for evaluation.
We mostly follow these metrics from (Jensen et al., 2014; Aanæs et al., 2016; Yao et al., 2018; Knapitsch et al., 2017).
B. C HAMFER DISTANCE
Before we explain the k -Nearest Chamfer distance, let us revisit the original Chamfer distance.
Based on the equations Eq. 8 and Eq. 6, we describe the Chamfer distance ( i.e ., Chamf.)
as:
[Equation]
Since the original Chamfer distance only considers the closest distances, it has difﬁculty in describing the optimal distance between two point sets having different number of points (Fan et al., 2017).
However, rather than to use Earth Mover distance that requires quadratic computational power and memory consumption, we formulate k -Nearest Chamfer distance by considering the k closest points instead of the single closest point ( i.e ., k =1 ).
Let us deﬁne the distance from a inferred point p ∈P pred to the k closest ground truth points g k ∈P GT as follow:
[Equation]
[Equation]
where g k is the k -th closest point to the inferred point p .
Similarly, we calculate the distance from a ground truth point g to the k -closest reconstructed points p k as follows:
[Equation]
Based on these equations, we compute k -Nearest Chamfer distance ( i.e ., k -Chamf.)
as below:
[Equation]
pred ) .
C Q UALITATIVE RESULTS
[Figure]
D R ESPONSE LETTER
Thank you for the precious comments.
Following the reviewer’s advice, we revise our initial manuscript as follows:
• Elaborate the related works section about a surface reconstruction task.
• Clarify the Sec. 3.2.
• Conduct experiments about possible combinations of three previous papers, which is added in Table 1 of the manuscript and Table 3 in the appendix.
• Include description in Sec.
4.4 about the strength of the proposed method, generalization and transferability.
• Add more qualitative results in the appendix.
• Describe the precise equations of metrics in the appendix.
• Implementation will be released after the review.
With this revision, the initial manuscript becomes more reasonable, analytic and feasible.
We hope that all reviewers are satisﬁed with the improved quality of the paper.
For the reviewers’ convenience, we include our response letter in the appendix.
D. R EVIEWER : VE KS
[Writings] “Q.1-1.
The high-level idea makes lots of sense to me.
Using voxelization to denoise and complete irregular point cloud is reasonable and smart.
Besides, we can sample as many points as we want from a voxel grid to get a dense surface.
If possible, the authors could further elaborate on these contributions.”
Thank you very much for this positive remark regarding using voxelization for this task.
As suggested, we underlined the ﬂexibility offered by our voxel based technique in Sec.
4.4 of the manuscript.
This advantage is also highlighted by the quantitative results presented in Table 1 of the manuscript and Table 1 of the appendix.
These results demonstrate that the number of output points P out can be adjusted by the hyper-parameters while preserving the overall reconstruction quality.
[Baseline] “Q.1-2.
The presented results are great, but the baselines are not very considerate.
This paper is arguing that the three tasks should be uniﬁed and hence multi-task joint-learning baselines should be considered.
The simplest setting is to ensemble all three baselines together in an optimized order.
Alternatives contain training each baseline method on all three tasks jointly.”
Admittedly, our initial experimental setup was lacking fairness, therefore, we have re-conducted a large set of experiments where all possible combinations between the three baselines have been investigated.
The obtained results as visible in Table 1 of the manuscript and Table 3 of the appendix.
Please take an example with some notations used in the tables in Table 1 of the manuscript.
For instance, PC( r =4 ) → PU( r =4 ) → PD represents the consecutive operation of point completion, and point upsampling and point denoising under upsampling ratio r =4 .
Note that we also take into account the point completion task since this task also considers point densiﬁcation on the visible surfaces.
None of the tested conﬁgurations is qualitatively nor quantitatively competitive with our point reconstruction technique.
It should however be noted that since only 2048 points have been used as input, the point denoising method (Luo & Hu, 2021) could not be used before upsampling (because (Luo & Hu, 2021) requires at least 10,000 points to operate).
Thus, we cannot provide results for every single possible combination.
Nonetheless, our method still achieves state-of-the-art performance among the possible orders of point denoising/completion/upsample networks.
[Multi-task learning] “Q.1-3.
The proposed method should also be evaluated for each single task to demonstrate advantages.”
We understand the intention of the reviewer veKS.
We guess that the worrying points derive from the reviewer veKS’s initial description, “I will use the word **multi-task** to refer to the problem setting in this work, in contrast to the single-task baselines.”
If this paper can be viewed as multitask learning of three different tasks, it is important to evaluate the results on each single task, such as denoise, densiﬁcation and completion task.
However, our point reconstruction technique cannot truly be considered as a multi-task learning pipeline.
There are two reasons.
First, the series of problems (sparsity, noise, and irregularity) are inherently resided in raw point clouds.
In real-world point clouds, these problems are completely mixed.
Second, we solve the problems in a completely joined manner.
As a result, we cannot isolate each individual task since our network does not have dedicated to design task-speciﬁc header networks nor task-oriented losses.
Moreover, even if we forcefully obtain the results in each single task, it will lack fairness and would not reﬂect our intention nor the capability of our point reconstruction technique.
We hope to have clariﬁed this particular point and that the reviewer will understand the context for which we have designed this point reconstruction strategy combining three tasks at once to resolve the inherent shortcomings of 3D point clouds.
[Ampliﬁed positional encoding] “Q.1-4.
I am not fully convinced by the story of positional encoding.
In the original form, the distance information is already expressed as the frequency of the signal.
The magnitude is just redundant information.
Why is it important?
And what kind of effect it will cause precisely?”
We apologize for the confusion.
The intuition behind our amplitude positional encoding is the following, when to embed high frequency signal?
We are inspired by NeRF (Mildenhall et al., 2020) where positional encoding results in preserving the high-frequency image signal.
In our case, we are facing a different situation.
high-frequency signal is not always welcomed.
When a voxel needs to move to a distant location, it is better to keep the low frequency signal, which is simply operated by decreasing the amplitude of the positional encoding vector as described in Fig.
4 of the manuscript.
Such control is possible since we already know the relative distance between voxels that are estimated from the 1 st stage network.
As underlined by the reviewer G4Aq, we believe that our ampliﬁed encoding is an important contribution of this work.
To better highlight the necessity of this component, we further elaborate the paragraphs about our ampliﬁed positional encoding in Sec. 3. of the manuscript.
[Generalization] “Q.1-5.
For the tested dataset, especially this synthetic ones (ICL-NUIM), how does the input noise/sparsity/incompleteness are created? ...
Also, regular synthetic noise ( e.g .,
Gaussian noise) are much simple than real-world noise.
How does the proposed framework deal with real-world noise? ...
It is unclear how does this framework work for real-world sparse point clouds ( e.g ., the remote areas in LiDAR data).”
Thank you very much for this remark regarding the transferability of the approach.
As the reviewer veKS mentioned, we train our network only in synthetic/object-scale dataset (ShapeNetPart).
Then, without ﬁne-tuning, we utilize the pre-trained weights to measure the quality of reconstruction using unmet point clouds in ScanNet or ICL-NUIM datasets.
Despite the domain gap between datasets, we think our network generalizes well toward various real datasets.
We conjecture that this transferability can be explained by two dominant reasons.
First, we optimize the nature of the objective function.
Indeed improving the proximity-to-surface instead of task-speciﬁc losses (such as denoising, densiﬁcation, and completion) seems to be a more generic and generalizable problem.
Regardless of the difference in synthetic and real-world noises, our network is trained to improve the quality of reconstruction.
Also, this is clearly different from the point completion task (Yuan et al., 2018) that concentrates on category-speciﬁc datasets for generating points in occluded regions of objects.
Second, we utilize the hash table for sparse convolution layer instead of k Nearest Neighbor.
To handle noisy and sparse point cloud, k NN is not a proper choice since it is sensitive to the density of point clouds (Mao et al., 2019).
Nonetheless, previous studies (Li et al., 2021;
Xiang et al., 2021;
Yu et al., 2018b) mainly rely on this clustering algorithm.
In contrast, sparse convolution layer takes absolute scale of receptive ﬁeld regardless of the input distribution, which results in better transferability toward unmet and real-world scenes.
Note that we apply random Gaussian noise into ICL-NUIM dataset following (Li et al., 2021).
This synthetic noise is truely much simpler than the real-world noise, as addressed by the reviewer.
Since we did not precisely analyze the noise modeling, we have improved the exposition in the paper to clarify the discussion.
[Sparse stacked hourglass network] “Q.1-6.
3D sparse stacked hourglass network is not a super popular network.
It is also not implemented in popular sparse conv net framework ( e.g ., Minkowski engine).
Why this network is adopted?
Maybe the authors can share some related works to justify the advantage.
It is not a common choice.”
In our understanding, stacked hourglass has unique properties and is widely used for various applications.
First of all, we found that the total number of citations of stacked hourglass network is 3364 (Newell et al., 2016b) and
(Newell et al., 2016a).
Note that the original stacked hourglass network is proposed in (Newell et al., 2016b) and the modiﬁed version of the network is introduced in (Newell et al., 2016a) by the same ﬁrst author.
Moreover, the variants of stacked hourglass network largely used in various tasks.
For instance in stereo matching network, PSMNet (cite: 706 ) (Chang & Chen, 2018), GANet (cite: 246 ) (Zhang et al., 2019), AANet (cite: 76 ) (Xu & Zhang, 2020), utilizes a volumetric version of hourglass network using 3D CNNs.
This paper clearly mentions their stacked hourglass network in their abstract, ”The 3D CNN learns to regularize cost volume using stacked multiple hourglass networks in conjunction with intermediate supervision.”
In another example, multi-view stereo task, DPSNet (cite: 98 ) (Im et al., 2019) also adopts 3D stacked hourglass network to aggregate cost volumes.
Different from deep and wide U-shape network, such as Minkowski-UNet (Choy et al., 2019), using multiple light encoder-decoder networks ( i.e ., stacked hourglass network) aims to sequentially reﬁne feature vectors by intermediate losses.
In summary, our 1 st stage network aims to sequentially reﬁne the given point cloud through the proposed sparse stacked hourglass network.
Similar to the idea of voxel carving, hourglass networks re-shape the sparse points in a coarse-to-ﬁne manner.
We hope that the reviewer veKS is satisﬁed with our understanding of stacked hourglass network (Newell et al., 2016b).
Also, we expect the future studies about point cloud reconstruction will widely use this sparse version of stacked hourglass network as stereo matching methods (Chang & Chen, 2018; Xu & Zhang, 2020; Zhang et al., 2019) and multi-view stereo (Im et al., 2019;
Choe et al., 2021a)
studies do.
[Missing details] “Q.1-7.
How many voxels are processed in the transformer on average?”
We apologize for the missing details.
We set k = 8 as an input of our transformer.
This information is included in Sec.
3.2 of the manuscript and Table 3 of the appendix.
[Missing details] “Q.1-8.
How is the dimension of P out decided?”
Since our 1 st stage network adaptively generates and prunes voxels, we cannot precisely decide for the dimension of P out ∈ R N (cid:48) × 3 where N (cid:48) is determined after the inference.
Thus, we report the average cardinality of the generated point clouds in Table 1 of the manuscript.
[Related works] “Q.1-9.
The difference between point upsampling and point completion is not very clear.
The very speciﬁc deﬁnition should be made in the very early stage of the writing so that there is no ambiguity between the challenges that these two problems pose. ”
The difference between the different tasks has been described in the ”related works” section.
The goal of point upsampling is to densify sparse but homogeneously distributed input points.
On the other hand, the point completion task, as described in the pioneering paper (Yuan et al., 2018), aims at extrapolating entire parts of the objects resulting from occlusions or the limited sensor resolution.
Regarding the experiments validating each task, we invite the reviewer to refer to our response entitled “Multi-task learning” in this rebuttal.
[Title] “Q.1-10.
If ﬁnd it misleading to their this task point cloud reconstruction .
A maybe improper analogy is that we do not call joint image denoising and super-resolution and image generation.
If the authors really want to use the term reconstruction, maybe call it conditioned point cloud reconstruction ?
Or something else that is more accurate like point cloud reﬁnement ?”
We are pleased for the reviewer’s recommendation.
We also think of this naming issue before the submission.
Our conclusion is that we put more weights on the quality of 3D reconstruction using 3D points, so called point cloud reconstruction.
As addressed by the reviewer dx4y , this task can be similar to surface reconstruction task because of the common goals, high-quality 3D description.
However, without point triangulation or mesh formulation, this task truely focuses on the point itself to describe 3D environments.
We hope that the reviewer is satisﬁed with our opinion.
[Missing details] “Q.1-11.
Where are the cluster information used in the later framework?
If I understand correctly, this used because the voxel-to-point step only operates on center voxels and hence we want a locally augmented feature representation?”
To be more informative, we revise the paragraphs in Sec. 3.2.
Let us describe the detailed process in the point re-localization network, which is illustrated in Fig.
Given output voxels V out , we collect the K (=8) closest voxels to each voxel v i ∈ R 1 × 3 using the hash table that we used in the 1 st stage network.
Then, we obtain a voxel set V i
= { v k } Kk =1 that is referenced to the query voxel v i .
Similarly, we cluster a set of voxel features F i = { f k } Kk =1 that correspond to P i .
As in Fig. 4, the 2 nd stage network regresses the location of a query point p
i (red voxel) using the geometric relation between the query voxel v i and its neighbor V i (white voxels) through self/cross attention layers.
While this formulation is close to PointTransformer (Zhao et al., 2021), our point re-localization network has three distinctive differences.
First, PointTransformer utilizes k -Nearest Neighbors to group 3D points, we re-use the voxel hashing that is computed in the previous stage.
Second, our network consists of both self-attention and cross-attention.
In our analysis, it is better to use two different attention layers than to solely adopt self-attention.
We are sincerely thankful regarding this comment on the editorial quality of the paper.
In Fig.of the manuscript, to faithfully highlight the ability of our network, we intentionally set the same point’s size for the visualization of the input and output point clouds.
This clearly emphasizes how precisely and densely our network reconstructs the given sparse points.
To clarify this point, we include more details about visualization in Fig.
2 of the manuscript.
[highlight] “Q.1-13.
In Tab.1, maybe the light color can be used to highlight the 2nd best among the baseline method.
Highlighting the 2nd best among all including **ours** cannot fully demonstrate the advantages.”
Following the request by the reviewer, we remove the 2 nd best highlight in Table 1 of the manuscript.
D. R EVIEWER :1M ES
[More qualitative results] “Q2-1.
Qualitative examples are good and would like to see more of them in the supplemental material.”
We newly include qualitative results in the appendix.
Also, we update the results from sequential uses of three baselines in Fig.
5 of the manuscript.
[Ampliﬁed positional encoding] “Q2-2.
The initial section of the paper until sec. 3 . 1 is clear and well explained, while section 3.2 needs a better explanation.
I still do not understand the exact intuition behind the amplitude-based positional embedding.”
We newly revise section 3 . 2 of the manuscript following the reviewer’s comment.
The intuition behind our amplitude positional encoding is clear, when to embed high frequency signal?
We are inspired by NeRF (Mildenhall et al., 2020) where positional encoding results in preserving the high-frequency image signal.
In our case, we are in a different situation.
high-frequency signal is not always welcomed.
When a voxel needs to move to a distant location, it is better to keep the low frequency signal, which is simply conducted by decreasing the amplitude of the positional encoding vector as in Fig.
4 of the manuscript.
Such control is possible since we already know the relative distance between voxels that are estimated from the 1 st stage network.
As underlined by the reviewer G4Aq, we believe that our ampliﬁed encoding is an important contribution of this work.
To better highlight the necessity of this component, we further elaborate the paragraphs about our ampliﬁed positional encoding in Sec. 3. of the manuscript.
[Writing] “Q2-3.
The baselines are not explained well, the exact characteristics of the SnowﬂakePC → Score-PD are not explained, likewise for Dis-PU.
These sections need rewriting.”
As requested by the reviewer, we revised Sec.
4.2 and Sec.
2 of the manuscript.
Also, we included additional experiments about ensembles of three related papers in Table 1 of the manuscript and Table 3 in the appendix.
Let us present the revised paragraphs in Sec.
4.2 below.
In particular, we analyze the efﬁcacy of ours compared to the combination of the existing studies.
For instance in Table 1, PC( r =4 ) → PU( r =4 ) → PD represents consecutive operation of point completion, and point upsampling and point denoising under upsampling ratio r =4 .
Note that we also take into account the point completion task since this task also considers point densiﬁcation on visible surface.
[Details] “Q2-4.
There is no evaluation independently for Stage-1 all evaluations.
I understand that maybe doing the full evaluation on Stage-1 is not possible.”
The evaluation for our 1 st network is already included in Table 2-(b) of the manuscript.
For clarity, we denote the name as 1 st stage.
[Details] “Q2-5.
The paper claims like past work when they upsample they do not have this scalingup factor of r
but I think in this case, r just manifests as the voxel density in stage 1.”
It is true that the generative sparse transposed convolution layer densiﬁes voxels in a ﬁxed ratio r .
Following the reviewer’s comment, we revise the sentences in Sec.
3.1 of the manuscript.
[Future direction] “Q2-6.
I am also not clear what happens when multiple voxel centers move to the same location in 3D points.”
We are pleased to answer this very well spotted observation.
This is, indeed, a limitation of our architectural design.
Speciﬁcally, when multiple points are closer than the voxel size, they are merged into a single voxel during the voxelization process.
Although this was not clearly visible in our experiments, we agree that the raised concerns can harm the results.
In the future, we think of a new network design that utilizes both voxel and point-level understanding to overcome such limitations.
D. R EVIEWER :G4A Q
[Generalization] “Q3-1.
It would be better if the paper could explain why the proposed new method can achieve such strong generalization ability, and why the generalization ability of the existing methods is poor.”
It is our pleasure for the reviewer to recognize the strength of our method, generalization.
We conjecture that this transferability can be explained by two dominant reasons.
First, we optimize the nature of the objective function.
Indeed improving the proximity-to-surface instead of task-speciﬁc losses (such as denoising, densiﬁcation, and completion) seems to be a more generic and generalizable problem.
Regardless of the difference in synthetic and real-world noises, our network is trained to improve the quality of reconstruction.
Also, this is clearly different from the point completion task (Yuan et al., 2018) that concentrates on category-speciﬁc datasets for generating points in occluded regions of objects.
Second, we utilize the hash table for sparse convolution layer instead of k Nearest Neighbor.
To handle noisy and sparse point cloud, k NN is not a proper choice since it is sensitive to the density of point clouds (Mao et al., 2019).
Nonetheless, previous studies (Li et al., 2021;
Xiang et al., 2021;
Yu et al., 2018b) mainly rely on this clustering algorithm.
In contrast, sparse convolution layer takes absolute scale of receptive ﬁeld regardless of the input distribution, which results in better transferability toward unmet and real-world scenes.
These are our analyses of our strength in better generalization and transferability.
With our sparse stacked hourglass network, we can achieve well-generalized reconstruction performance.
These voxelized results are further reﬁned by the 2 nd stage network that converts discrete voxels into 3D points.
Finally, we revised Sec.
4.4 of the manuscript to clarify the discussion about generalization and transferability of our deep point cloud reconstruction network.
D. R EVIEWER : DX 4 Y
[Point cloud reconstruction] “Q4-1.
Since many scans are combined into a single scan, there is another challenge that is entirely overlooked, misalignment.
If the idea is to combine all challenges into one, this should another challenge to address.”
Point cloud registration is another topic that is widely addressed in previous studies (Choy et al., 2020;
Lu et al., 2019; Gojcic et al., 2020).
However, in our opinion, point cloud registration (misalignment) is not an inherent problem of point clouds.
Misalignment usually derives from the extrinsic parameters between multiple 3D scans.
In our point of view, this problem happens when we fuse multiple 3D scans that are taken in different views or time stamps.
We think that it will be a wonderful future direction that extends the applicability of point cloud reconstruction toward multiple 3D scans.
[Related works] “Q4-2.
Missing related work.
There is a very large body of recent works on neural shape implicit representation which focus on the surface reconstruction task.
..., one could simply sample points on the zero level set and achieve point cloud reconstruction.”
We apologize for the missing references about surface reconstruction.
Following the reviewer’s advice, we included a paragraph in Sec. 2.
However, we claim that the point cloud reconstruction task is clearly different from surface reconstruction.
Let us brieﬂy describe their difference.
First, mesh triangulation is not related to the inherent shortcomings of point cloud.
According to the recent papers, PointTriNet (Sharp & Ovsjanikov, 2020) and (Rakotosaona et al., 2021), conventional studies about point triangulation often require dense and uniform 3D points.
It implies that the surface reconstruction task puts more weight on mesh generation from given 3D points.
Second, according to the DeepSDF (Park et al., 2019), this paper also utilizes ShapeNet dataset (Chang et al., 2015), but relies on a large number of points for training, which is described as “To prepare data, we start by normalizing each mesh to a unit sphere and sampling 500,000 spatial points x’s: we sample more aggressively near the surface of the object as we want to capture a more detailed SDF near the surface” .
Therefore, while being different, we believe that our research can be very complementary with such type of work.
For instance, point cloud reconstruction can be applied as a pre-processing to provide a dense and well distributed point cloud to train DeepSDF.
Third, point sampling from mesh cannot be ahead of point cloud reconstruction task.
In our case, we utilize Poisson Disk Sampling
(Yuksel, 2015;
Zhou et al., 2018) to obtain ground truth point cloud from ground truth mesh provided in the ShapeNet dataset (Chang et al., 2015; Yi et al., 2016).
Based on these three reasons, we claim that point cloud reconstruction has clear characteristics compared to the surface reconstruction task.
Moreover, to obtain such a large amount of points to train recent implicit methods (Park et al., 2019), point cloud reconstruction should be taken as a pre-processing stage, ahead of surface reconstruction.
We hope the reviewer is satisﬁed with the necessity of point cloud reconstruction task.
[Details] “Q4-3.
It was not clear to me how the ﬁnal number of points is speciﬁed.
Also, not clear how color information is determined for the newly generated points.”
As we described in Sec. 3.1, our network jointly generates and prunes voxels in an adaptive manner.
Even though transposed generative convolution layer upsamples voxels in a ﬁxed ratio of r , voxel pruning layer removes outliers based on their signals.
Thus, we cannot strictly say the number of outputs and report the average of P out in Table 1.
[Evaluation] “Q4-4.
I ﬁnd the choices of the baseline questionable.
First, there was no uniﬁed approach used (denoising + completion + sampling).
Second, It would make more sense to me compare the proposed method on the other’s task since it is expected that a denoising method will not do well for upsampling.
Third, in the combined baselines (Snowﬂake-PC → Score-PD and Dis-PU → score-PD), it is not clear the order of application (it makes sense to ﬁrst denoise and upsample/complete but the text suggest the opposite which should be reported).
Finally, as mentioned in the related work, the paper lacks comparison to implicit representations surface reconstruction.”
Following the request of the reviewers, we conduct additional experiments about the combination of the three models, denoising, completion and upsampling.
It turns out that the different orders of three models do not improve the quality of point reconstruction compared to the results in Table of the manuscript.
We claim that these results consistently support the necessity of a joint solution for point cloud reconstruction.
We think that our point reconstruction technique cannot truly be considered as a multi-task learning pipeline since it solves the problems in a completely joined manner.
As a result, we cannot isolate each individual task since our network does not have dedicated to design task-speciﬁc header networks nor task-oriented losses.
Instead, we assume that these problems are inherent to point cloud obtained by scanning devices and should be resolved together.
Moreover, even if we forcefully obtain the results in each single task, it will lack fairness and would not reﬂect our intention nor the capability of our point reconstruction technique.
We hope to have clariﬁed this particular point and that the reviewer will understand the context for which we have designed this point reconstruction strategy combining three tasks at once to resolve the inherent shortcomings of 3D point clouds.
[Novelty] “Q4-5.
This paper is missing an important subsection that provides some insight into what the different modules have learned and what is that beneﬁcial to the task.
This raises some questions about novelty - how is this approach better than the sum of its technical parts?”
As noted by three reviewers (veKS, 1Mes, and G4Aq), our contributions derive from a two-stage pipeline that aims to handle raw point clouds via voxelization.
This module is very simple and straightforward, but as we describe in Sec. 3.1, each layer are deeply related to the point cloud reconstruction task.
This architectural design leads to better generalization and transferability toward unmet and real-world 3D scans as in Table 4.
Let us brieﬂy explain the reasons behind our strength of generalization and transferability.
Throughout this paper, we conjecture that this transferability can be explained by two dominant reasons.
First, we optimize the nature of the objective function.
Indeed improving the proximity-to-surface instead of task-speciﬁc losses (such as denoising, densiﬁcation, and completion) seems to be a more generic and generalizable problem.
Regardless of the difference in synthetic and real-world noises, our network is trained to improve the quality of reconstruction.
Also, this is clearly different from the point completion task (Yuan et al., 2018) that concentrates on category-speciﬁc datasets for generating points in occluded regions of objects.
Second
[Table]
we utilize the hash table for sparse convolution layer instead of k Nearest Neighbor.
To handle noisy and sparse point cloud
[Table]
k NN is not a proper choice since it is sensitive to the density
[Table]
of point clouds (Mao et al., 2019).
Nonetheless, previous studies (Li et al., 2021;
Xiang et al., 2021;
Yu et al., 2018b) mainly rely on this clustering algorithm.
In contrast, sparse convolution layer takes absolute scale of receptive ﬁeld regardless of the input distribution, which results in better transferability toward unmet and real-world scenes.
Nonetheless, we ﬁnd that we need to further provide a reasonable analysis of our strength in generalization ability and transferability.
Following the advice from the reviewer dx4y, we elaborate the strength of our method in Sec.
4.4 of the manuscript.
[Writing] “Q4-6.
In its current form, the paper is not self-contained and the method is not reproducible due to a lack of technical details”
Following the request from the reviewer, we newly update Fig.
3 and Fig.
Every detail of channel dimensions and connectivity between layers is precisely described.
Moreover, we are planning to release the code after the review.
Since this is a pioneering paper that opens a new task, as a researcher, we think it is our responsibility to clearly open everything for the community.
[Minor-Writing] “Q4-7.
There is an issue with the term ”continuous 3D points” which repeats itself in the paper.
Point clouds are not continuous, they are sampled on a continuous surface.”
We apologize for the confusing terminology.
We revise all these words in the manuscript.
[Minor-Metrics] “Q4-8.
The evaluation metrics were not properly speciﬁed.
When performing standardized experiments it may be sometimes acceptable to commit this, however, in these cases, the paper proposes a new task, therefore the evaluation metrics should be speciﬁed.”
Following the request by the reviewer, we describe the precise equations of all metrics that we used in the appendix.
[Minor -Metrics] “Q4-9.
V mid is missing.
We apologize for the missing details.
Following the reviewer’s request, we newly update the Fig.of the manuscript.
Note that V mid is the intermediate results from each hourglass network in the 1 st stage network.