PatchComplete: Learning Multi-Resolution Patch Priors for 3D Shape Completion on Unseen Categories
Abstract
While 3D shape representations enable powerful reasoning in many visual and perception applications, learning 3D shape priors tends to be constrained to the specific categories trained on, leading to an inefficient learning process, particularly for general applications with unseen categories.
Thus, we propose PatchComplete , which learns effective shape priors based on multi-resolution local patches, which are often more general than full shapes (e.g., chairs and tables often both share legs) and thus enable geometric reasoning about unseen class categories.
To learn these shared substructures, we learn multi-resolution patch priors across all train categories, which are then associated to input partial shape observations by attention across the patch priors, and finally decoded into a complete shape reconstruction.
Such patch-based priors avoid overfitting to specific train categories and enable reconstruction on entirely unseen categories at test time.
We demonstrate the effectiveness of our approach on synthetic ShapeNet data as well as challenging real-scanned objects from ScanNet, which include noise and clutter, improving over state of the art in novel-category shape completion by 19.3% in chamfer distance on ShapeNet, and 9.0% for ScanNet.
Introduction
The prevalence of commodity RGB-D sensors (e g., Intel RealSense, Microsoft Kinect, iPhone, etc.) has enabled significant progress in 3D reconstruction, achieving impressive tracking quality [25, 17, 27, 8, 40, 12] and even large-scale reconstructed datasets [10, 4].
Unfortunately, 3D scanned reconstructions remain limited in geometric quality due to clutter, noise and incompleteness (e g., as seen in the objects in Figure 1).
Understanding complete object structures is fundamental towards constructing effective 3D representations, that can then be used to fuel many applications in robotic perception, mixed reality, content creation and more.
Recently, significant progress has been made in shape reconstruction, across a variety of 3D representations, including voxels [9, 13, 34], points [15, 43], meshes [39, 11, 32], and implicit field representations [28, 21, 14, 18, 31, 3, 16, 33].
However, these methods tend to rely heavily on strong synthetic supervision, producing impressive reconstructions on similar train class categories but struggling to generalize to unseen categories.
This leads to an expensive compute and data requirement in adapting to new objects in different scenarios, whose class categories may not necessarily have been seen during training and so must be re-trained or fine-tuned for.
In order to encourage more generalizable 3D feature learning to represent shape characteristics, we observe that while different class categories may have very different global structures, local geometric structures are often shared (e.g., a long, thin structure could represent a chair leg, a table leg, a lamp rod, etc.).
We thus propose to learn a set of multi-resolution patch-based priors that captures such shared local substructures across the training set of shapes, which can be applied to shapes outside of the train set of categories.
Our local patch-based priors can thus capture shared local structures, across different resolutions, that enable effective shape completion on novel class categories of not only synthetic data but also challenging real-world observations with noise and clutter.
We propose PatchComplete, which first learns patch priors for shape completion by correlating regions of observed partial inputs to the learned patch priors through an attention-based association, and decoding to reconstruct a complete shape.
These patch priors are learned at different resolutions to encompass potentially different sizes of local substructures; we then learn to fuse the multi-resolution priors together to reconstruct the output complete shape.
This enables learning generalizable local 3D priors that facilitate effective shape completion even for unseen categories, outperforming state of the art in synthetic and real-world observations by 19.3% and 9.0% on Chamfer Distance.
In summary, our contributions are:
• We propose generalizable 3D shape priors by learning patch-based priors that characterize shared local substructures that can be associated with input observations by cross-attention.
This intermediate representation preserves structure explicitly, and can be effectively leveraged to compose complete shapes for entirely unseen categories.
• We design a multi-resolution fusion of different patch priors at various resolutions in order to effectively reconstruct a complete shape, enabling multi-resolution reasoning about the most informative learned patch priors to recover both global and local shape structures.
Related Work 2.1 3D Shape Reconstruction and Completion
Understanding how to reconstruct 3D shapes is an essential task for 3D machine perception.
In particular, the task of shape completion to predict a complete shape from partial input observations has been studied by various works toward understanding 3D shape structures.
Recently, many works have leveraged deep learning techniques to learn strong data-driven priors for shape completion, focusing on various different representations, e.g. volumetric grids [41, 13], continuous implicit functions [29, 21, 28, 7, 36], point clouds [30, 33], and meshes [11, 19].
These works tend to focus on strong synthetic supervision on a small set of train categories, achieving impressive performance on unseen shapes from train categories, but often struggling to generalize to unseen classes.
We focus on learning more generalizable, local shape priors in order to effectively reconstruct complete shapes on unseen class categories.
2.2 Few-Shot and Zero-Shot 3D Shape Reconstruction
Several works have been developed to tackle the challenging task of few-shot or zero-shot shape reconstruction, as observations in-the-wild often contain a wide diversity of objects.
In the few-shot scenario where several examples of novel categories are available, Wallace and Hariharan [38] learn to refine a given category prior.
Michalkiewicz et al.
[22] further propose to learn compositional shape priors for single-view reconstruction.
In the zero-shot scenario without any examples seen for novel categories, Naeem et al.
[24] learn priors from seen categories to generate segmentation masks for unseen categories.
Zhang et al.
[44] additionally proposed to use spherical map representations to learn priors for the reconstruction of novel categories.
Thai et al.
[35] recently developed an approach to transfer knowledge from an RGB image for shape reconstruction.
We also tackle a zero-shot shape reconstruction task, by learning a multi-resolution set of strong local shape priors to compose a reconstructed shape.
Several recent works have explored learning shape priors by leveraging a VQ-VAE backbone with autoregressive prediction to perform shape reconstruction [23, 42].
In contrast, we propose to learn multi-resolution shape priors without requiring any sequence interpretation, which enables direct applicability to real-world scan data that often contains noise and clutter.
Additionally, hierarchical reconstruction has shown promising results for shape reconstruction [2, 6].
Our approach also takes a multi-resolution perspective, but learns explicit shape and local patch priors and their correlation to input partial observations for robust shape completion.
Method 3.1 Overview
Our method aims to learn effective 3D shape priors that enable general shape completion from partial input scan data, on novel class categories not seen during training.
Key to our approach is the observation that 3D shapes often share repeated local patterns – for instance, chairs, tables, and nightstands all share a support surface, and chair or table legs can share a similar structure with lamp rods.
Inspired by this, we regard a complete object as a set of substructures, where each substructure represents a local geometric region.
We thus propose PatchComplete to learn such local priors and assemble them into a complete shape from a partial scan.
An overview of our approach is shown in
Figure 2.
We learn to build mappings from local regions in a partial object scan S to local priors based on complete learnable shape priors {T c } .
An input encoder takes the input of S to encode its local features (represented by patches Q ).
Analogously, we adopt a prior encoder to process each prior in {T c } to construct a local prior feature pool {K c } .
In parallel, we chunk priors in {T c } into patch volumes {V c } , which are correspondingly fused by the attention to input patches to compose the most informative patch priors: we use each incomplete local patch in Q to query the keys {K c } from complete prior patches, and assemble their corresponding patch volumes {V c } for shape completion.
3.2 Learning Local Patch-Based Shape Priors
We first learn local shape priors from ground-truth train objects.
We represent both the input partial scan S and the ground-truth shape G as 3D truncated signed distance field grids of size D 3 .
Figureillustrates the process of learning local shape priors.
We learn to build mappings from local regions in incomplete input observations to local priors based on complete shapes, in order to robustly reconstruct the complete shape output.
We first aim to learn patch-based priors, which we extract from learnable global shape priors {T c } .
We denote G c as the set of ground-truth train objects in the c -th category.
These {T c } priors are initialized per train category c based on mean-shift clustering within shapes in G c .
Thus T c is a set of representative samples in each category, which are encoded parallel to the input scan S .
Both encoders are analogously structured as 3D convolutional encoders which spatially downsample by a factor of R , resulting in a 3D encoding of size N 3 ; N = D/R .
The input encoding (from S ) are uniformly chunked into patches {Q i } , i =1,..., N 3 ; similarly, the encoded priors (from {T c } ) are chunked into patches {K ci } .
We then use each local part Q i to query for the most informative encodings of complete local regions {K c i } , building this mapping based on cross-attention [37].
Then for each input patch, we calculate its similarity with all local patches in representative shapes on training categories:
[Equation]
where d is the dimension of the encoded vectors of Q i , K ci ; C is the category number.
We then reconstruct complete shape patches {P i } by:
[Equation]
where V =
{V c i } is the set of N 3 chunks (of resolution R ) from shapes in all categories {T c } , where each V c i is paired with K c i ∈ K .
We can then recompose {P i } to the predicted full shape P .
Loss.
We use an ℓ 1 reconstruction loss to train the learned local patch priors.
Note that {T c } are learned along with the network weights, enabling learning the most effective global shape priors for the shape completion task.
R } R } learning model ( M R ) in Section 2 under different resolutions ( R =32, 8, 4)
Then, we use attention (Equation 3) to generate intermediate features {O R } , which we then recursively fuse to decode a complete shape
3.3 Multi-Resolution Patch Learning
In Section 3.2, we learn to complete shapes in a single patch resolution R .
Since local substructures may exist at various scales, we learn patch priors under varying resolutions, which we then use to decode a complete shape.
We use three patch resolutions ( R =4, 8, 32), for which we learn patch priors.
This results in three pairs of trained {input encoder, prior encoder} (see Figure 3).
Given a partial input scan S , each pair outputs a set of 1) input patch features {Q Ri } , and 2) prior patch features K R , under the resolution of R =4, 8, 32.
In this section, we decode a complete shape from these multi-resolution patch priors.
Since K R stores the patch priors, we use it as the (key, value) term into an attention module, where each input patch feature Q Ri is used to query for the most informative prior features in K R under different resolutions, from which we complete partial patches in feature spaces with a multi-scale fashion.
We formulate this process by
[Equation]
where we concatenate input patch feature with the attention result to compensate information loss on observable regions.
This outputs {O Ri } as the intermediate feature of the i -th patch; i = 1 , ..., N 3 R .
N R denotes the number of patches under the resolution R ; N R = D/R .
We recompose all generated patch features into a volume O R =
{O Ri | i = 1 , ..., N 3 R } .
Note that the dimension of each grid feature in {Q Ri } and K R equals to d (see Eq. 1).
Then O R is with the dimension of N 3 R × 2 d .
We then sequentially use 3D convolution decoders to upsample and concatenate O R from low to high resolution to fuse all the shape features as in
[Equation]
In Eq.
4, O R r has a lower resolution than O R r
+1 .
We then use deconvolution layers to upsample
O R r to match the resolution of O R r +1 , and then concatenate them together.
We recursively fuse O R r into O R r +1 , which produces O R 3 as the final fusion result with the resolution of ( D/ 4) 3 .
An extra upsampling followed by a convolution layer is adopted to upsample O R 3 into our shape prediction
O pred with the dimension of D 3 .
In training the feature fusion, we fix all parameters from the {input encoder, prior encoder} under the three resolutions, since they are pre-trained on Section 3.2 and provide better-learned priors and attention maps under these strong constraints.
The whole pipeline for this section can be found in
Figure 4.
We use an ℓ 1 loss to supervise the TSDF value regression.
We weight the ℓ 1 loss to penalize false predictions based on the predicted signs as Eq.
5 shows, which represents whether this voxel grid is occupied or not.
It is also used in the loss function in Section 3.2.
[Equation]
In Eq.
5, O emptypred represents the false positive TSDF values, where the ground truth has negative signs and the prediction has positive signs, which in general indicates the missing predictions.
O occpred represents false negative TSDF values, where the ground truth has positive signs and the prediction has negative signs, which indicates extra predictions.
O correctpred means those predicted TSDF values with the same signs as the ground-truth.
During training, we choose the weight for false positive ( w empty ) as 5, and the weight for false negative ( w occ ) as 3.
3.4 Implementation Details
We train our approach on a single NVIDIA A6000 GPU, using an Adam optimizer with batch sizefor a synthetic dataset and batch size 16 for a real-world dataset, and an initial learning rate of 0.001.
We train for 80 epochs until convergence, and then decrease the learning rate by half after epoch 50.
We use the same settings for learning priors and the multi-resolution decoding, which train for 4 and 15.5 hours respectively.
For additional network architecture details, we refer to the supplemental material.
Note that for training for real scan data, we first pre-train on synthetic data and then fine-tune only the input encoder.
We use 112 shape priors in our method, which are clustered from 3202 train shapes, and represented as TSDFs with truncation of 2.5 voxels.
Experiments and Analysis 4.1 Experimental Setup
Datasets.
We train and evaluate our approach on synthetic shape data from ShapeNet [5] as well as on challenging real-world scan data from ScanNet [10].
For ShapeNet data, we virtually scan the objects to create partial input scans, following [26, 13].
We use 18 categories during training, and test on 8 novel categories, resulting in 3,202/1,325 train/test models with 4 partial scans for each model.
For real data, we use real-scanned objects from ScanNet extracted by their bounding boxes, with corresponding complete target data given by Scan2CAD [1].
We use 8 categories for training, comprising 7,537 train samples, and test on 6 novel categories of 1,191 test samples.
For all experiments, objects are represented as 32 3 signed distance grids with truncation of 2.5 voxel units for ShapeNet and 3 voxel units for ScanNet.
The objects in ShapeNet are normalized into the unit cube, while we keep the scaling for ScanNet objects, and save their voxel sizes separately to keep the real size information.
Additionally, to train and evaluate on real data, all methods are first pre-trained on ShapeNet and then fine-tuned on ScanNet.
Baselines.
We evaluate our approach against various state-of-the-art shape completion methods.
We compare with state-of-the-art shape completion methods 3D-EPN [13] and IF-Nets [7], which learn effective shape completion on dense voxel grids and with implicit neural field representations, respectively, without any focus on unseen class categories.
We further compare to the state-of-the-art few-shot shape reconstruction approach of Wallace and Hariharan [38] (referred to as Few-Shot) leveraging global shape priors, which we apply in our zero-shot unseen category scenario.
Finally,
AutoSDF [23] uses a VQ-VAE module with a transformer-based autoregressive model over latent patch priors to produce TSDF shape reconstruction.
Evaluation Metrics.
To evaluate the quality of reconstructed shape geometry, we use L 1 Chamfer
Distance (CD) and Intersection over Union (IoU) between predicted and ground truth shapes.
To evaluate methods that output occupancy grids, we use occupancy thresholds used by the respective methods to obtain voxel predictions, i.e. 0.4 for [38] and 0.5 for [7].
To evaluate methods that output signed distance fields, we extract the iso-surface at level zero with marching cubes [20].
10K points are sampled on surfaces for CD calculation.
Both Chamfer distance and IoU are evaluated on objects in the canonical system, and we report Chamfer Distance scaled by × 10 2 .
[Table]
4.2 Evaluation on Synthetic Data
In Table [Table] we evaluate our approach in comparison with prior arts on unseen class categories of synthetic ShapeNet [5] data.
Our approach on learning attention-based correlation to learned local shape priors results in notably improved reconstruction performance, with coherent global and local structures, as shown in Figure 5.
In Table 1, our work outperforms other baselines both instance-wise and category-wise.
One of the key factors is that our method learns multi-scale patch information from seen categories to complete unseen categories with enough flexibility, while most of the other baselines are designed for 3D shape completion on known categories, which hardly leverage shape priors across categories.
4.3 Evaluation on Real Scan Data
Table 2 evaluates our approach in comparison with prior arts on real scanned objects from unseen categories in ScanNet [10].
Here, input scans are not only partial but often contain noise and clutter; our multi-resolution learned priors enable more robust shape completion in this challenging scenario.
Results in Figure 6 further demonstrate that our approach presents more coherent shape completion than prior methods by using cross-attention with learnable priors, which better preserves the global structures in coarse and cluttered environments. [Table]
[Table]
4.4 Ablation Analysis
Does multi-resolution patch learning help shape completion for novel categories?
In Table 3, we evaluate shape completion with each resolution in comparison with our multi-resolution approach.
Learning only global shape priors (i.e., 32 3 ) tends to overfit to seen train categories, while the local patch resolutions can provide more generalizable priors.
Combining all results in complementary feature learning for the most effective shape completion results.
Does cross-attention to learn local priors help?
We evaluate our approach to learn both local priors and their correlation to input observations with cross-attention in Table 4, which shows that this enables more effective shape completion on unseen categories. Here, the no attention experiment replaces the attention score calculation in Eq. 1 by using MLPs (on concatenated input/prior features) to predict weights for each input-prior pair
[Table]
[Table]
[Table]
What is the effect of synthetic pre-training for real scan completion?
Table 5 shows the effect of synthetic pre-training for shape completion on real scanned objects.
This encourages learning more robust priors to output cleaner local structures as given in the synthetic data, resulting in improved performance on real scanned objects.
[Table]
Does learning the priors help completion?
In Table 6, we evaluate our learnable priors in comparison with using fixed priors (by mean-shift clustering of train objects) for shape completion on
ShapeNet [5].
Learned priors receive gradient information to adapt to best reconstruct the shapes, enabling improved performance over a fixed set of priors.
What is the effect of the train category split?
Our novel category split was designed based on the number of objects in each category, to mimic real-world scenarios where object categories with larger numbers of observations are used for training.
[Table] To show that our approach is independent of the category splitting strategy, we add two new settings for evaluation on ShapeNet [5]. We use the same overall categories (26) as originally, then randomly shuffle them for train/novel categories. Table 7 shows that all the splits achieve similar results, which indicates that our approach is robust across different splits.
It is because the learned patch priors share universal partial information among categories, which reduces the dependency on train categories for shape completion tasks.
Figure 7 shows qualitative results for novel category samples in the alternative splits.
4.5 Limitations
While PatchComplete has presented a promising step towards learning more generalizable shape priors, various limitations remain.
For instance, output shape completion is limited by the dense voxel grid resolution in representing fine-scale geometric details.
Additionally, detected object bounding boxes are required for real scan data as input to independent shape completion predictions, while formulation considering other objects in the scene or an end-to-end framework could learn more effectively from global scene contextual information.
4.6 Societal Impact
This work proposes a method for 3D completion for unseen categories with less dependency on data capturing, with potential application to robotics or content creation scenarios.
For instance, a robot could potentially more efficiently grasp unseen objects when estimating complete shape structure; or self-driving cars could perform obstacle avoidance by robust 3D shape estimation for a variety of objects.
While potential real-world applications are promising, this requires careful consideration of personal data privacy, as well as any biases embedded in the training dataset.
Conclusion
We have proposed PatchComplete to learn effective local shape priors for shape completion that enables robust reconstruction for novel class categories at test time.
This enables learning shared local substructures across a variety of shapes that can be mapped to local incomplete observations by cross attention, with multi-resolution fusion producing coherent geometry at global and local scales.
This enables robust shape completion even for unseen categories with different global structures, across synthetic as well as challenging real-world scanned objects with noise and clutter.
We believe that such robust reconstruction of real scanned objects takes an important step towards understanding 3D shapes, and hope this inspires future work in understanding real-world shapes and scene structures. Acknowledgements
This project is funded by the Bavarian State Ministry of Science and the Arts and coordinated by the
Bavarian Research Institute for Digital Transformation (bidt).
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes] , [No] , or [N/A] . You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example: • Did you include the license to the code and datasets? [Yes] See Section XXX. • Did you include the license to the code and datasets?
[No] The code and the data are proprietary. • Did you include the license to the code and datasets? [N/A] Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below. For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See abstract, Section 1 and Section(b)
Did you describe the limitations of your work?
[Yes] Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Section 4.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the link in abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 3.4, Section 4.1, and the Data Generation section of the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 1, Table 2, and Table 3, more details can be found in the Category-wise Evaluations with Error Bars section of the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 3.4. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We use and cite ShapeNet [5] and ScanNet [10].
(b) Did you mention the license of the assets?
[Yes] See the Data Generation section of the supplemental material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See the Data Generation section of the supplemental material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See the Data Generation section of the supplemental material.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]