CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds
Abstract
We present a novel two-stage fully sparse convolutional 3D object detection framework, named CAGroup3D. Our proposed method first generates some high-quality 3D proposals by leveraging the class-aware local group strategy on the object surface voxels with the same semantic predictions, which considers semantic consistency and diverse locality abandoned in previous bottom-up approaches.
Then, to recover the features of missed voxels due to incorrect voxel-wise segmentation, we build a fully sparse convolutional RoI pooling module to directly aggregate fine-grained spatial information from backbone for further proposal refinement.
It is memory-and-computation efficient and can better encode the geometry-specific features of each 3D proposal.
Our model achieves state-of-theart 3D detection performance with remarkable gains of + 3.6% on ScanNet V and + 2.6 % on SUN RGB-D in term of mAP@0.25.
Code will be available at https://github.com/Haiyang-W/CAGroup3D .
Introduction
As a crucial step towards understanding 3D visual world, 3D object detection aims to estimate the oriented 3D bounding boxes and semantic labels of objects in real 3D scenes.
It has been studied for a long time in both academia and industry since it benefits various downstream applications, such as autonomous driving [2, 36], robotics [55, 37] and augmented reality [1, 3].
In this paper, we focus on detecting 3D objects from unordered, sparse and irregular point clouds.
Those natural characteristics make it more challenging to directly extend well-studied 2D techniques to 3D detection.
Unlike 3D object detection from autonomous driving scenarios that only considers bird’s eye view (BEV) boxes [30, 50, 19, 11, 32, 31], most of existing 3D indoor object detectors [27, 38, 22, 5, 52] typically handle this task through a bottom-up scheme, which extracts the point-wise features from input point clouds, and then groups the points into their respective instances to generate a set of proposals.
However, the above grouping algorithms are usually carried out in a class-agnostic manner, which abandons semantic consistency within the same group and also ignores diverse locality among different categories.
For example, VoteNet [27] learns the point-wise center offsets and aggregates the points that vote to similar semantic-irrelevant local region.
Though impressive, as shown in Figure 1, these methods may fail in cluttered indoor scenes where various objects are close but belong to different categories.
Also, the object sizes are diverse for different categories, so that a class-agnostic
[Figure]
Hence, we propose CAGroup3D, a two-stage fully convolutional 3D object detection framework.
Our method consists of two novel components.
One is the class-aware 3D proposal generation module, which aims to generate reliable proposals by utilizing class-specific local group strategy on the object surface voxels with same semantic predictions.
The other one is an efficient fully sparse convolutional RoI pooling module for recovering the features of the missed surface voxels due to semantic segmentation errors, so as to improve the quality of predicted boxes.
Specifically, a backbone network with 3D sparse convolution is firstly utilized to extract descriptive voxel-wise features from raw point clouds.
Based on the learned features, we conduct a class-aware local grouping module to cluster surface voxels into their corresponding instance centroids.
Different from [27], in order to consider the semantic consistency, we not only shift voxels of the same instance towards the same centroid but also predict per-voxel semantic scores.
Given the contiguously distributed vote points with their semantic predictions, we initially voxelize them according to the predicted semantic categories and vote coordinates, so as to generate class-specific 3D voxels for different categories.
The voxel size of each category is adaptive to its average spatial dimension.
To maintain the structure of fully convolution, we apply sparse convolution as grouping operation centered on each voted voxel to aggregate adjacent voxel features in the same semantic space.
Note that these grouping layers are class-dependent but share the same kernel size, thus the larger classes are preferred to be aggregated with larger local regions.
Secondly, given the proposal candidates, fine-grained specific features within 3D proposals need to be revisited from 3D backbone through certain pooling operation for the following box refinement.
However, state-of-the-art pooling strategies [31, 8] are memory-and-computation intensive due to the hand-crafted set abstraction [25].
Besides that, its max-pooling operation also harms the geometry distribution.
To tackle this problem, we propose RoI-Conv pooling module, which directly adopts the well-optimized 3D sparse convolutions to aggregate voxel features from backbone.
It can encode effective geometric representations with a memory-efficient design for further proposal refinement.
In summary, our contributions are three-fold: 1)
We propose a novel class-aware 3D proposal generation strategy, which considers both the voxel-wise semantic consistency within the same local group and the object-level shape diversity among different categories.
We present RoI-Conv pooling module, an efficient fully convolutional 3D pooling operation for revisiting voxel features directly from backbone to refine 3D proposals.
Our approach outperforms state-of-the-art methods with remarkable gains on two challenging indoor datasets, i.e. , ScanNet V2 [6] and SUN RGB-D [33], demonstrating its effectiveness and generality.
Related Work
3D Object Detection on Point Clouds.
Detecting 3D objects from point clouds is challenging due to orderless, sparse and irregular characteristics.
Previous approaches can be coarsely classified into two lines in terms of point representations, i.e. , the voxel-based methods [54, 46, 32, 45, 31, 50, 32, 9, 40] and the point-based methods [27, 38, 5, 22, 52, 47].
Voxel-based methods are mainly applied in outdoor autonomous driving scenarios where objects are distributed on the large-scale 2D ground plane.
They process the sparse point clouds by efficient 3D sparse convolution, then project these 3D volumes to 2D grids for detecting bird’s eye view (BEV) bboxes by 2D ConvNet.
Powered by
PointNet series [25, 29], point-based methods are also widely used to predict 3D bounding bboxes.
Most of existing methods are in a bottom-up manner, which extracts the point-wise features and groups them to obtain object features.
This pipeline has been a great success for estimating 3D bboxes directly from cluttered and dense 3D scenes.
However, due to the hand-crafted point sampling and computation intensive grouping scheme applied in PointNet++ [29], they are difficult to be extended to large-scale point clouds.
Hence, we propose an efficient fully convolutional bottom-up framework to efficiently detect 3D bboxes directly from dense 3D point clouds.
Feature Grouping.
Feature grouping is a crucial step for bottom-up 3D object detectors [27, 38, 22, 5, 52, 35], which clusters a group of point-wise features to generate high-quality 3D bounding boxes.
Among the numerous successors, voting-based framework [27] is widely used, which groups the points that vote to the same local region.
Though impressive, it doesn’t consider the semantic consistency, so that may fail in cluttered indoor scenes where the objects of different classes are distributed closely.
Moreover, voting-based methods usually adopt a class-agonistic local region for all objects, which may incorrectly group the boundary points of large objects and involve more noise points for small objects.
To address the above limitations, we present a class-aware local grouping strategy to aggregate the points of the same category with class-specific center regions.
Two-stage 3D Object Detection.
Many state-of-the-art methods considered applying RCNN style 2D detectors to the 3D scenes, which apply 3D RoI-pooling scheme or its variants [30, 32, 8, 31, 48, 44] to aggregate the specific features within 3D proposals for the box refinement in a second stage.
These pooling algorithms are usually equipped with set abstraction [25] to encode local spatial features, which consists of a hand-crafted query operation ( e.g. , ball query [25] or vector query [8]) to capture the local points and a max-pooling operation to group the assigned features.
Therefore these RoI pooling modules are mostly computation expensive.
Moreover, the max-pooling operation also harms the spatial distribution information.
To tackle these problems, we propose RoI-Conv pooling, a memory-and-computation efficient fully convolutional RoI pooling operation to aggregate the specific features for the following refinement.
Methodology
In this paper, we propose CAGroup3D, a two-stage fully convolutional 3D object detection framework for estimating accurate 3D bounding boxes from point clouds.
The overall architecture of CAGroup3D is depicted in Figure 2.
Our framework consists of three major components: an efficient 3D voxel
CNN with sparse convolution as the backbone network for point cloud feature learning (§3.1), a class-aware 3D proposal generation module for predicting high quality 3D proposals by aggregating voxel features of the same category within the class-specific local regions (§3.2) and RoI-Conv pooling module for directly extracting complete and fine-grained voxel features from the backbone to revisit the miss-segmented surface voxels and refine 3D proposals.
Finally, we formulate the learning objective of our framework in §3.4.
3.1 3D Voxel CNN for Point Cloud Feature Learning
For generating accurate 3D proposals, we first need to learn discriminative geometric representation for describing input point clouds.
Voxel CNN with 3D sparse convolution [32, 45, 54, 13, 12] is widely used by state-of-the-art 3D detectors thanks to its high efficiency and scalability of converting the point clouds to regular 3D volumes.
In this paper, we adopt sparse convolution based backbone for feature encoding and 3D proposal generation.
3D backbone network equipped with high-resolution feature maps and large receptive fields is critical for accurate 3D bounding box estimation and voxel-wise semantic segmentation.
The latter is closely related to the accuracy of succeeding grouping module.
To maintain these two characteristics, inspired by the success of HRNet series [39, 16, 34] in segmentation community, we implement a 3D voxel bilateral network with dual resolution based on ResNet [15].
For brevity, we refer it as BiResNet.
As shown in Figure 2, our backbone network contains two branches.
One is the sparse modification of
[Figure]
Block2 (1/4)
Block3 (1/8)
Feat (1/2) Feat(1/2)
Input
Voxelize
N x
Block3 (1/16)
[Figure]
Generation
[Figure]
a. Class-Aware 3D Proposal Generation
b. RoI-Conv Point Cloud Feature Pooling and Box Refinement
[Figure] [Figure]
ResNet18 [15] where all 2D convolutions are replaced with 3D sparse convolutions.
It can extract multi-scale contextual information with proper downsampling modules.
The other one is a auxiliary branch that maintains a high-resolution feature map whose resolution is 1/2 of the input 3D voxels.
Specifically, the auxiliary branch is inserted following the first stage of ResNet backbone and doesn’t contain any downsampling operation.
Similar to [39], we adopt the bridge operation between the two paths to perform the bilateral feature fusion.
Finally, the fine-grained voxel-wise geometric features with rich contextual information are generated by the high-resolution branch and facilitate the following module.
Experiments also demonstrate that our voxel backbone performs better than previous FPN-based ResNet [20].
More architecture details are in Appendix.
3.2 Class-Aware 3D Proposal Generation
Given the voxel-wise geometric features generated by the backbone network, a bottom-up grouping algorithm is generally adopted to aggregate object surface voxels into their respective ground truth instances and generate reliable 3D proposals.
Voting-based grouping method [27] has shown great success for 3D object detection, which is performed in a class-agnostic manner.
It reformulates
Hough voting to learn point-wise center offsets, and then generates object candidates by clustering the points that vote to similar center regions.
However, this method may incorrectly group the outliers in the cluttered indoor scenarios ( e.g. , votes are close together but belong to different categories), which degrades the performance of 3D object detection.
Moreover, due to the diverse object sizes of different categories, class-agnostic local regions may mis-group the boundary points of large objects and involve more noise points for small objects.
To address this limitation, we propose the class-aware 3D proposal generation module, which first produces voxel-wise predictions ( e.g. , semantic maps and geometric shifts), and then clusters the object surface voxels of the same semantic predictions with class-specific local groups.
Voxel-wise Semantic and Vote Prediction.
After obtaining the voxel features from backbone network, two branches are constructed to output the voxel-wise semantic scores and center offset vectors.
Specifically, the backbone network generates a number of N non-empty voxels { o i } Ni =1 from backbone, where o i =
[ x i ; f i ] with x i ∈ R 3 and f i ∈ R C .
A voting branch encodes the voxel feature f i to learn the spatial center offset ∆ x i ∈ R 3 and feature offset ∆ f i ∈ R C .
Based on the learned spatial and feature offset, we shift voxel o i to the center of its respective instance and generate vote point p i as follow:
[Equation]
The predicted offset ∆ x i is explicitly supervised by a smooth- ℓ 1 loss with the ground-truth displacement from the coordinate of seed voxel x
i to its corresponding bounding box center.
In parallel with the voting branch, we also construct a semantic branch to output semantic scores
S =
{ s i } Ni =1 for all the voxels over N class classes as
[Equation]
where MLP sem ( · ) is a one-layer multi-layer-perceptron (MLP) network and s i indicates the semantic probability for all classes of voxel o i .
We adopt focal loss [21] for calculating voxel segmentation loss to handle the class imbalance issue.
Notably, the vote and semantic targets of each voxel are associated with the ground-truth 3D boxes not the instance or semantic masks, so that it can be easily generalized to the 3D object detection datasets with bounding box annotations.
To be specific, for each voxel, only the ground-truth bounding boxes that includes this voxel are selected.
Considering the ambiguous cases that a voxel is in multiple ground truth bounding boxes, only the box with the least volume is assigned to this voxel.
Class-Aware Local Grouping.
This step aims to produce reliable 3D proposals in a bottom-up scheme based on the above voxel-wise semantic scores { s i } Ni =1 and vote predictions { p i } Ni =1 .
To carry out grouping with semantic predictions, we first define a score threshold τ for all categories to individually determine whether a voxel belongs to a category instead of utilizing the one-hot semantic predictions.
It can allow the voxel to be associated with multiple classes and thus improve the recall of semantic voxels for each category.
Given the semantic scores S = { s i } Ni =1 , we iterate over all the classes, and slice a point subset from the whole scene of each class that has the score higher than the threshold τ , to form the class-dependent vote set { c
[Equation]
The above semantic subset is generated in a contiguous euclidean space and the vote points are distributed irregularly.
To maintain the structure of pure convolutions and facilitate the succeeding class-aware convolution-based local grouping module, we individually voxelize the vote points in each semantic subset to { V j } N class j = by employing a voxel feature encoding (VFE) layer with a class-specific 3D voxel size, which is proportional to the class average spatial dimension.
Specifically, this re-voxelization process for each class can be formulated as follows:
[Equation]
where c j is the class-dependent vote set of class index j and | V j | is the number of non-empty voxels after class individual re-voxelization.
α · d j is the class-specific voxel size, where α is a predefined scale factor and d j = ( w j , h j , l j ) is the category average spatial dimension.
VFE ( · , α · d j , Avg ) means that the average pooling operation is adopted to voxelize vote features on j -th class subset with the voxel size α · d j .
Importantly, the voxel size is adaptive among different categories, which is more diverse than the widely used FPN-based prediction structure.
Given the predicted vote voxels of j -th class { v i } | V j | i =1 , we apply sparse convolutions with a predefined kernel size k ( a ) on each voxel, and automatically aggregate local context inside the class to generate class-specific geometric features A ( j ) as follow:
[Equation]
where SparseConv ( j ) 3D ( · center , · support voxels , · kernel size ) is the standard sparse 3D convolution [13] and specific for different classes.
A shared anchor-free head is appended to the aggregated features for predicting classification probabilities, bounding box regression parameters and confidence scores.
3.3 RoI-Conv point cloud feature pooling for 3D Proposal Refinement
Due to the semantic segmentation errors in stage-I, class-aware local grouping module will mis-group some object surface voxels.
So an efficient pooling module is needed to recover the missed voxel features and also aggregate more fine-grained features from backbone for proposal refinement.
Stateof-the-art 3D RoI pooling strategies [31, 8] usually adopt set abstraction operation to encode local patterns, which is computation expensive compared to traditional convolution and hand-crafted with lots of hyper-parameters ( e.g. , radius, the number of neighbors).
Moreover, its max-pooling operation also harms the spatial distribution information.
To tackle these limitations and hold a fully convolution structure, we propose RoI-Conv pooling operation, which builds a hierarchical grouping module with well optimized 3D sparse convolutions to directly aggregate RoI-specific features from backbone for further proposal refinement.
Our hierarchical structure is composed by a number of sparse abstraction block, which contains two key components: the RoI-guidence sampling for selecting a subset of input voxels within the 3D proposals, which defines the centroids of local regions, and a shared sparse convolution layer for encoding local patterns into feature vectors.
Sparse Abstraction.
As shown in Figure 2, the inputs to this block are a number of | I | input voxels
I = { l n } I n =1 and | M | proposals M = { ρ m } M m =1 , where ρ m is the proposal parameters for guiding the point sampling.
The output is a number of | Q | pooling RoI-specific voxels Q = { q k } |Q| k =1 .
Specifically, given the input voxels and proposals, instead of directly sampling from the whole input space, we adopt the RoI-guidence sampling to uniformly sample G x × G y × G z grid points within each 3D proposal in voxel space, which are denoted as G = { g k ∈
Z 3 } G x × G y ×
G z ×M k = .
G x × G y ×
G z is the proposal sampling resolution, which are the hyper-parameters independent on proposal sizes.
Considering the overlap of different proposals, we merge the repeated grid points and generate a unique points set (cid:101) G = { g k } | (cid:101)
G| k =1 , where | (cid:101) G | is the number of unique grid points.
Then, with the
RoI-specific points set (cid:101) G , we exploit a sparse convolution centered on each sampled point to cover a set of neighboring input voxels ( e.g. , N k = { l 1 k , l 2 k , ..., l L k k } for g k ) within the kernel size k ( p ) as:
Q = (cid:26) q k | q k = (cid:26) SparseConv 3D ( g k
[Equation]
N k
[Equation]
k ( p
[Equation] [Equation]
if L k >
[Equation]
ϕ
[Equation]
if L k = 0 , for k = 1 , ..., | (cid:101) G | (cid:27) (6) where L k is the number of neighboring voxels queried by the k -th RoI-specific point with kernel size k ( p ) .
SparseConv 3D ( · center , · support voxels , · kernel size ) is a shared sparse 3D convolution for all the proposals and only applied on the points that their neighboring voxel sets are non-empty.
ϕ means empty voxel.
Then, to hold the surface geometry and reduce computation cost, we abandon the empty voxels and output the RoI-specific voxels set Q .
RoI-Conv Pooling Module.
Our pooling network is equipped with two-layers sparse abstraction block and progressively abstracts the voxel features within each 3D proposal from the backbone to
RoI-specific features iteratively as:
[Equation]
where SparseAbs ( · input voxels , · proposals , · sampling resolution , · kernel size ) denotes our sparse abstraction block.
Notably, to encode the voxel features of oriented proposals, we follow the transformation strategy in [32] before the last block, and normalize the input voxels belonging to each proposal to its individual canonical systems.
With the RoI feature F ∈ R C of each proposal, the refinement network predicts the size and location ( i.e. , bbox dimension, center and orientation) residuals relative to the proposal in Stage-I and the targets are encoded by the traditional residual-based method [31, 32].
3.4 Learning Objective
Our proposed approach is trained from scratch with semantic loss L sem , voting loss L vote-reg , centerness loss L cntr , bounding box estimation loss L box , classification losses L cls for Stage-I and bbox refinement loss L rebox for Stage-II, which are formulated as follows:
[Equation]
L sem-cls is a multi-class focal loss [21] used to supervise voxel-wise semantic segmentation.
L vote is a smooth- ℓ 1 loss for predicting the center offset of each voxel.
In term of the 3D proposal generation
[Table]
module, we follow the same loss functions L cntr , L box and L cls defined in [7] to optimize object centerness, bounding box estimation and classification respectively.
For the second stage, L rebox is the residual-based smooth- ℓ 1 box regression loss for 3D box proposal refinement
[Table]
which contains size, box center and angle refinement loss.
Besides that, we also add the same IoU loss L iou as used in stage-
I
[Table]
and the final box refinement loss is as follows
[Equation]
where ∆ r is the predicted residual and ∆ r ∗ is the corresponding ground truth.
The detailed balancing factors are in Appendix.
Experiments 4.1 Datasets and Evaluation Metric
Our CAGroup3D is evaluated on two indoor challenging 3D scene datasets,
i
[Table]
e. , ScanNet V2 [6] and
SUN RGB-D [33].
For all datasets, we follow the standard data splits adopted in [27].
ScanNet V2 contains richly-annotated 3D reconstructed indoor scenes with axis-aligned bounding box for most common 18 object categories.
It contains 1201 training samples and the remainingscans are left for validation.
We follow [27] to sample point clouds from the reconstructed meshes.
SUN-RGB-D is a single-view indoor dataset which consists of 10,355 RGB-D images for 3D scene understanding.
It contains ∼ 5K training images annotated with the oriented 3D bounding boxes and the semantic labels for 10 categories.
To feed the point data to our method, we follow [27] and convert the depth images to point clouds using the provided camera parameters.
All the experiment results on both datasets are evaluated by a standard evaluation protocol [27, 7], which uses mean average precision(mAP) with different IoU thresholds, i.e. , 0.25 and 0.50.
[Table]
[Table] [Table]
4.2 Implementation Details
Network Architecture Details.
For both datasets, we set the voxel size as 0 .
02 m .
As for the backbone, we use the same 3D voxel ResNet18 introduced in [14] as the downsample branch to extract rich contextual information and set the voxel size of high-resolution branch to 0 .
04 m .
In terms of class-aware 3D proposal generation module, we set the scale factor of re-voxelization α to 0.15.
The semantic threshold τ is initially set to 0.15 and then decreases by 0.02 every 10 epochs for
ScanNet V2 and 4 epochs for SUN RGB-D until it reaches the minimum value of 0.05.
Moreover, the kernel size of convolution-based grouping module k ( a ) is 9.
We stack two sparse abstraction blocks for extracting RoI-specific representations, in where the proposal sampling resolutions are set to 7 × 7 × 7 and 1 × 1 × 1 , and the kernel sizes of sparse grouping k ( p ) are set to 5 and 7 respectively.
Training and Evaluation Scheme.
Our model is trained in an end-to-end manner by AdamW optimizer [23].
Following the [7], we set batch size, initial learning rate and weight decay are 16, 0.001 and 0.0001 for both datasets.
Training ScanNet V2 requires 120 epochs with the learning rate decay by 10x on 80 epochs and 110 epochs.
SUN RGB-D takes 48 epochs and learning rate decayed on 32, 44 epochs.
All models are trained on two NVIDIA Tesla V100 GPUs with a 32 GB memory per-card.
The gradnorm clip [51, 18] is applied to stabilize the training dynamics.
We follow the same evaluation scheme in [22], which runs training for 5 times and test each trained model fortimes.
We report both the best and average metrics across all results.
4.3 Benchmarking Results
We report the comparison results with state-of-the-art 3D detection methods on ScanNet V2 [6]] and
SUN RGB-D [33] benchmark.
Same as [22], we report the best and average results of 5 × 5 trials.
As shown in Table 1, our approach leads to 75.1 in terms of mAP@0.25 and 61.3 in terms of mAP@0.50 on ScanNet V2 [6], which is +3.6 and +4.0 better than the state-of-the-art [7].
For SUN
RGB-D, our approaches achieves 66.8 and 50.2, which gains +2.6 and +1.3 in terms of mAP, with 3D IoU threshold 0.25 and 0.5 respectively.
Moreover, our approach outperforms all the multi-sensor based methods, which is more challenging for SUN RGB-D due to its relatively poor point clouds.
4.4 Ablation Studies and Discussions
We conduct extensive ablation studys on the val sets of ScanNet V2 to analyze individual components of our proposed method.
Following [22], we report the average performance of 25 trials by default.
[Table] [Table]
[Table] [Table]
Effect of class-aware local grouping module.
We first ablate the effects of class-aware local grouping module in Table 2, 3, 4.
In Table 2, the base competitor ( 1 st row) is the fully sparse convolutional VoteNet [27] we implemented.
Compared with CAGroup3D, it abandons the two stage refinement, replaces the BiResNet with FPN-based variant and groups the vote voxels with the voxel size of 0.02 in a class-agnostic manner.
As evidenced in the 1 st and the 2 nd rows, by considering the semantic consistency, our model performs better, i.e. , 68.22 → 69.24, 53.17 → 54.05 on mAP@0.
and mAP@0.5.
Combining the semantic predictions with class-specific local groups ( 3 rd rows), our model achieves a significant improvement, i.e. , 68.22 → 72.10, 53.17 → 57.07.
That verifies our motivation that semantic consistency within the same group and diverse locality among different categories are crucial for an effective grouping algorithm.
Our class-aware local grouping module also works well for a range of hyper-parameters, such as the scale factor α of class-specific re-voxelization and the semantic threshold τ .
Table 3 shows the performance of our model with different scale factors.
We first set k ( a ) to 9 and reduce the scale factor α gradually.
With the decrease of α , the output voxels are more fine-grained, which is beneficial to estimate accurate bounding boxes and localize small objects.
However, a very small voxel size will degrade the performance.
With the fixed k ( a ) , smaller voxel sizes will lead to smaller local regions, which may mis-group the boundary object points, and thus it is hard for the network to accurately capture local object geometry.
Table 4 also ablates the effectiveness of different semantic threshold.
We observe that the performance gradually improves with the decrease of τ .
However, too small threshold can also drops the performance due to the abandonment of semantic consistency.
Effect of RoI-Conv pooling module.
Table 2 demonstrates the effectiveness of two-stage refinement with RoI-Conv pooling module.
By comparing the 4 rd and 5 th rows, we find that our refinement module can really help to detect more accurate 3D bounding boxes, especially in term of mAP@0.50, i.e. , 57.18 → 60.31.
To further ablate the high performance of our RoI-Conv pooling module, we compare it to several pooling strategies [30, 32] widely used in 3D object detection.
For a fair comparison, we only switch the RoI pooling algorithm while all other settings remain unchanged, ( e.g. , class-aware local grouping and BiResNet).
Table 5 shows that our approach surpasses others on the performance of both detection scores and computation cost with a remarkable margin.
Move details about the above competitors are in Appendix.
Table 6 shows the impact of stacking different number of sparse abstraction blocks with k ( p )
=
5 except the last layer.
We choose the relative shallow design of two layers as there are no obvious improvement by additional deepening.
We further ablate the influence of proposal sampling resolution and sparse kernel size k ( p ) based on our two layers architecture.
Table 7 and 8 show that both the larger G and k ( p ) can capture more fine-grained geometric details and lead to better performance.
Considering the trade-off between memory usage and performance improvement, our model finally sets G and k ( p ) of the 1 st layer to 7 × 7 × 7 and 5.
Effect of bilateral feature learning.
In Table 2, we investigate the effects of bilateral backbone (BiResNet18) by replacing it with FPN-based ResNet18 [7, 14].
The 3 th and 4 th rows exhibit that the performance drops, especially in term of mAP@0.25, from 73.21 → 72.10, when adopts FPN-based backbone.
This phenomenon validates that our bilateral backbone could learn much richer contextual information while maintain the high-resolution representations.
Conclusion
In this paper, we propose CAGroup3D, a two-stage fully convolutional 3D object detector, which generates some 3D proposals by utilizing the class-aware local grouping module on the object voxels with same semantic predictions.
Then, to efficiently recover the features of the missed voxels due to incorrect semantic segmentation, we design a fully sparse convolutional RoI pooling module, which is memory-and-computation efficient and could better encode spatial information than previous maxpooling based RoI methods.
Equipped with the above designs, our model achieves state-of-the-art on
ScanNet V2 and SUN RGB-D benchmarks with remarkable performance gains.
Limitations.
CAGroup3D mainly focuses on the inter-category locality, which is class-specific and diverse among the different classes, but ignores the intra-category discriminations.
Due to the incompleteness of the point cloud and the scale variance within the classes, the object spatial dimension of the same class is also variational, which leads to the diverse intra-category locality.
Although our grouping algorithm can implicitly handle this problem in some degree by the learnable convlutional aggregation module, it is still an open problem and will be studied in the future.
Acknowledgement
Liwei Wang is supported by National Science Foundation of China (NSFC62276005), The Major Key
Project of PCL (PCL2021A12), Exploratory Research Project of Zhejiang Lab (No. 2022RC0AN02), and Project 2020BD006 supported by PKUBaidu Fund.
We gratefully acknowledge the support of
MindSpore * .
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[No] Our work is only for academic research purpose.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Our implementation details are posted on Section 4.2 and the supplemental materials.
The data are public datasets and our code will be available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.2 and Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report both the best and average performance oftrials by default.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.2. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] ScanNet V2 [6] and SUN RGB-D
[33] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] We don’t include any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] ScanNet V2 [6] and SUN RGB-D [33] are two standard benchmarks.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] ScanNet V2 [6] and SUN RGB-D
[33] are two standard benchmarks.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
We didn’t use crowdsourcing or conduct research with human subjects...
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
We didn’t use crowdsourcing or conduct research with human subjects...
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A] We didn’t use crowdsourcing or conduct research with human subjects...