B IDIRECTIONAL P ROPAGATION FOR C ROSS -M ODAL 3D O BJECT D ETECTION
A BSTRACT
Recent works have revealed the superiority of feature-level fusion for cross-modal 3D object detection, where fine-grained feature propagation from 2D image pixels to 3D LiDAR points has been widely adopted for performance improvement.
Still, the potential of heterogeneous feature propagation between 2D and 3D domains has not been fully explored.
In this paper, in contrast to existing pixelto-point feature propagation, we investigate an opposite point-to-pixel direction, allowing point-wise features to flow inversely into the 2D image branch.
Thus, when jointly optimizing the 2D and 3D streams, the gradients back-propagated from the 2D image branch can boost the representation ability of the 3D backbone network working on LiDAR point clouds.
Then, combining pixel-to-point and point-to-pixel information flow mechanisms, we construct an bidirectional feature propagation framework, dubbed BiProDet.
In addition to the architectural design, we also propose normalized local coordinates map estimation, a new 2D auxiliary task for the training of the 2D image branch, which facilitates learning local spatial-aware features from the image modality and implicitly enhances the overall 3D detection performance.
Extensive experiments and ablation studies validate the effectiveness of our method.
Notably, we rank 1 st on the highly competitive KITTI benchmark on the cyclist class by the time of submission.
The source code is available at https://github.com/Eaphan/BiProDet.
I NTRODUCTION
In recent years, 3D object detection has received increasing academic and industrial attention driven by the thriving of autonomous driving scenarios, where the two dominating data modalities 3D point clouds and 2D RGB images demonstrate complementary properties in many aspects.
3D point clouds scanned by LiDAR encode accurate structure and depth cues in the geometric domain but typically suffer from severe sparsity, incompleteness, and non-uniformity.
The 2D RGB images captured by optical cameras convey rich semantic features in the visual domain and facilitate the application of well-developed image learning architectures (He et al., 2016) but pose challenges in reliably modeling spatial structures.
More importantly, due to the essential difficulties in processing irregular and unordered point clouds, the actual learning ability of neural network backbones working on 3D LiDAR data is still relatively insufficient , thus limiting the performance of LiDAR-based detection frameworks.
Practically, despite the efforts made by previous studies (Wang et al., 2019;
Vora et al., 2020) devoted to cross-modal learning, how to reasonably mitigate the big modality gap between images and point clouds and fuse the corresponding heterogeneous feature representations still remains an open and non-trivial problem.
Depending on specific fusion strategies between image and point cloud modalities, existing crossmodal 3D object detection pipelines can be categorized into: 1) proposal-level ; 2) result-level ; 3) pseudo-LiDAR-based ; and 4) point-level processing paradigms, as illustrated in Fig.
Concretely, proposal-level fusion methods (Chen et al., 2017; Ku et al., 2018) perform feature fusion for both modalities over each of the proposals/anchors, while result-level fusion methods (Pang et al., 2020) directly manipulate the output 2D and 3D object candidates by leveraging geometric and semantic consistencies.
However, the former two fusion paradigms do not make full use of fine-grained correspondence between 3D LiDAR points and 2D images, and thus show inferior detection performance.
For another alternative fusion paradigm based on pseudo-LiDAR representations (You et al., 2020), auxiliary learning modules specialized for stereo depth estimation are integrated to synthesize dense pseudo-LiDAR point clouds as additional complements to the original real-scanned sparse LiDAR data.
Such a strategy indeed brings performance boosts but it becomes more technically cumbersome and demanding, since ground-truth depth maps are required as supervision signals.
Moreover, as investigated previously (Qian et al., 2020), it is non-trivial to harmonize the joint end-to-end optimization of the object detector and the depth estimator.
To make use of point-pixel correspondences between images and point clouds, there has been another family of point-level fusion approaches (Huang et al., 2020; Vora et al., 2020) characterized by using projection from the 3D space of point cloud to 2D image planes.
That is, for each 3D point, its geometric feature extracted from the 3D
LiDAR learning branch is fused with the semantic feature (retrieved from the 2D image learning branch) of the corresponding 2D pixel.
Benefiting from more fine-grained feature manipulation, point-level fusion typically demonstrates more significant performance improvement and thus is considered as a more promising cross-modal fusion paradigm.
Still, based on our observations, despite the superiority of point-level fusion, previous studies have not fully exploited its great potential in terms of specific architectural design and technical workflow.
Fundamentally, the performance boost achieved by previous methods can be mainly credited to the fact that additional discriminative 2D semantic information is incorporated into the 3D detection pipeline.
However, there is no evidence that the actual representation capability of the 3D
LiDAR backbone network, which is also one of the most critical influencing factors, is strengthened.
Besides, it is also noticed that the choice of 2D auxiliary tasks used for training the image learning branch also makes a difference, which essentially determines the consequent 2D feature representations to be propagated.
Previous methods directly resort to classic semantic understanding tasks (e.g., image segmentation) for the learning of visual patterns, which may not be the optimal task that harmonizes with the 3D LiDAR learning branch for geometric modeling.
Given the above issues, in this paper we propose a bidirectional feature propagation architecture, namely BiProDet, for cross-modal 3D object detection.
Foremost, our core component lies in a novel point-to-pixel feature propagation mechanism allowing 3D geometric features of LiDAR point clouds to flow into the 2D image learning branch.
Our BiProDet differentiates from previous crossmodal 3D detection pipelines that solely pay attention to unidirectional 2D-to-3D (pixel-to-point) feature propagation while neglecting the potential effects of the opposite 3D-to-2D (point-to-pixel) direction.
Functionally, different from existing pixel-to-point propagation, the proposed point-topixel propagation turns out to be capable of enhancing the representation ability of 3D LiDAR backbone networks .
In fact, it is worth emphasizing that this is an interesting phenomenon, and we reason that this is because the gradients back-propagated from the auxiliary 2D image branch built upon mature 2D convolutional neural networks (CNNs) with the impressive learning ability can effectively strengthen the 3D LiDAR branch .
Taking a step forward, we integrate pixel-to-point and point-to-pixel propagation directions into a unified interactive bidirectional feature propagation framework such that 2D and 3D learning branches are inter-strengthened, eventually contributing to more significant performance gains.
In addition, we also particularly customize a new 2D auxiliary task called normalized local coordinate (NLC) map estimation, which facilitates learning local spatial-aware feature representations from the 2D image.
Such a task also implicitly helps to capture more discriminative geometric information from the main 3D LiDAR point cloud learning branch, thus improving the overall detection performance, especially for hard cases of distant and/or highlyoccluded objects.
We conduct extensive experiments on the prevailing KITTI (Geiger et al., 2012) benchmark dataset.
Compared with state-of-the-art single-modal and cross-modal 3D object detectors, our framework achieves remarkable performance improvement.
Notably, our method ranks 1 st on the cyclist class of KITTI 3D detection benchmark 1 .
Conclusively, our main contributions are three-fold:
• In contrary to existing pixel-to-point cross-modal fusion strategies, we explore an opposite point-to-pixel feature propagation paradigm, which turns out to be capable of enhancing the representation capability of 3D backbone networks working on LiDAR point clouds.
• Combining the existing pixel-to-point and the newly proposed point-to-pixel feature propagation schemes, we further construct the powerful bidirectional feature propagation for crossmodal 3D objection detection.
• We reveal the importance of specific 2D auxiliary tasks used for training the 2D image learning branch which has not received enough attention in previous works, and introduce 2D NLC map estimation to facilitate learning spatial-aware features and implicitly boost the overall 3D detection performance.
R ELATED W ORK
LiDAR-based 3D Object Detection could be roughly divided into two categories.
1) Voxel-based 3D detectors typically voxelize the point clouds into grid-structure forms of a fixed size (Zhou &
Tuzel, 2018).
Yan et al.
(2018) introduced a more efficient sparse convolution to accelerate training and inference.
He et al.
(2020a) employed auxiliary tasks including center estimation and foreground segmentation to guide the network to learn the intra-object relationship.
2) Point-based 3D detectors consume the raw 3D point clouds directly and generate predictions based on (downsampled) points.
Shi et al.
(2019) applied a point-based feature extractor and generated high-quality proposals on foreground points.
3DSSD (Yang et al., 2020) adopted a new sampling strategy named F-FPS as a supplement to D-FPS to preserve enough interior points of foreground instances.
It also built a onestage anchor-free 3D object detector based on feasible representative points.
Zhang et al. (2022c) and Chen et al.
(2022) introduced semantic-aware down-sampling strategies to preserve foreground points as much as possible.
Shi & Rajkumar (2020b) proposed to encode the point cloud by a fixedradius near-neighbor graph.
Besides, PV-RCNN (Shi et al., 2020a) adopted a voxel-based RPN and leverages point-wise features extracted from the voxel set abstraction module to refine the proposals.
Camera-based 3D Object Detection.
Early works (Mousavian et al., 2017; Manhardt et al., 2019) designed monocular 3D detectors by referring to 2D detectors (Ren et al., 2015; Tian et al., 2019) and utilizing 2D-3D geometric constraints.
Another way is to convert images to pseudo-lidar representations via monocular depth estimation and then resort to LiDAR-based methods (Wang et al., 2019).
Chen et al.
(2021) built dense 2D-3D correspondence by predicting object coordinate map and adopted uncertainty-driven PnP to estimate object location.
Recently, multi-view detectors like
BEVStereo (Li et al., 2022) have also achieved promising performance, getting closer to LiDARbased methods.
Cross-Modal 3D Object Detection can be roughly divided into four categories: proposal-level, result-level, point-level, and pseudo-lidar-based approaches.
Proposal-level fusion methods (Chen et al., 2017; Ku et al., 2018; Liang et al., 2019) adopted a two-stage framework to fuse image features and point cloud features corresponding to the same anchor or proposal.
For result-level fusion,
Pang et al.
(2020) exploited the geometric and semantic consistencies of 2D detection results and 3D detection results to improve single-modal detectors.
Qi et al.
(2018) generated 2D region proposals using a CNN and lifted them to frustum proposals, where a PointNet-based network is utilized to estimate 3D box.
Obviously, both proposal-level and decision-level fusion strategies are coarse and do not make full use of correspondence between LiDAR points and images.
Pseudo-LiDAR based methods (You et al., 2020; Wu et al., 2022; Liang et al., 2019) employed depth estimation/completion and converted the estimated depth map to pseudo-LiDAR points to complement raw sparse point clouds.
However, such methods require extra depth map annotations that are high-cost.
Regarding point-level fusion methods, Liang et al.
(2018) proposed the continuous fusion layer to retrieve corresponding image features of nearest 3D points for each grid in BEV feature map.
Especially, Vora et al.
and
Huang et al.
(2020) retrieved the semantic scores or features by projecting points to image plane.
Liu et al.
(2021) proposed EPNet++, where an LI-Fusion layer is used to enable more interaction between the two modalities to obtain more comprehensive features.
And Wang et al.
(2021) decorated point clouds with middle CNN features extracted from 2D detection models.
However, neither semantic segmentation nor 2D detection tasks can enforce the network to learn 3D-spatial-aware image features.
By contrast, we introduce NLC Map estimation as an auxiliary task to promote the image branch to learn local spatial information to supplement the sparse point cloud.
Besides, Bai et al.
(2022) proposed TransFusion, a transformer-based detector that performs fine-grained fusion with attention mechanisms on the BEV level.
Philion & Fidler (2020) and Liang et al.
(2022) estimated the depth of multi-view images and transformed the camera feature maps into the BEV space before fusing them with LiDAR BEV features.
P ROPOSED M ETHOD
Overview .
Architecturally, the overall processing pipeline of cross-modal 3D object detection consists of an image branch and a point cloud branch, learning feature representations from 2D RGB images and 3D LiDAR point clouds, respectively.
Existing methods implement cross-modal learning only by incorporating pixel-wise semantic cues from the 2D image branch into the 3D point cloud branch for feature fusion.
Here, motivated by the observation that point-based networks typically show insufficient learning ability due to the essential difficulties in processing irregular and unordered point cloud data modality (Zhang et al., 2022a), we additionally seek to boost the expressive power of the point cloud backbone network with the assistance of the 2D image branch, whose potential is ignored in previous studies.
It is also worth reminding that our ultimate goal is not to design a more powerful point cloud backbone network structure but to elegantly make full use of the available resources in this cross-modal application scenario.
As shown in Fig.
2, in addition to pixel-to-point feature propagation explored by mainstream pointlevel fusion methods (Liang et al., 2018; Huang et al., 2020), we propose point-to-pixel propagation allowing features to flow inversely from the point cloud branch to the image branch, based on which we can achieve bidirectional feature propagation implemented in a multi-stage fashion.
In this way, not only the image features propagated via the pixel-to-point module can provide additional semantic information, but also the gradients backpropagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone.
Besides, we also employ auxiliary tasks to train the pipeline, aiming to enforce the network to learn rich semantic and spatial representations.
In particular, we propose NLC map estimation to promote the image branch to learn spatial-aware features, providing a necessary complement to the sparse spatial representation extracted from point clouds, especially for distant or highly occluded cases.
We refer the readers to
Appendix A.2 for detailed network structures of the image and point cloud backbones.
3. B IDIRECTIONAL F EATURE P ROPAGATION
As illustrated in Fig.
3, the proposed bidirectional feature propagation consists of a pixel-to-point module and a point-to-pixel module, which bridges the two learning branches that operate on RGB images and LiDAR point clouds.
Functionally, the point-to-pixel module applies grid-level interpolation on point-wise features to produce a 2D feature map, while the pixel-to-point module retrieves the corresponding 2D image features by projecting 3D points to the 2D image plane.
In general, we partition the overall 2D and 3D branches into the same number of stages, such that we can perform bidirectional feature propagation at each stage between the corresponding layers of the image and point cloud learning networks.
Without loss of generality, below we only focus on a certain stage, where we can acquire a 2D feature map F ∈ R C 2 D × H ′ × W ′ with C 2 D channels and dimensions H ′ × W ′ from the image branch and a set of C 3 D -dimensional
embedding vectors
G = (cid:8) g i ∈ R C 3 D (cid:9) N ′ i =
1 of the corresponding N ′ 3D points.
Point-to-Pixel Module.
We design this module to propagate geometric information from the 3D point cloud branch into the 2D image branch.
Formally, we expect to construct a 2D feature map
F P 2 I ∈ R C 3 D × H ′ × W ′ by querying from the acquired point-wise embeddings G .
More specifically, for each pixel position ( r, c ) over the targeted 2D grid of F P 2 I of dimensions H ′ × W ′ , we collect the corresponding geometric features of points that are projected inside the range of this pixel and then perform feature aggregation by max-pooling MAX ( · ) :
[Equation]
where n counts the number of points that can be projected inside the range of the pixel located at the r th row and the c th column.
Particularly, for empty pixel positions where there is no point projected inside, we simply define its feature value as 0 .
After that, instead of directly feeding the “interpolated” 2D feature map F P 2 I into the subsequent 2D convolutional stage, we introduce a few additional refinement procedures for feature smoothness as described below:
[Equation]
where CAT [ · , · ] and CONV ( · ) stand for feature channel concatenation and the convolution operation, respectively, and the resulting F fuse is further fed into the subsequent layers of the 2D image branch.
Pixel-to-Point Module.
From an opposite direction, this module propagates semantic information from the 2D image branch into the 3D point cloud branch.
We start by projecting the corresponding
N ′ 3D points onto the 2D image plane, obtaining their planar coordinates denoted as X
= { x i ∈
R 2 } N ′ i =1 .
Considering that these 2D coordinates are continuously and irregularly distributed over the projection plane, we apply bilinear interpolation to compute exact image features at projected positions, deducing a set of point-wise embeddings F I 2 P = { F ( x i
) ∈ R C 2 D | i = 1 , ..., N ′ } .
Similar to our practice in the point-to-pixel module, the interpolated 3D point-wise features F I 2 P need to be refined by the following procedures: [Equation]
[Equation]
where MLP ( · ; θ ) denotes shared multi-layer perceptrons parameterized by learnable weights θ , and the resulting G fuse further passes through the subsequent layers of the 3D point cloud branch.
3. A UXILIARY T ASKS FOR T RAINING
The semantic and spatial-aware information is essential for determining the category and boundary of objects.
To promote the network to uncover such information of input data, we also leverage auxiliary tasks when training the whole pipeline.
For the point cloud branch, following SA-SSD
(He et al., 2020b), we introduce 3D semantic segmentation and center estimation to learn structure-aware features.
For the image branch, we particularly propose NLC map estimation, in addition to 2D semantic segmentation.
Normalized Local Coordinate (NLC) System.
We define the NLC system as taking the center of an object as the origin, aligning the x -axis towards the head direction of its ground-truth (GT) bounding box, then normalizing the local coordinates with respect to the size of the GT bounding box.
Fig.
4 (a) shows an example of the NLC system for a typical car.
With the geometry relationship between the input RGB image and point cloud, we can project NLCs to the 2D image plane to construct a 2D NLC map with three channels corresponding to three spatial dimensions, as illustrated in Fig.
4 (b).
We refer the reader to
Appendix A.1 for the detailed process.
NLC Map Estimation.
Intuitively, vision-based semantic information is useful for network to distinguish the foreground from the background.
However, the performance bottleneck of existing detectors lies in the limited localization accuracy in distant or highly occluded cases, where the spatial structure is incomplete due to the sparsity of point clouds.
To this end, we further take NLC map as supervision to learn the relative position of each pixel inside the object from the image.
We expect such an auxiliary task can drive the image branch to learn local spatial-aware features, which serve as a complement to the sparse spatial representation extracted from point clouds.
Besides, this task could have an effect on augmenting the representation ability of the point cloud branch.
Remark .
The proposed NLC map estimation shares the identical objective with pseudo-LiDARbased methods, i.e, enhancing the spatial representation limited by the sparsity of point clouds.
However, compared with the manner of learning a pseudo-LiDAR representation via depth completion, our NLC map estimation has the following advantages: 1) the local NLC is easier to learn than the global depth owing to its scale-invariant property at different distances; 2) our NLC map estimation can be naturally incorporated and trained with the detection pipeline end-to-end, while it is non-trivial to optimize the depth estimator and LiDAR-based detector simultaneously in pseudoLiDAR based methods though it can be implemented in a somewhat complex way (Qian et al., 2020); and 3) the pseudo-LiDAR representations require ground-true dense depth images as supervision, which may not be available in reality, but our method does not require such information.
Semantic Segmentation.
Leveraging semantics of 3D point clouds has demonstrated effectiveness in point-based detectors, owing to the explicit preservation of foreground points at downsampling (Zhang et al., 2022c; Chen et al., 2022).
Therefore, we further introduce the auxiliary semantic segmentation tasks not only in the point cloud branch but also in the image branch, to exploit richer semantic information.
Needless to say, additional semantic features extracted from images facilitate the network to distinguish true positive candidate boxes from the false-positive.
3. L OSS F UNCTION
The NLC map estimation task is optimized with the loss function defined as
[Equation]
where p i is the i th LiDAR point, N pos is the number of foreground LiDAR points, ∥·∥ H is the
Huber-loss, 1 p i indicates the loss is calculated only with foreground points, and p NLCi and ˆ p NLCi are the NLCs of ground-truth and prediction at corresponding pixel for foreground points.
We use the standard cross-entropy loss to optimize both 2D and 3D semantic segmentation, denoted as L 2 Dsem and L 3 Dsem respectively.
And the loss of center estimation is computed as
[Equation]
where ∆ p i is the target offsets from points to the corresponding object center, and ∆ˆ p i denotes the output of center estimation head.
Besides the auxiliary tasks, we follow (Chen et al., 2022) to define the loss of the RPN stage as
[Equation]
where L c is the classification loss,
L r the regression loss, L s the shifting loss between the predicted shifts and residuals from the sampled points to their corresponding object centers in candidate generate layer (Yang et al., 2020), and L d the point segmentation loss in SA layers to enable semanticguided sampling.
In addition, we also adopt commonly used proposal refinement loss L rcnn as defined in (Shi et al., 2020a).
In all, the loss for optimizing the overall pipeline in an end-to-end manner is written as
[Equation]
where λ 1 , λ 2 , λ 3 , and λ 4 are hyper-parameters that are empirically set to 1.
E XPERIMENTS 4. E XPERIMENT S ETTINGS
Datasets and Metrics.
We conducted experiments on the prevailing KITTI benchmark dataset, which contains two modalities of 3D point clouds and 2D RGB images.
Following previous works (Shi et al., 2020a), we divided all training data into two subsets, i.e., 3712 samples for training and the rest 3769 for validation.
Performance is evaluated by the Average Precision (AP) metric under IoU thresholds of 0.7, 0.5, and 0.5 for car, pedestrian, and cyclist categories, respectively.
We computed APs with 40 sampling recall positions by default, instead of 11.
For the 2D auxiliary task of semantic segmentation, we used the instance segmentation annotations as provided in (Qi et al., 2019).
Besides, we also conducted experiments on the Waymo Open Dataset (WOD) (Sun et al., 2020), which can be found in Appendix A.4.
Implementation Details.
For the image branch, we used ResNet18 (He et al., 2016) as the backbone encoder, followed by a decoder composed of pyramid pooling module (Zhao et al., 2017) and several upsampling layers to give the outputs of semantic segmentation and NLC map estimation.
For the point cloud branch, we deployed a point-based network like (Yang et al., 2020) with extra FP layers and further applied semantic-guided farthest point sampling (S-FPS)
(Chen et al., 2022) in SA layers.
Thus, we implemented bidirectional feature propagation between the top of each SA or FP layer and their corresponding locations at the image branch.
Please refer to Appendix A.2 for more details.
4. C OMPARISON WITH S TATE - OF - THE -A RT M ETHODS
We submitted our results to the official website of KITTI.
As compared in Table 1, our BiProDet outperforms existing state-of-the-art methods on the KITTI test set by a remarkable margin, i.e., an absolute increase of 2.1% mAP over the second best method EQ-PVRCNN.
Notably, by the time of submission, we ranked 1 st on the KITTI 3D detection benchmark for the cyclist class.
Particularly, it is observed that BiProDet shows consistent and more obvious superiority on “moderate” and “hard” levels, i.e., the distant or highly-occluded objects with sparse points.
We ascribe such performance gains to the fact that our proposed bidirectional feature propagation mechanism contributes to more adequate exploitation of complementary information between multi-modalities as well as the effects of 2D auxiliary tasks (as verified in Sec. 4.3).
Besides, we can observe from the actual visual results (as presented in Appendix A.7) that our BiProDet is able to produce higher-quality 3D bounding boxes in varying scenes.
[Table] [Table]
[Table]
A BLATION S TUDY
We conducted comprehensive ablation studies to validate the effectiveness and explore the impacts of key modules involved in the overall learning framework
[Table]
Effectiveness of Bidirectional Propagation.
We performed detailed ablation studies on specific multi-modal feature exploitation and interaction strategies.
We started by presenting a single-modal baseline (Table 2 (a)) that only preserves the point cloud branch of our BiProDet for both training and inference.
Based on our complete bidirectional propagation pipeline (Table 2 (e))
[Table]
we explored another two variants as shown in (Table 2 (c)) and (Table 2 (d)), solely preserving the point-to-pixel and pixel-to-point feature propagation in our 3D detectors, respectively.
Note that in Table 2 (c) the point-to-pixel feature flow was only enabled during training, and we detached the point-to-pixel module as well as the whole image branch during inference.
In addition, in Table 2 (b), we replaced the bidirectional feature propagation of our BiProDet with the input fusion strategy as proposed in (Wang et al., 2021) that decorates point clouds with CNN features deduced from the image branch.
Empirically, we can draw several aspects of conclusions that strongly support our preceding claims.
First , the performance of Table 2 (a) turns out to be the worst among all variants, which reveals the superiority of cross-modal learning.
Second, combining Table 2 (a) and Table 2 (c), the mAP is largely boosted from 75.88% to 77.01%.
Considering that, during the inference stage, these two variants have identical forms of input and network structure, the resulting improvement strongly indicates that the representation ability of the 3D LiDAR branch is indeed strengthened.
Third, comparing Table 2 (c) and Table 2 (d) with Table 2 (e), we can verify the superiority of bidirectional propagation (78.64%) over the unidirectional schemes (77.01% and 77.47%).
If we particularly pay attention to Table 2 (d) and Table 2 (e), we can conclude that our newly proposed point-to-pixel feature propagation direction further brings 1.17% mAP increase based on the previously explored pixel-to-point paradigm.
Besides , by comparing Table 2 (b) (77.07%) and Table 2 (d) (77.47%) whose information propagation directions are both from the 2D image domain to the 3D point cloud domain, we can demonstrate the superiority of feature-level propagation over its input-level counterpart (Wang et al., 2021).
Effectiveness of Image Branch.
Comparing Table 3 (d) with Tables 3 (e)-(g), the performance is stably boosted from 75.88% to 78.64%, as the gradual addition of the 2D image branch and two
[Table]
auxiliary tasks.
These results indicate that the additional semantic and geometric features learned from the image modality are indeed effective supplements to the point cloud representation, leading to significantly improved 3D object detection performances.
Effectiveness of Auxiliary Tasks.
Comparing Table 3 (a) with Tables 3 (b)-(d), the mAP is boosted by 0.70% when incorporating 3D semantic segmentation and 3D center estimation
[Table]
and can be further improved by 2.76% after introducing 2D semantic segmentation and NLC map estimation (Tables 3 (d)-(g)).
However, only integrating image features without 2D auxiliary tasks (comparing results of Tables 3 (d) and (e)) brings limited improvement of 0.5%.
This observation shows that the 2D auxiliary tasks, especially the proposed 2D NLC map estimation, do enforce the image branch to learn complementary information for the detector.
Robustness against Input Corruption.
We also conducted extensive experiments to verify the robustness of our BiProDet to sensor perturbation.
Specifically, we added Gaussian noises to the reflectance value of points or RGB images.
Fig.
5 shows that the mAP value of our cross-modal
BiProDet is consistently higher than that of the single-modal baseline and decreases slower with the
LiDAR noise level increasing.
Particularly, as listed in Table 4, when the variance of the LiDAR noise is set to 0.15, the perturbation affects our cross-modal BiProDet much less than the singlemodal detector.
Besides, even applying the corruption to both LiDAR input and RGB images, the mAP value of our BiProDet only drops by 2.49%.
[Figure]
[Table]
[Figure]
C ONCLUSION
We have presented a novel cross-modal 3D detector, namely BiProDet, by fully exploiting complementary information from the image domain in two aspects.
First, we proposed point-to-pixel feature propagation, which enables gradients backpropagated from the training loss of the image branch to augment the expressive power of the 3D point cloud backbone.
Second, we proposed
NLC map estimation as an auxiliary task to promote the image branch to learn local spatial-aware representation rather than just semantic features.
Our BiProDet achieves state-of-the-art results on the KITTI detection benchmark.
And extensive ablative experiments demonstrate the robustness of our BiProDet against sensor noises and generalization to LiDAR signals with fewer beams.
The decent results demonstrate the potential of joint training between 3D object detection and more 2D scene understanding tasks.
We believe our new perspective will further inspire investigations on multi-task multi-modal learning for scene understanding in autonomous driving.
A CKNOWLEDGEMENT
This work was supported in part by the Hong Kong Research Grants Council under Grants 11219422, 11202320, and 11218121, and in part by Hong Kong Innovation and Technology Fund under Grant MHP/117/21.
R EPRODUCIBILITY S TATEMENT
We provide the source code and configuration in the supplementary material for the experimental results presented in the main text.
We specify the settings of hyper-parameters, the training scheme, and the implementation details of our method in Section 4.1 and Appendix A.2.
Besides, we also give a clear explanation of used datasets.
We repeat the experiments on the KITTI dataset several times to check the rightness and reproducibility of the implementation.
E THICS S TATEMENT
Proposing effective cross-modal 3D object detectors would benefit a wide range of applications.
For example, the incomplete information obtained from the single-modal sensor in autonomous driving vehicles may lead to serious traffic accidents.
And the cross-modal detectors are expected to achieve more accurate detection results and be more robust to environment perturbation.
Besides, this study was conducted using publicly available datasets and without surveying participants for experiments.
Thus we do not consider this study to raise privacy or data security issues.
We cited the creators when using existing datasets.
A A PPENDIX
In this appendix, we provide the details omitted from the manuscript due to space limitation.
We organize the appendix as follows.
• Section A.1: Details of the normalized local coordinate (NLC) System.
• Section A.2: Implementation details.
• Section A.3: More quantitative results.
• Section
A.4:
Experimental results on Waymo dataset.
• Section A.5: More ablation studies.
• Section A.6: Efficiency analysis.
• Section A.7: Visual results of 3D object detection.
• Section A.8: Visual results of 2D semantic segmentation.
• Section A.9: Details on the official KITTI test leaderboard.
A. D ETAILS OF N ORMALIZED L OCAL C OORDINATE (NLC) S YSTEM
The aim of the cross-modal 3D detector is to estimate the bounding box parameterized by the object dimension ( w , l , h ), the center location ( x c , y c , z c ) , and the orientation angle θ .
The input contains an RGB image X ∈ R 3 × H × W , where H and W represent the height and width of the image respectively, and a 3D LiDAR point cloud { p i | i = 1 , ..., N } , where N is the number of points, and each point p i ∈ R 4 is represented with its 3D location ( x p , y p , z p ) in the LiDAR coordinate system and the reflectance value ρ .
To build the correspondence between the 2D and 3D modalities, we project the observed points from the 3D coordinate system to the 2D coordinate system on the image plane:
[Equation]
where u , v , d c denote the corresponding coordinates and depth on the image plane, R ∈ R 3 × 3 and
T ∈ R 3 × 1 denote the rotation matrix and translation matrix of the LiDAR relative to the camera, and K ∈ R 3 × 3 is the camera intrinsic matrix.
Given the point cloud and the bounding box of an object, LiDAR coordinates of foreground points can be transformed into the NLC system by proper translation, rotation, and scaling, i.e.,   x NLCp y NLCp z NLCp 
 = (cid:34) 1 /l 01 /w 0 1 /h (cid:35) ·   x LCSp y LCSp z LCSp 
 + (cid:34) 0 .
5 0 .0 .(cid:35)
= (cid:34)
/l 01 /w 0 1 /h (cid:35) · (cid:34) cosθ − sinθ 0 sinθ cosθ 0  (cid:35) · (cid:34) x
p − x c y p − y c
z p − z c (cid:35) + (cid:34) 0 .
5 0 .0 .(cid:35)
, (9) where ( x NLCp , y NLCp , z NLCp ) and ( x LCSp , y LCSp , z LCSp ) denote the coordinates of points in NLC system and local coordinate system, respectively.
Suppose that the bounding box is unknown with known global coordinates and NLCs of points, we can build a set of equations about the parameters of box (i.e., the x c , y c , z c , w, l, h, θ ).
In order to solve the total of 7 parameters, global coordinates and NLCs of at least 7 points are required to construct the equations.
Discussion.
Notably, it is difficult to infer NLCs of points with only LiDAR data for objects far away or with low reflectivity, since the point clouds are sparse.
Meanwhile, the contours and appearance of these cases are still visible in RGB images, so we propose to estimate the normalized local coordinates from images.
Then, we can retrieve the estimated NLCs of observed points based on the 2D-3D correspondence.
Finally, we expect the proposed cross-modal detector to achieve higher detection accuracy with estimated NLCs and known global coordinates.
A. I MPLEMENTATION D ETAILS
Network Architecture.
Fig.
6 illustrates the architectures of the point cloud and image backbone networks.
For the encoder of the point cloud branch, we further show the details of multi-scale
[Table]
grouping (MSG) network in Table 5.
Following 3DSSD (Yang et al., 2020), we take key points sampled by the 3 rd SA layer to generate vote points and estimate the 3D box.
Then we feed these 3D boxes and features output by the last FP layer to the refinement stage.
Besides, we adopt densityaware RoI grid pooling (Hu et al., 2022) to encode point density as an additional feature.
Note that 3D center estimation (He et al., 2020b) aims to learn the relative position of each foreground point to the object center, while the 3D box is estimated based on sub-sampled points.
Thus, the auxiliary task of 3D center estimation differs from 3D box estimation and can facilitate learning structureaware features of objects.
Training Details.
Through the experiments on KITTI dataset, we adopted Adam (Kingma & Ba, 2015) ( β 1 =0.9, β 2 =0.99) to optimize our BiProDet.
We initialized the learning rate as 0.003 and updated it with the one-cycle policy (Smith, 2017).
And we trained the model for a total ofepochs in an end-to-end manner.
In our experiments, the batch size was set to 8, equally distributed on 4 NVIDIA 3090 GPUs.
We kept the input image with the original resolution and padded it to
[Table] [Table]
the size of 1248 × 376 , and down-sampled the input point cloud to 16384 points during training and inference.
Following the common practice, we set the detection range of the x , y , and z axis to [0m, 70.4m], [-40m, 40m] and [-3m, 1m], respectively.
Data Augmentation.
We applied common data augmentation strategies at global and object levels.
The global-level augmentation includes random global flipping, global scaling with a random scaling factor between 0.95 and 1.05, and global rotation around the z -axis with a random angle in the range of [ − π/ 4 , π/ 4 ].
Each of the three augmentations was performed with a 50% probability for each sample.
The object-level augmentation refers to copying objects from other scenes and pasting them to current scene (Yan et al., 2018).
In order to perform sampling synchronously on point clouds and images, we utilized the instance masks provided in (Qi et al., 2019).
Specifically, we pasted both the point clouds and pixels of sampled objects to the point cloud and images of new scenes, respectively.
A. M ORE QUANTITATIVE R ESULTS
Performance on KITTI Val Set.
We also reported the performance of our BiProDet on all three classes of the KITTI validation set in Table 6, where it can be seen that our BiProDet also achieves the highest mAP of 77.73%, which is obviously higher than the second best method CAT-Det.
Performance of Single-Class Detector.
Quite a few methods (Deng et al., 2021; Zheng et al., 2021) train models only for car detection.
Empirically, the single-class detector performs better in the car class compared with multi-class detectors.
Therefore, we also provided performance of BiProDet trained only for the car class, and compared it with several state-of-the-art methods in Table 7.
Performance of NLC Map Estimation.
The NLC map estimation task is introduced to guide the image branch to learn local spatial-aware features.
We evaluated the predicted NLC of pixels containing at least one projected point with mean absolute error (MAE) for each object:
[Equation]
[Table]
where N obj is the number of LiDAR points inside the ground-truth box of the object, 1 p i indicates that the evaluation is only calculated with foreground points inside the box,
q NLC p i and ˆ q NLC p i are the normalized local coordinates of the ground-truth and the prediction at the corresponding pixel for the foreground point.
Finally, we obtained the mean value of MAE q for all instances, namely mMAE q .
We report the metric over different difficulty levels for three categories on the KITTI val set at Figure 7.
We can observe that, for the car class, the mMAE error is only 0.0619, i.e., ± 6 .cm error per meter.
For the challenging pedestrian and cyclist categories, the error becomes larger due to the smaller size and the non-rigid shape.
Generalization to Asymmetric Backbones.
As shown in Fig.
6, we originally adopted an encoderdecoder network in the LiDAR branch that is architecturally similar to the image backbone.
Nevertheless, it is worth clarifying that our approach is not limited to symmetrical structures and can be generalized to different point-based backbones.
Here, we replaced the 3D branch of the original framework with an efficient single-stage detector—SASA (Chen et al., 2022), using a backbone only with the encoder in the LiDAR branch, which is asymmetric with the encoder-decoder structure of the image backbone.
Accordingly, the proposed bidirectional propagation is only performed between the 3D backbone and the encoder of the 2D image backbone.
The experimental results are shown in Table 8.
We can observe that the proposed method works well even when the two backbones are asymmetric, which demonstrates the satisfactory generalization ability of our method for different LiDAR backbones.
A. R ESULTS ON W AYMO O PEN D ATASET
The Waymo Open Dataset (Sun et al., 2020) is a large-scale dataset for 3D object detection.
It contains 798 sequences (15836 frames) for training, and 202 sequences (40077 frames) for validation.
According to the number of points inside the object and the difficulty of annotation, the objects are further divided into two difficulty levels: LEVEL 1 and LEVEL 2.
Following common practice, we adopted the metrics of mean Average Precision (mAP) and mean Average Precision weighted by
[Table]
[Table] [Table]
heading accuracy (mAPH), and reported the performance on both LEVEL 1 and LEVEL 2.
We set the detection range to [-75.2m, 75.2m] for x and y axis, and [-2m, 4m] for z axis.
Following Wang et al.
(2021) and Bai et al.
(2022), the training on Waymo dataset consists of two stages to allow flexible augmentations.
First, we only trained the LiDAR branch without image inputs and bidirectional propagation for 30 epochs.
We enabled the copy-and-paste augmentation in this stage.
Then, we trained the whole pipeline for another 6 epochs, during which the copy-and-paste is disabled.
Note that the image semantic segmentation head is disabled, since ground-truth segmentation maps are not provided (Sun et al., 2020).
As shown in Table 9, our method achieves substantial improvement compared with previous stateof-the-arts.
Particularly, unlike existing approaches including PointAugmenting (Wang et al., 2021) and TransFusion (Bai et al., 2022) where the camera backbone is pre-trained on other datasets and then frozen, we trained the entire pipeline in an end-to-end manner.
It can be seen that even without the 2D segmentation auxiliary task, our method still achieves higher accuracy under all scenarios except “Ped L2”, demonstrating its advantage.
A. M ORE A BLATION S TUDIES
Effectiveness of Multi-stage Interaction.
As mentioned before, both 2D and 3D backbones adopt an encoder-decoder structure, and we perform bidirectional feature propagation at both downsampling and upsampling stages.
Here, we conducted experiments to verify the superiority of the multistage interaction over single-stage interaction.
As shown in Table 10, only performing the bidirectional feature propagation in the encoder (i.e., Table 10 (b)) or the decoder (i.e., Table 10 (c)) leads to worse performance than that of performing the module in both stages (i.e., Table 10 (d)).
Effect of Semantic-guided Point Sampling.
When performing downsampling in the SA layers of the point cloud branch, we adopted S-FPS (Chen et al., 2022) to explicitly preserve as many foreground points as possible.
We report the percentage of sampled foreground points and instance recall (i.e., the ratio of instances that have at least one point) in Table 11, where it can be seen that exploiting supplementary semantic features from images leads to substantial improvement of the ratio of sampled foreground points and better instance recall during S-FPS.
[Table]
Influence on 2D Semantic Segmentation.
We also aimed to demonstrate that the 2D-3D joint learning paradigm benefits not only the 3D object detection task but also the 2D semantic segmentation task.
As shown in Table 12, the deep interaction between different modalities yields an improvement of 4.42% mIoU.
The point features can naturally complement RGB image features by providing 3D geometry and semantics, which are robust to illumination changes and help distinguish different classes of objects, for 2D visual information.
The results suggest the potential of joint training between 3D object detection and more 2D scene understanding tasks in autonomous driving.
Conditional Analysis.
To better figure out where the improvement comes from when using additional image features, we compared BiProDet with the single-modal detector on different occlusion levels and distant ranges.
The results shown in Table 13 and Table 14 include separate APs for objects belonging to different occlusion levels and APs for moderate class in different distance ranges.
For car detection, our BiProDet achieves more accuracy gains for long-distance and highly occluded objects, which suffer from the sparsity of observed LiDAR points.
The cyclist and pedestrian are much more difficult categories on account of small sizes, non-rigid structures, and fewer training samples.
For these two categories, BiProDet still brings consistent and significant improvements on different levels even in extremely difficult cases.
[Table] [Table]
Generalization to Sparse LiDAR Signals.
We also compared our BiProDet with the single-modal baseline on LiDAR point clouds with various sparsity.
In practice
[Table]
following Pseudo-LiDAR++ (You et al., 2020)
[Table]
we simulated the 32-beam, 16-beam
[Table]
and 8-beam LiDAR signals by selecting LiDAR points whose elevation angles fall within specific intervals.
As shown in Table
[Table]
the proposed
BiProDet outperforms the single-modal baseline under all settings.
The consistent improvements suggest our method can generalize to sparser signals.
Besides, the proposed BiProDet significantly performs better than the baseline in the setting of LiDAR signals with fewer beams, demonstrating the effectiveness of our method in exploiting the supplementary information in the image domain.
A. E FFICIENCY A NALYSIS
We also compared the inference speed and number of parameters of the proposed BiProDet with state-of-the-art cross-modal approaches in Table 16.
Our BiProDet has about the same number of parameters as CAT-Det (Zhang et al., 2022b), but a much higher inference speed at 9.52 frames per second on a single GeForce RTX 2080 Ti GPU.
In general, our BiProDet is inevitably slower than some single-modal detectors, but it achieves a good trade-off between speed and accuracy among cross-modal approaches.
A. V ISUAL R ESULTS OF 3D O BJECT D ETECTION
In Figure 8, we present the qualitative comparison of detection results between the single-modal baseline and our BiProDet.
We can observe that the proposed BiProDet shows better localization capability than the single-modal baseline in challenging cases.
Besides, we also show qualitative
[Table] [Table]
results of BiProDet on the KITTI test split in Figure 9.
We can clearly observe that our BiProDet performs well in challenging cases, such as pedestrians and cyclists (with small sizes) and highlyoccluded cars.
A. V ISUAL R ESULTS OF 2D S EMANTIC S EGMENTATION
Several examples are shown in Figure 10.
For distant cars in the first and the second row as well as the pedestrian in the sixth row, the size of objects is small and PSPNet tends to treat them as background, while our BiProDet is able to correct such errors.
In the third row, our BiProDet finds the dim cyclist missed by PSPNet.
Our BiProDet also performs better for the highly occluded objects as shown in the fourth and the fifth lines.
This observation shows the 3D feature representations extracted from point clouds can boost 2D semantic segmentation, since the image-based method is sensitive to illumination and can hardly handle corner cases with only single-modal inputs.
A. D ETAILS ON O FFICIAL KITTI T EST L EADERBOARD
We submitted the results of our BiProDet to the official KITTI website, and it ranks 1 st on the 3D object detection benchmark for the cyclist class.
Figure 11 shows the screenshot of the leaderboard.
Figure 12 illustrates the precision-recall curves along with AP scores on different categories of the
KITTI test set.
The samples of the KITTI test set are quite different from those of training/validation set in terms of scenes and camera parameters, so the impressive performance of our BiProDet on the test set demonstrates it also achieves good generalization.