DeepInteraction: Exploring Multi-modal Interaction for 3D Object Detection
Anonymous Author(s)
AffiliationAddress email
Abstract
Existing multi-modal 3D object detectors typically consider a unilateral associationstrategy with a biased inclination on 3D LiDAR point clouds whilst treating the2D multi-camera images as an auxiliary information source.
As a result, thoseuseful high-resolution information unique with the images is rigidly thrown awayin modality association.
In essence, the intrinsic complementary nature between thetwo modalities is fully overlooked by prior arts.
In this work, we introduce a novel3D object detection architecture, dubbed as DeepInteraction , characterized bybilateral interaction and association throughout both representation encoding anddecoding, in order to maximally exploit the inter-modal complementary property.
Extensive experiments verify the accuracy superiority of DeepInteraction overthe state-of-the-art methods by large margin on the large scale nuScenes benchmark.
Introduction
[Figure]
is always preferred the modalities are
3D object detection is critical for autonomous driving by localizing and recognizing decisionsensitive objects in the 3D space.
For reliable object detection, LiDAR and camera sensors havebeen systematically deployed to provide low-resolution point clouds with depth information and highresolution visual images, respectively.
The two modalities exhibit naturally strong complementaryeffects due to such different perception characteristics.
However, it is non-trivial to associate andjointly exploit both modalities due to the intrinsic sensor misalignment challenge [38].
As highlightedin the red boxes of Figure 1, one modality would be favored over the other depending on whichunconstrained driving conditions.
Neither modality is always preferred throughout, which meansthe two modalities are strongly complementary.
As a specific example, due to the low-resolutionnature with LiDAR sensor, small objects will be significantly under-represented with only one point
[Figure]
or very few points; In this case, the coupled image will be useful by offering more visual appearanceinformation thanks to its high-resolution property.
Existing top-performing multi-modal 3D object detection methods fall into two categories.
The firstcategory of methods ( e.g. , PointPainting [29] and PointAugmentation [30]) decorate the LiDARpoints at the input level (Figure 2(a)).
Specifically, the image features are used to enrich the lowresolution point cloud information ( e.g. , shape and depth).
Such decorated point clouds are limitedin improving the performance due to the big inter-modality discrepancy.
To address this issue, thesecond category of methods ( e.g. , TransFusion [1]) take a feature-level fusion paradigm (Figure 2(b)).
Technically, this strategy maximizes the correlation of LiDAR and camera data in the representationspace.
Commonly, both paradigms treat the coupled images as an auxiliary modality with a biasedinclination on the LiDAR modality.
As a consequence, those useful information unique with the 2Dimage data is simply thrown away during the unilateral fusion step.
That being said, the intrinsiccomplementary property of the two modalities as clearly exhibited in the raw data is not properly andfully leveraged.
To overcome the aforementioned architectural limitations of existing solutions, in this work we introduce a novel multi-modal 3D object detection architecture capable of conducting bilateral interactionbetween 3D point clouds and multi-camera visual images (Figure 1), dubbed as DeepInteraction .
Specifically, the whole pipeline of DeepInteraction has three stages: (1)
In the feature extractionstage, 3D point clouds and 2D multi-view images are encoded into intermediate representationsusing two modality-specific encoders, respectively; (2)
In the encoding stage, the two representationsintensively are interacted and fused with each other with a multi-modal encoder to build more expressive embedding; (3)
In the decoding stage, these embeddings are further alternatively aggregatedwith a multi-modal dynamic decoder for generating predictions.
In design, the two modalities areassociated and interacted in both directions so that modality-specific strengths can be fully leveragedfor maximally capitalizing their complementary effects.
The contributions of this work are summarized as follows: (i) With in-depth assessment into themodel design, we reveal an architectural limitation of existing multi-modal 3D object detectionmethods that multi-camera 2D images with fine visual appearance information is only exploitedunilaterally as an auxiliary modality.
This ends up with insufficient exploitation of the multi-modalityperception data coming with intrinsically strong complementary nature.
(ii) To solve this limitation,we introduce a novel DeepInteraction architecture based on bilateral interaction and associationacross modalities, through both representation encoding and decoding in both directions.
(iii)
Extensive experiments on the nuScenes dataset show that our DeepInteraction yields new state ofthe art for multi-modality 3D object detection, often with a large margin over the alternatives.
Related work
3D object detection with single modality Automated driving vehicles are generally equippedwith both LiDAR and multiple surround-view cameras.
Many previous methods preform 3D object detection by exploiting data captured from only a single form of sensors.
For camera-based 3Dobject detection, since depth information is not directly accessible from RGB images, some previousworks [17, 31, 27] lift 2D features into a 3D space by conducting depth estimation, followed byperforming object detection in the 3D space.
Another line of works [32, 23, 21] resort to thedetection Transformer [5] architecture.
They leverage 3D object queries and 3D-2D correspondenceto incorporate 3D computation into the detection pipelines.
Despite the rapid progress of camerabased approaches, the state-of-the-art of 3D object detection is still dominated by LiDAR-basedmethods.
Most of LiDAR-based detectors quantify point clouds into regular grid structures such asvoxels [39, 35], pillars [20] or range images [2, 15, 7] before processing them.
Due to the samplingcharacteristics of LiDAR, these grids are naturally sparse and hence fit the Transformer design.
So,a number of approaches [24, 14] have applied the Transformer for point cloud feature extraction.
Differently, several methods use the Transformer decoder or its variants as their detection head [1, 33].
In contrast, we present an encoder-decoder architecture as an unified framework for both featureinteraction and prediction.
Multi-modality fusion for 3D object detection Leveraging the perception data from both cameraand LiDAR sensors typically leads to improved performance, which hence emerges as a promisingdirection.
Existing 3D detection methods typically perform multi-modal association/fusion at one ofthe three points: raw input, intermediate feature, and object proposal.
For example, PointPainting [29]is the pioneering input fusion method [30, 18, 12].
The main idea is to decorate the 3D point cloudswith the category scores or semantic features from the 2D instance segmentation network.
Whilst4D-Net [25] placed the fusion module in the point cloud feature extractor for allowing the point cloudfeatures to dynamically attend to the image features.
The proposal based fusion methods [19, 8]keep the feature extraction of two modalities independently and aggregate multi-modal features viaproposals or queries at the detection head.
The first two categories of methods take a unilateral fusionstrategy with bias to 3D LiDAR modality due to the superiority of point clouds in distance and spatialperception.
Instead, the last category fully ignores the intrinsic association between the two modalitiesin representation.
As a result, all these previous methods fail to fully exploit both modalities withstrong complementary nature.
In this work, we address this limitation with all previous solutions bydeveloping a bilateral interaction and association based architecture for exploring the inter-modalcomplementary effect.
DeepInteraction
We adopt DETR [6] as the backbone architecture due to no need of hand-crafted components likeanchor generation and non-maximum suppression.
At the presence of multi-modal inputs, extendingthe standard DETR, originally designed for single-modal 2D object detection, becomes non-trivial.
This is primarily due to an extra challenge of reliably associating the 3D LiDAR points and 2D imagepixels without perfect correspondence.
We hence introduce a novel bilateral association framework( DeepInteraction ) for multi-modal 3D object detection, characterized by deep interaction between3D point clouds and 2D multi-camera images.
In contrast to all prior arts, we treat the 3D LiDARand 2D camera modalities equally and conduct multi-modal interaction through both representationencoding and decoding.
An overview of our DeepInteraction is shown in Figure 3(a).
In the following, we will first provide the preliminary of 3D object detection with Transformers in
Section 3.1.
We then describe how to perform multi-modal interaction in both the encoder (Section3.2) and decoder (Section 3.3).
3.1 Preliminary: 3D object detection with Transformers
Given multi-modal input including 3D LiDAR point clouds x p and surrounding 2D RGB imagesx c , the model need to predict a set of 3D bounding boxes along with category labels ˆ y .
We startwith the context representation extraction per modality: h p = f ψ p ( x p ) for 3D point cloud by a3D backbone and h c = f ψ c ( x c ) for 2D image by a 2D backbone. Adopting the original structure
[Figure]
of the Transformer , our model consists of two modules: (i) the multi-modal encoder, and (ii) themulti-modal decoder.
Taking the context representations h p and h c as input, the encoder outputs anenhanced multi-modal embeddings:
[Equation]
where A ( · ) is an association function that combines the heterogeneous inputs from two modalities.
This A ( · ) can be instantiated as addition, concatenation, or a learnable module such as the multi-headcross-attention as in Transfusion [1].
The decoder needs to make the final predictions by taking themulti-modal representations h as input:
[Equation]
Following DETR, the optimal bipartite matching is computed for matching the set of predictionsˆ y with the ground-truth y annotations.
During optimization, the model is trained to minimize thematching cost.
Remarks As described above, the multi-modal association in Transfusion [1] by design is unilateral,conducted before the encoder input (Eq. 1).
Thus, the LiDAR-camera data is not fully interacted andexploited for final prediction (Eq. 2).
Instead of employing a stand-alone and unilateral association,we formulate A ( · ) within both encoder and decoder, so that the two modalities can interact deeplyfor maximally capitalizing their complementary nature.
3.2 Multi-modal interaction in encoding
Multi-modal bilateral mapping To enable multi-modal interaction in the encoder, we first need tobuild the correspondence relationship between the context representations h p and h c .
To that end,we build a bilateral mapping between the 3D LiDAR point cloud and 2D camera images: M p → c and
M c → p , to locate and sample the spatially corresponding features between the two modalities.
For the 3D LiDAR to 2D camera mapping M p
→ c (Figure 4(a)), we first obtain the point coordinates{ ( x, y, z ) } of a pillar.
Then, we project the 3D locations into a set of pixel coordinates { ( i, j ) } onthe corresponding image views, according to the extrinsic and intrinsic camera matrices.
Also, thebilinear interpolation is used to sample 2D image features { h c ( x, y, z ) } located at { ( i, j ) } for thepillar from multi-view image context representations h c .
For the 2D camera to 3D LiDAR mapping M c → p (Figure 4(b)), we project each point ( x, y, z )
on3D point cloud to the spatially corresponding image location ( i, j ) to obtain the sparse depth mapsd sparse .
Then, we complete sparse depth maps d sparse into dense depth map d dense and project thedepth to 3D coordinates ( x, y, z ) , which is used to locate a BEV feature pillar h p ( i, j ) at the locationof ( x, y, z ) from h p for the 2D image location ( i, j ) .
All located BEV feature pillars are reorganizedinto a 2D shape.
[Figure]
Multi-modal feature interactions We aim to construct multilateral interaction, enabling information to flow in different modality.
We introduce inner-modal interaction ( i.e. , f ϕ p → p and f ϕ c → c ) toexplore the intrinsic connections of the inner-modal modalities themselves.
Besides, to associate thespatial and semantic information on the same object, we also design the cross-modal interaction ( i.e. ,f ϕ c → p represents the interaction from image to point clouds and f ϕ p → c vice versa).
In the following,we detail the four multi-modal feature interactions (Figure 3(b)).
For all types of interaction module, we use local attention [26] to capture the local correlation.
Thiscan be formulated analogously as:
[Equation]
where q is the query feature and N q denotes a group of features in the spatial neighborhood of q .
The inner-modal interaction of point clouds ( h p → p p = f
ϕ p
→ p ( h p ) ) and images ( h c → c c = f ϕ c → c ( h c ) )allow interaction within the neighboring feature pixels using Eq. 3.
In this case, we define N q asthe inner-modal features in 2D spatial k × k neighborhood of query feature pixel, where k = 9 .
Wetransform each pixel of input feature into q , and transform N q into k and v .
In doing so, the finalfeatures that are weighted by the attention map ( i.e. , h p → p p and h c
→ c c ) are more likely to focus onlocally important information.
Cross-modal interactions take the same spirit of locality, with a slight modification on the definitionof N q in Eq. 3.
For the interaction from point clouds to images ( h p → c c = f ϕ p → c ( h p , h c ) ), the LiDAR BEV featuremap h p is first reorganized into the same form with multi-view image feature maps h c via 3D LiDARto 2D camera mapping M p
→
c .
Then, for each query q = h c
[ i, j ] in Eq.
3, the neighborhood N q is aset of feature pixels within a square patch centered around ( i, j ) on reorganized LiDAR BEV featuremap, with spatial extent k = 9 .
On the contrary, the interaction from images to point clouds ( h c → p p =
f ϕ c → p ( h c , h p ) ) is proposedto spread visual signals from multi-view image feature maps h c to LiDAR BEV feature map h p .
Specifically, given q = h p [ i, j ] as query, its neighborhood N q is defined as a group of image featuressampled by M c → p .
In the above processes, four interacted multi-modal feature maps ( i.e. , h p → p p , h c → c c , h c → p p , and h p → c c )are generated by multi-modal feature interactions described above respectively.
Then, a pair of MLPlayers (denoted as MLP ( · ) in Eq. 4,5) are conducted in succession to fuse the augmented featuresbefore and after interactions into the final enriched representation (Figure 3(b)).
The computation ofeach encoder layer can be formulated as:
[Equation]
Furthermore, we can construct more expressive representations by stacking more encoder layers,where each subsequent layer takes the multi-modal enriched representations h ′
MHSA
Add & Norm
Dync.
FFN
[Figure]
CLS
[Figure]
if is odd if is even
3.3 Multi-modal interaction in decoding
After obtaining the interacted representations h ′ p and h ′ c for LiDAR and camera, we further considerto maximize the complementary effect in the decoder.
As single-modal representations alone may notcover enough useful information for making robust predictions.
Therefore, we propose a multi-modalalternate query refinement strategy to perform multi-modal interaction in the decoder.
Multi-modal alternate query refinement As illustrated in Figure 3(c)
, the proposed multi-modaldecoder contains N alternate decoder layers which iteratively refine the bounding box predictions byaggregating features from two modalities.
Specifically, we follow Transfusion [1] to obtain the initialqueries q of decoder from combining LiDAR interacted features h ′ p and original LiDAR contextfeatures h p to speed up convergence.
To enable more flexible interaction between modalities, weemploy the dynamic instance interaction (Dync.)
head [28] as the decoder interaction layer, as shownin Figure 5, in which we compute the predictions by alternately taking the interacted representationsh ′ p and h ′ c of two modalities as input.
In doing so, the decoder queries q are refined iteratively byfully exploiting both LiDAR and camera information.
3.4 Optimization
During training, we utilize a set-to-set loss [6] between predicted and ground-truth objects:
[Equation]
(6) where σ ( · ) denotes the optimal bipartite matching between y and ˆ y found by Hungarian algorithm,
L cls is the focal loss for classification, and L reg is the ℓ 1 loss for bounding box regression.
Thematching cost L match used for computing optimal assignment σ ( · ) is the weighted sum of L cls , L regand the negative 3D IoU.
We also compute the auxiliary loss with the same form as Eq.
6 for thepredictions of each layer to stabilize convergence.
In addition, we provide direct supervision forheatmap prediction in query initialization by adding an extra Guassian Focal Loss [36].
Experiments
4.1 Experimental setup
Dataset We evaluate our approach on the nuScenes dataset [3].
It provides 32-beam point cloudsfrom LiDAR and images with resolution of 1600 × 900 from 6 surrounding cameras.
The total of1000 scenes, where each sequence is roughly 20 seconds long and annotated every 0.5 second, isofficially split into train/val/test set with 700/150/150 scenes.
For the 3D object detection task,1.4M objects in scenes are annotated with 3D bounding boxes and classified into 10 categories: car,truck, bus, trailer, construction vehicle, pedestrian, motorcycle, bicycle, barrier, and traffic cone.
Wefollow the common practice to transform points from the past 9 sweeps to the current frame.
Metric Mean average precision (mAP) [13] and nuScenes detection score (NDS) [3] are used asthe evaluation metric of 3D detection performance.
The final mAP is computed by averaging over
[Table]
[Table]
distance thresholds of 0.5m, 1m, 2m, 4m across 10 classes.
NDS is a weighted average of mAP and
[Table]
other attributes metrics, including translation, scale, orientation, velocity, and other box attributes.
4.2 Implementation details
Our implementation is based on the public code base mmdetection3d [11].
For image branch, weinitialize our backbone and FPN [22] from the instance segmentation model Cascade Mask R-CNN [4]with resnet-50 [16] pretrained on nuImage [3].
To save the computation cost, we rescale the inputimage to 1/2 of its original size before feeding into the network, and freeze the weights of imagebranch during training.
The voxel size is set to (0 .
075 m, 0 . 075 m, 0 .
2 m ) , and the detection range isset to [ − 54 m, 54 m ] for for X and Y axis and [ − 5 m, 3 m ] for Z axis.
For data augmentation, we follow TransFusion [1] and adopt random rotation with a range ofr ∈ [ − π/ 4 , π/ 4] , random scaling with a factor of r ∈ [0 . 9 , 1 . 1] , random translation with standarddeviation 0.5 in three axis, and random horizontal flipping.
We also use the class-balanced resampingin CBGS [40] to balance the class distribution for nuScenes.
We use TransFusion-L [1] as our
LiDAR-only baseline.
For training schedule, we use Adam optimizer with one-cycle learning ratepolicy, with max learning rate 1 × 10 − 3 , weight decay 0.01 and momentum 0.85 to 0.95, following
CBGS [40].
Our LiDAR-only baseline is trained for 20 epochs and LiDAR-camera fusion for 6epochs with batch size of 16 using 8 NVIDIA V100 GPUs.
4.3 Comparison to the state of the art
We compare with state-of-the-art alternatives on the nuScenes test set.
We test single model withand without test-time augmentation respectively, as well as model ensemble.
Following the commonpractice [36] for model ensemble we merge the results from multiple models with voxel size of 0.05m,0.075m, 0.1m, 0.125m and 0.15m. As shown in Table 1, our model achieves 70.8% mAP and 73.4%
NDS on the nuScenes test set without bells and whistles, outperforming alternative Transfusion [1]by 1.9% mAP and 1.7% NDS.
Our model beats the closest rival Focals Conv-F [9] by a considerablemargin under the same configuration, which clearly verifies the effectiveness of our multi-modalinteraction approach.
To the best of our knowledge, DeepInteraction surpasses all the published
LiDAR-camera methods on the nuScenes 3D object detection test set leaderboard.
[Table]
4.4 Ablation studies
3D Backbone As shown in Table 3e, we examine our framework’s compatibility with different 3Dbackbones: PointPillars [20] and VoxelNet [39].
For PointPillars backbone, we set the voxel size to(0.2m, 0.2m) while keeping all the other settings the same as other experiments.
To be fair, we havekept the same number of queries as TransFusion [1].
Benefiting from deep feature interactions, ourmodel exhibits consistent improvements over LiDAR-only baseline methods (see Figure 6) on bothbackbones (by 5.5% mAP for voxel-based backbone, and 4.4% mAP for pillar-based backbone).
Thisresult demonstrates the proposed DeepInteraction is generic to different point cloud encodingmanners.
Where does the improvement come from?
To investigate exactly where these improvements comefrom, we conduct extensive experiments on the nuScenes val dataset to test the effectiveness of eachcomponent.
From Table 2, we can see that the alternative interaction in the decoder brings a gain of0.7% mAP and 0.5% NDS compared to the lidar-only baseline [1].
Further gains of 3.5% mAP and2.7% NDS are achieved by the bidirectional multi-modal interactions before detection.
This suggeststhat both our bilateral interaction encoder and multi-modal interaction decoder are helpful, and thatinteraction in the early stage (encoder) appears more critical than those in the decoder.
Bilateral interaction For an in-depth understanding of bilateral interaction encoder, we present aperformance comparison about inner-modal interaction (denoted as ‘Inner-Int.’) and cross-modalinteraction (denoted as ‘Cross-Int.’) in Table 3a.
With only inner-modal interaction, there is aperformance improvement of 1.7% mAP and 1.2% NDS.
When only cross-modal interaction isintroduced, the performance improves significantly by 3.1% mAP and 1.8% NDS.
With both innermodal and cross-modal interactions are applied, our method achieves state-of-the-art performancewith 69.9% mAP and 72.7% NDS.
[Table]
[Table] [Table]
[Table] [Table]
PointPillars [20]
L
[Table]
+ CenterPoint
L
[Table]
[Table]
+ Transfusion-L
L
[Table]
+ Transfusion [1]
L+C
[Table]
+ DeepInteraction
L+C
[Table]
VoxelNet [39]
L
[Table]
+ CenterPoint
L
[Table]
+ Transfusion-L
L
[Table]
+ Transfusion [1]
L+
[Table]
+ DeepInteraction
[Table]
[Table]
Alternative interaction We study the impact of multi-modal interaction in the decoder.
As shown in
Table 3d, increasing the number of decoder layers can consistently improve the performance exceptthat 6 layers experience a little performance drop.
Table 3b shows the comparison between differentinteraction candidates, including Transformer decoder layer [1] and dynamic instance interaction.
The second row of Table 3b presents the mixing strategy, i.e. , using Transformer decoder layer foraugmented LiDAR BEV features and Dync. for the augmented image features, which leads to aboost of 0.7% mAP compared with the pure Transformer counterparts.
The best result comes fromusing Dync. for both modalities.
This suggests that our multi-modal decoder with dynamic instanceinteraction is capable of exploiting the local context for specific object query.
Since object queriesare non-parametric and input-dependent [1], we can modify the number of queries during inference.
In Table 3f, we report the results under different combinations of query numbers in the training andtesting.
As seen, the best performance is achieved when training the model with 200 queries andtesting with 300 queries.
Conclusion
In this work, we have presented a novel 3D object detection architecture DeepInteraction forexploring the intrinsic inter-modal complementary nature.
This key idea is to conduct bilateralinteraction and association across modalities in representation learning.
This strategy is designedparticularly to resolve the fundamental limitation of existing alternatives that 2D multi-camera imagesare insufficiently exploited due to their auxiliary source role treatment.
Extensive experiments on thenuScenes benchmark demonstrate that DeepInteraction surpasses the state-of-the-art methods bya large margin, yielding new state of the art.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 4.
(b) Did you describe the limitations of your work?
[Yes] See supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seesupplementary material.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you
used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]