Unifying Voxel-based Representation with Transformer for 3D Object Detection
Abstract
In this work, we present a unified framework for multi-modality 3D object detection, named UVTR.
The proposed method aims to unify multi-modality representations in the voxel space for accurate and robust single- or cross-modality 3D detection.
To this end, the modality-specific space is first designed to represent different inputs in the voxel feature space.
Different from previous work, our approach preserves the voxel space without height compression to alleviate semantic ambiguity and enable spatial connections.
To make full use of the inputs from different sensors, the cross-modality interaction is then proposed, including knowledge transfer and modality fusion.
In this way, geometry-aware expressions in point clouds and context-rich features in images are well utilized for better performance and robustness.
The transformer decoder is applied to efficiently sample features from the unified space with learnable positions, which facilitates object-level interactions.
In general, UVTR presents an early attempt to represent different modalities in a unified framework.
It surpasses previous work in single- or multi-modality entries.
The proposed method achieves leading performance in the nuScenes test set for both object detection and the following object tracking task.
Code is made publicly available at https://github.com/dvlab-research/UVTR.
Introduction
Detecting 3D objects with multi-modality sensors ( i e., LiDAR and camera) is regarded as a fundamental task in real-world scenes.
For accurate object detection, data from different modalities are utilized to provide complementary knowledge, like accurate positions from point clouds and rich context from images.
Toward this purpose, a unified representation is essential to facilitate knowledge transfer and feature fusion across modalities.
However, due to the lack of accurate depth from cameras, images can not be naturally represented in voxel space like that of point clouds.
In the unified progress several representations have been studied that can be roughly separated into input- and feature-level streams.
For the first one, multi-modality data is aligned at the beginning of network.
In particular, pseudo point clouds in Figure 1a are transformed from image aided by predicted depth [1, 2], while the range-view image in Figure 1b is projected from point clouds [3, 4].
Because of inaccurate depth in pseudo point clouds and collapsed 3D geometry in range-view images, the spatial structure of data is damaged, which brings inferior results.
For feature-level method, a typical approach is to transform image features as frustum and then compress to BEV space [5, 6], like that in Figure 1c.
However, due to ray-like trajectories in the frustum [7], height compression at each position aggregates features from various objects and thus introduces semantic ambiguity.
Meanwhile, other implicit manners in contemporary work [8, 9, 10] can hardly support explicit feature interactions in 3D space and restrict further knowledge transfer.
Therefore, a more unified representation is desired to bridge modality gap and facilitate interactions from multiple aspects.
predicteddepth
Image
Pseudo Point Cloud * + 3D Conv
Backbone frustrumfeature
ImageFeature
BEV Feature Space * 2D Conv
CNN-basedHead samplewithprob
ImageFeature
Voxel Feature Space * + value
TransformerDecoder (-,/,0)
Range View Point Cloud * + 2D Conv
Backbone project position (a) Pseudo point cloud transformed from image predicteddepth
Image
Pseudo Point Cloud * + 3D Conv
Backbone frustrumfeature
ImageFeature
BEV Feature Space * 2D Conv
CNN-basedHead samplewithprob
ImageFeature
Voxel Feature Space * + value
TransformerDecoder (-,/,0)
Range View Point Cloud * + 2D Conv
Backbone project position (b) Range view projected from point cloud predicteddepth
Image
Pseudo Point Cloud * + 3D Conv
Backbone frustrumfeature
ImageFeature
BEV Feature Space * 2D Conv
CNN-basedHead samplewithprob
ImageFeature
Voxel Feature Space * + value
TransformerDecoder (-,/,0)
Range View Point Cloud * + 2D Conv
Backbone project position
[Figure]
feature predicteddepth
Image
Pseudo Point Cloud * + 3D Conv
Backbone frustrumfeature
ImageFeature
BEV Space * 2D Conv
CNN-basedHead samplewithprob
ImageFeature
Image Voxel Space * + value
TransformerDecoder (-,/,0)
Range View Point Cloud * + 2D Conv
Backbone project position
[Figure]
space by sampling features from the image plane and represents multi-modalities uniformly without height-level compression in 1c that brings semantic ambiguity.
In this paper, we present a simple yet effective framework to unify the voxel-based representation with transformer, called UVTR.
In particular, features from images and point clouds are represented and interacted in the explicit voxel-based space.
For images, we construct the voxel space by sampling features from the image plane according to predicted depth scores and geometric constraints, as briefly depicted in Figure 1d.
For point clouds, the accurate position naturally allows us to associate features with voxels.
Then, voxel encoder is introduced for spatial interaction that establishes the relationship among adjacent features.
In this way, cross-modality interaction is naturally conducted with features in each voxel space.
For object-level interaction, deformable transformer [11] is adopted as the decoder that samples specific feature for each object query with position ( x, y, z ) in the unified voxel space, as illustrated in Figure 1d.
Meanwhile, the introduction of 3D query position efficiently alleviates the semantic ambiguity brought by height compression in BEV space as analysed before.
Compared with previous and even concurrent studies [8, 9], more key advances can be achieved with the proposed framework.
First , the explicit voxel-based representation supports spatial interaction in 3D space and multi-frame scenes that bring significant improvements.
Second , the proposed unified manner facilitates cross-modality learning and can be naturally applied for knowledge transfer and feature fusion, which further boosts the performance.
Finally , data augmentation for both modalities can be directly synchronized in the voxel space without the complex aligning process [12, 7].
The overall framework, called UVTR, can be easily instantiated and improved with various imageor voxel-based backbones for single- and multi-modality 3D object detection.
Extensive empirical studies are conducted in Section 4 to reveal the effect of each component.
The proposed UVTR attains leading performance in various settings.
For detection, it achieves 69.7 %, 55.1 %, and 71.1 %
NDS on nuScenes test set with point clouds, images, and multi-modality inputs, respectively.
Given naive association strategy, UVTR also achieves strong tracking results with 67.0 %, 51.9 %, and 70.1 %
AMOTA on LiDAR-based, camera-based, and multi-modality setting, respectively.
Related Work
LiDAR-based 3D Detection.
With point clouds captured from LiDAR, traditional methods process the irregular input and generate 3D boxes with different representations, e.g., point, voxel, and range view.
Point-based detectors usually aggregate features from raw point clouds with set abstraction [13] and then predict box proposals [14, 15, 16].
For voxel-based methods, point clouds are transformed into regular grids and processed with 3D sparse convolutions [17, 18] or 2D convolutions [19, 20, 21] directly.
Final predictions are usually generated on top of the bird-eye view (BEV) space with the flatted height axis [22, 23, 24].
There are also studies [3, 4] that project point clouds to range view and process them like images.
However, due to the collapsed 3D geometry in range-view images,
TransformerDecoder
Unified Voxel Space (",$,%) ' ( ) value position
[Figure] [Figure]
Backbone
Point VoxelSpace
VoxelEncoder VoxelEncoder
KnowledgeTransfer
Modality Switch
ObjectQueries
Self Attention
Cross Attention
FFN
Add&Norm ×+ ObjectDetection
Add&Norm
Add&Norm X !
X "
F !
F "
V !
V "
V #
V !
$
V "$
Q the relationship in point clouds cannot be fully explored.
In this work, we follow the voxel-based pipeline but keep the fine-grained voxel space without height compression, as shown in Figure 2.
Camera-based 3D Detection.
Camera-based methods perform 3D detection on single- or multi-view images.
With monocular image, previous approaches try to predict 3D boxes based on image features directly [25, 26, 27] or utilize the middle representation [1, 2, 5].
For multi-view input, image features are usually optimized in the constructed 3D geometry volume [28, 29].
Most recently, multi-view features are projected and merged in the frustum feature space with the aid of predicted depth [6].
Following the LiDAR-based paradigm, the frustum feature is collapsed to the BEV space, as briefly introduced in Figure 1c.
However, the accuracy of the predicted depth map is much inferior to that of
LiDAR, which brings semantic ambiguity to BEV space.
Other recent studies try to capture geometry clues from multi-view images in an implicit manner [8, 10], which losses the chance for direct spatial interactions.
In this paper, we represent image features in an explicit voxel space to alleviate the semantic ambiguity and facilitate further feature interactions, as depicted in Figure 1d.
Cross-modality Interaction.
With input data from various sensors, cross-modality interaction is conducted to benefit from different inputs, e.g., modality fusion and knowledge transfer.
For modality fusion, the model takes data from different sensors and conducts fusion at point- and instance-level.
Specifically, point-level fusion [30, 31, 32, 7] combines features from different modalities at the early stage of the network, which enables sufficient interaction.
And instance-level fusion [33, 34, 35] is usually applied at the later stage to combine object-level features.
Cross-modality knowledge transfer aims to distill specific knowledge [36] across modalities in the training phase.
Compared with cross-modality fusion, knowledge transfer is seldom studied for 3D object detection.
A prior work is
LIGA-Stereo [37] that transfers geometry-aware representations from LiDAR to stereo images via distillation.
Different from [37], UVTR represents each modality in a unified manner and supports cross-modality fusion and knowledge transfer simultaneously, which further enables distillation from multi-modality or consecutive frames to the single input.
UVTR Framework
The overall framework of UVTR is relatively simple: modality-specific space is constructed to unify the representation of inputs; cross-modality interaction is designed for feature learning across spaces; and transformer decoder is introduced for object-level interaction and final prediction.
3.1 Modality-specific Space
Given images X I captured from cameras and point cloud X P from LiDAR, different branches are utilized to respectively generate and enhance voxel space for each modality, as presented in Figure 2.
samplewithprob
Depth Distribution
F !
D !
F ! *
+
V !
Image Voxel Space
V !
Conv
BN
ReLU
Voxel Encoder ……
V !
$
V "
Conv
BN
ReLU
Voxel Encoder ……
S !
T " (',),*) ℒ &' KnowledgeTransfer
V "$ samplewithprob
Depth Distribution
F !
D !
F ! *
+
V !
Voxel FeatureSpace
V !
Conv
BN
ReLU
Voxel Encoder ……
V !
$
V "
Conv
BN
ReLU
Voxel Encoder ……
S !
T " (',),*) ℒ &' KnowledgeTransfer
V "$
Image Voxel Space.
For image voxel space, a shared backbone is adopted to extract features from multi-view or multi-frame images.
In this process, FPN [38] is utilized for multi-scale context aggregation that is summed to formulate the feature
F I ∈ R H
× W × C , where H and W vary with
FPN stages.
To construct the voxel feature for images, we then transform the image feature of each view to the predefined space with the designed view transform in Figure 3.
Motivated by [39, 5], we first generate the depth distribution D I ∈
R D
× H × W of each image with a single convolution as
[Equation]
Here, ( u, v ) indicates coordinate in the image plane, and D is set to 64 to represent the perception limit 64 m .
It is noted that D I is predicted without supervision.
With the predicted D I in D depth bins, we can easily get the depth distribution of each pixel in F I .
Let ( x, y, z ) indicates a sampling point that is generated at the center of each bin from the voxel space V I .
The point ( u, v, d ) in the image plane is calculated from ( x, y, z ) with the calibration matrix P , where d denotes the reference depth along axis D of D I .
Thus, the corresponding feature in voxel space V I is easily captured by
[Equation]
where D I ( u, v, d ) represents the occupancy probability of feature F I ( u, v ) in voxel ( x, y, z ) .
For the multi-frame setting with n sweeps, we use the shared network for all of them and formulate n voxel spaces in total.
In this process, each calibration matrix P is aligned to the ego vehicle in the initial frame.
To gather temporal cues in each voxel space, relative time offsets from the initial frame are attached along the channel axis and merged using a single convolution.
Then, n voxel spaces are concatenated together, and the space-level fusion is conducted with a convolutional layer.
In this way, features along the temporal dimension are integrated into a unified space
V I , which is proved to bring significant gain in Table 3.
Different from methods [5, 6] for BEV space, we preserve the 3D voxel space without collapsing in Z axis to avoid the aforementioned semantic ambiguity and enable further interactions.
The effectiveness of the 3D voxel space is empirically studied in Table 1.
Point Voxel Space.
With the accurate position, we naturally split point cloud X P into several regular voxels.
Then, the voxel backbone in Figure 2 is utilized to process input voxels with sparse convolution [17].
To enhance multi-scale features in the generated voxel space, parallel heads with various strides are designed to extract feature F P from the output.
In particular, several 2D convolutions are applied in each head to aggregate the spatial cues at each height.
Then, multi-scale features are upsampled to a same resolution and summed together to formulate the voxel space
V P ∈ R X × Y ×
Z × C .
For multi-frame setting with n sweeps, we follow previous work [24] and attach all point clouds together with relative time offsets to formulate the input X P .
Due to the accurate position of point cloud, the semantic ambiguity in Z axis is much reduced compared with that of images.
But we still preserve the 3D space V P without height compression for convenient cross-modality interaction in Section 3.2 and fine-grained object interaction in Section 3.3.
This is also proved to bring superior experimental results in Table 1.
Voxel Encoder.
In the above-generated space V I , features of adjacent voxels projected from different views have no connection with each other.
To solve this issue and facilitate local feature interaction, the voxel encoder is proposed in each voxel space, as presented in Figure 2.
Specifically, we keep the simplicity of UVTR, and only three basic convolutional blocks are applied in each voxel encoder of Figure 4.
In this process, features in each space V I or V P are aggregated in both coplanar and vertical dimensions.
The spatial interaction in voxel space establishes connections among adjacent features, which is proved to be essential in Table 2, especially for V I .
3.2 Cross-modality Interaction
With the unified representation in space V I and V P , interactions across modalities can be easily conducted.
Given the prior that LiDAR is advanced in localization and cameras provide context for classification, the cross-modality interaction is proposed from two separate aspects, i.e. , transferring geometry-aware knowledge to images in a single-modality setting and fusing context-aware features with point clouds in a multi-modality setting.
In particular, knowledge transfer aims to optimize the features of the student with guidance from the teacher in the single-modality setting.
Meanwhile, modality fusion is designed to better utilize all modalities in both training and inference stages.
Knowledge Transfer.
Considering single modality input in the inference stage, knowledge transfer is first designed to optimize features of the student with guidance from the teacher during training, which is important in an environment that lacks multi-modality data.
Due to inherent properties, the geometry structure contained in images can be further exploited with the aid of point clouds, while the rich context in images can hardly be transferred to sparse point clouds.
Therefore, we mainly focus on transferring knowledge from the geometry-rich modality to the poor one in this work.
Benefiting from unified feature spaces, the cross-modality transfer can be easily supported, as illustrated in Figure 4.
In particular, we take features before the last ReLU layer in the voxel encoder of V P as the geometry-rich teacher, marked as T P .
Meanwhile, the feature in the same position of V I is taken as the geometry-poor student, denoted as S I .
If we take one object query position ( x, y, z ) from Section 3.3, the feature distance for knowledge transfer is formulated as
[Equation]
where PL 2 represents the partial L 2 distance [40].
Without bells-and-whistles, the optimization objective for knowledge transfer is averaged from N object queries of transformer decoder in
Section 3.3, namely L KT = 1 N (cid:80) i ( d KT ) .
It should be noted that the whole network is optimized in an end-to-end manner, with no need for extra procedures.
Given the object position in each query, we can directly minimize the object-level distance with no need to exclude background features like [37].
In a similar pipeline, the knowledge transfer is further extended to support more input streams, like multi-frame images.
The proposed cross-modality knowledge transfer is flexible with input modalities and brings consistent gains over various baselines in Tables 5 and 7.
Modality Fusion.
Different from the knowledge transfer, modality fusion aims to better utilize all modalities in both training and inference stages, which utilizes the complementary knowledge of point cloud and images to improve the performance and robustness.
Thanks to the unified representation of each modality, feature fusion can be naturally applied.
To be specific, given the processed feature space V ′
I and V ′ P , we first select candidate modality for final prediction via modality switch, as depicted in Figure 2.
That means we support single- or multi-modality input for prediction according to different settings.
If both modalities are taken, V ′ I and V ′ P are added together to formulate the unified voxel space V U ∈ R X
× Y ×
Z × C .
In this way, both modalities are well expressed in a unified manner, which can be further fused with a single convolution.
The space V U unifies modalities with the explicit representation, which provides an expressive space for object interactions in Section 3.3.
3.3 Transformer Decoder
To obtain accurate and robust predictions, the transformer decoder is utilized for further object-level interaction in the unified voxel space V U .
We draw inspirations from deformable DETR [11] and apply reference positions to efficiently sample representative features, regardless of the spatial size of 3D voxel spaces.
In particular, we first initialize N object queries Q ∈ R N × C and generate
N reference points from object query embedding.
Then, object queries are interacted with each other in the self-attention module and summed via skip-connection, as shown in Figure 2.
Let q represents a specific query in Q with corresponding reference point p
= ( x, y, z ) .
The process of the cross-attention module in Figure 2 is modeled as
[Equation]
where V U ( p ) denotes the sampled feature at ( x, y, z ) of V U , and DeformAttn indicates the deformable attention in [11].
With the feed-forward network and normalization, each object query can easily interact with unified features from V U inside each block.
There are total M blocks in the transformer decoder, where M is respectively set to 3 and 6 for LiDAR-based and camera-based settings.
Finally, a shared MLP head is utilized for prediction according to the output of each block.
And iterative box refinement [11, 8] is applied to refine 3D bounding boxes based on the predictions.
3.4 Optimization Objectives
Following a general paradigm in recent transformers [41, 11], Hungarian algorithm [42] is adopted for one-to-one target assignment in the training phase.
Thus, a set-to-set loss L Det is computed to optimize detection results, including box regression loss and classification loss.
If knowledge transfer is applied, the loss L KT is contained to reduce cross-modality feature distance with a weight 0.01.
Experiments
In this section, we first introduce our detailed experimental setup.
Then, analyses of each component are conducted on different modalities.
Comparisons with several leading benchmarks on the nuScenes [43] dataset are presented in the end.
More results are attached in supplementary material.
4.1 Experimental Setup
Dataset.
nuScenes [43] dataset is a large-scale benchmark for autonomous driving, which is widely adopted for single- or multi-modality 3D object detection.
It contains 700, 150, 150 scenes in the train , val , and test set, respectively.
We use the synced data with 10 object categories that are captured from a 32-beam LiDAR at 20Hz and six cameras in a 360-degree field of view at 12Hz.
Only annotations of keyframes are given at 2Hz.
Here, ablation studies are optimized on a mini 1/4 train split by default, and final models are optimized on the whole train set.
Implementation Details.
In this work, we conduct experiments on different modalities withobject queries Q .
Constructed voxel spaces V I , V P , and
V U share the same shape 128 × 128 × Z , where Z indicates the height of voxel space and is further investigated in Table 1.
The channel number
C in voxel spaces and transformer decoder is set to 256.
And the amount of block M in the decoder is set to 3, 6, and 6 for LiDAR-based, camera-based, and fusion settings, respectively.
In particular, for the LiDAR-based setting, only the branch with voxel space V P is kept.
With grid size 0 .
1 m , the input point clouds are filtered in range [ − 51 .
2 m, 51 .
2 m ] for X and Y axis with [ − 5 .
0 m, 3 . 0 m ] for
Z axis.
While for grid size 0 .
075 m , the range in X and Y axis is modified to [ − 54 .
0 m, 54 . 0 m ] .
The framework is trained with AdamW optimizer with an initial learning rate 2 e − 5 for 20 epochs.
For a camera-based setting, the network is optimized with an initial learning rate 1 e − 4 for 24 epochs.
As for fusion, we initialize two modality-specific branches with corresponding pretrained models and optimize the model with an initial learning rate 4 e − 5 for 20 epochs.
The whole framework is trained in an end-to-end manner with different modalities.
More details are given in supplementary material.
4.2 Component-wise Analysis
In this subsection , we use a randomly sampled 1/4 split of nuScenes train set for efficient validation.
Effect of Height in Voxel Space.
As elaborated in Section 3.1, the height axis
Z plays a vital role in voxel space, especially for camera-based V I .
In Table 1, we validate this with different heights on both modalities.
Compared with the BEV space with height 1, the increase in height contributes significantly for camera-based V I , which respectively improves 3.1% and 4.2% NDS with heightand 11.
Consistent with our analysis, the gain brought by increase of height contributes less to that of
LiDAR because of the accurate position, which is up to 1% NDS and 1.9% mAP.
Operations in Voxel Encoder.
The voxel encoder in Section 3.1 aims to facilitate spatial feature interactions that are essential, especially in the camera-based manner.
As presented in Table 2, camera-based network cannot converge if given no spatial interaction.
For LiDAR-based network, it performs satisfactorily without the voxel encoder, which can be attributed to the established relations at the early stage.
Overall, for both of them, 3D spatial interaction still plays a vital role and improves 2.6% and 0.6% NDS over 2D convolution for the camera- and LiDAR-based methods, respectively.
Effect of Multi-frame Input.
To further release the potential of the designed paradigm, we input multi-frame sweeps and represent them in each voxel space, as shown in Table 3.
With more input frames, networks for both modalities achieve consistent gains.
The performance gap reaches 5% and 18.1% NDS for the camera- and LiDAR-based manner with 5 and 10 sweeps, respectively.
Networks for Voxel Space.
In Table 4, we exploit the network for voxel space generation.
It is clear that the deeper network with larger voxel space contributes more to the final result.
With ResNet-
[Table] [Table]
[Table] [Table]
[Table] [Table]
and height 11 for space V
I , the camera-based method attains nearly 5% gains over the baseline in both NDS and mAP.
For LiDAR-based manner, the model performs slightly better with finer grid size that brings higher resolution to voxel space.
It means the image-based voxel space requires strong backbones to extract expressive features, while the LiDAR-based one is less dependent on that.
Knowledge Transfer.
Benefiting from the unified representation, knowledge can be easily transferred across modalities.
In Table 5, we compare combinations with different students and teachers.
In particular, the camera-based student captures geometry-aware cues from camera sweeps or the
LiDAR-based teacher, which brings up to 1.9% NDS gain.
If coupled with context features in the multi-modality setting, the gap is enlarged to 2.6% NDS and 1.8% mAP.
For the LiDAR-based student, the increase brought by knowledge transfer saturated with 0.6% NDS.
This can be attributed to that rich context in images can not be well expressed only with sparse points during inference.
Therefore, images are required as input if rich context is supplemented, like the following fusion part.
Cross-modality Fusion.
In Table 6, we validate capability of UVTR with cross-modality fusion.
As presented in the table, feature fusion brings significant gains over a single modality.
And the LiDARbased representation dominates the final results, while the camera-based one provides supplementary context.
It is reasonable because point clouds are more accurate in position and more representative in geometry expression.
Cameras still provide sufficient context for better classification, which yields up to 1.6% NDS and 3.9% mAP gain compared with the LiDAR-based baseline.
With finer voxel grids, the performance gap is enlarged to 2.5% NDS and 5.5% mAP.
[Table] [Table]
CenterPoint [24]
[Table]
4.3 Main Results
In this section
[Table]
we first report results with various modalities that are optimized on the whole train set.
Then, we give analyses of the framework robustness, including camera view drop and translational noise.
Comparisons with leading methods on the nuScenes test set are presented in the end
[Table]
Results with Different Modalities.
In Table
[Table]
we carry out experiments with different modalities on the nuScenes val set.
Compared with classic methods, UVTR achieves significant improvement.
In particular, for LiDAR-based method, it attains 1.5% NDS and 2.7% mAP gain over CenterPoint [24].
And a finer resolution contributes better results with 67.7% NDS.
For camera-based manner, UVTR-C performs better in single frame setting with 1.6% NDS gain over DETR3D [8].
If applied knowledge transfer in UVTR-L2C, the gap is enlarged to 2.5% NDS.
The performance improves with more frames and attains up to 48.8% NDS.
For multi-modality input, UVTR-M achieves 4.5% mAP gain over UVTR-L and outperforms the contemporary FUTR3D [9] with 1.9% NDS in a same setting.
Results in Different Conditions.
In Table 8, we report the performance with different distances, weather conditions, and light situations.
(1) Distance:
For LiDAR-based approaches, the proposed
UVTR-L achieves better performance in all situations compared with CenterPoint [24].
Equipped with both LiDAR and camera inputs in UVTR-M, the framework attains significant gains, especially in a relatively far distance (3.3% NDS gain in 20-30 m ).
If the object is too far (>30 m ), the performance gain decreases to 1.6% NDS, but still much better than CenterPoint and UVTR-L. (2)
Weather condition:
It is clear that the proposed UVTR-L achieves significant gain compared with CenterPoint in both conditions.
And additional camera input brings much better results, especially in rainy weather (4.1% NDS gain).
(3) Light situation: Compared with that in the daylight situation, both
LiDAR-based and camera-based approaches perform inferior in the dark night.
Compared with
4 Dropped View
ND S ( % ) 44.
Camera LiDAR2Camera
Multi-modality (a) Robustness of dropped camera view 0.
1.0 Translational Noise (m)
ND S ( % ) 44.
Camera LiDAR2Camera
Multi-modality (
[Figure]
Robustness of
[Figure]
noise
[Table]
[Table] [Table]
[Table] [Table]
AFDetV2 [47]
[Table]
FCOS3D [27]
[Table] [Table]
DETR3D
[Table] [Table]
BEVDet
[Table] [Table]
PETR
[Table] [Table]
FusionPainting [49]
[Table]
- - - - - MVP [32]
[Table]
- - - - - PointAugmenting [50] V0.
[Table]
DLA
[Table] [Table]
[Table]
CenterPoint, the proposed UVTR-L still performs better.
And the camera inputs still bring significant gains in both situations, especially in a daylight environment (2.5% NDS gain).
Robustness of the Framework.
To validate the robustness of UVTR, we simulate two typical sensor errors during inference in Figure 5.
For loss of view, the multi-modality manner achieves well robustness in Figure 5a.
Because LiDAR can still capture surrounding scenes if cameras are offline.
As for the camera-based manner, the model losses inputs from dropped scenes, but the network still works well and outputs predictions within the field of view.
For sensor jitter in Figure 5b, the model performs stable especially in the multi-modality setting because of the accurate perception from
LiDAR.
Meanwhile, knowledge transfer consistently improves performance in both settings.
Comparison with Leading Methods.
In Table 9, we present comparisons with leading methods on the nuScenes test set.
For LiDAR-based method, UVTR-L surpasses AFDetV2 [47] with 1.2%
NDS and attains 69.7% NDS.
For camera-based manner, with single frame, UVTR-L2C outperforms
[Table]
PETR [10] with 1.8% NDS and 1.1% mAP.
With 3 camera sweeps, UVTR-L2CS3 obtains significant gain and attains 55.1% NDS
[Table]
which can be further improved with more frames.
For the multi-modality setting
[Table]
we directly adopt pretrained models from LiDAR- and camera-based manner with simple fine-tuning without bells-and-whistles.
Compared with similar approaches without special module,
UVTR-M achieves 71.1% NDS and 67.1% mAP, which is on par with leading approaches.
Tracking Extension.
To better illustrate the capability and generality of the proposed UVTR, we further conduct experiments on the downstream tracking task.
In particular
[Table]
we follow the classic tracking-by-detection paradigm and apply the simple greedy tracker in CenterPoint.
The only difference lies in that we adopt threshold 0.2 and NMS to remove low quality results.
As presented in Table 10, the proposed UVTR achieves leading tracking performance with the greedy tracker in different settings.
Specifically, in a camera-based setting, the proposed UVTR-L2CS surpasses previous SOTA at the leaderboard (BEVTrack) with 17.8 % AMOTA.
It further proves the effectiveness and generality of the proposed cross-modality interaction in UVTR.
Discussion and Conclusion
We presented the UVTR, a conceptually simple yet effective framework for multi-modality 3D object detection.
The key innovation lies in that it unifies the voxel-based representation for different modalities and facilitates multi-level interactions.
In particular, it uniformly encodes inputs from different sensors in the modality-specific space to reduce the semantic ambiguity and enable spatial interaction.
With the unified representation, the cross-modality interaction can be easily conducted for knowledge transfer and modality fusion.
Moreover, object-level interactions in the unified space are further supported by the transformer decoder for accurate and robust detection.
Experiments on the nuScenes dataset prove the effectiveness of UVTR, which attains consistent improvements over various benchmarks and surpasses previous methods with leading performance.
There still exist certain limitations in the current method.
First, to construct voxel space for multi-view images, we need to process all of them in the shared image backbone, which brings computational cost, especially for multi-frame setting.
In the future, we plan to explore a new manner for voxel space construction at the early stage of the network, like that of point clouds.
Moreover, we construct the voxel space with a spatial resolution 128 × 128 .
We believe a higher resolution and more image frames could bring stronger voxel space with better results, which remains to be explored.
Societal Impacts.
The proposed method focuses on 3D object detection that can be used in autonomous driving.
Theoretically, a better 3D detector leads to safer autonomous vehicles.
However, in the short term, the current technique could not solve all the corner cases and extreme situations.
It may bring potential risks to the decision process in real-world autonomous systems.
Acknowledgement.
This work is supported by Shenzhen Science and Technology Program
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We give detailed experimental setup in Section 4.1 and publicly release the code.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.1 and supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] We report them in the supplementary material.
Our experiments in this paper require about 8 NVIDIA Tesla V100 GPUs.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.1.
(b) Did you mention the license of the assets?
[Yes] We conduct experiments on nuScenes [43] dataset with custom non-commercial license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] We conduct experiments only on public datasets.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]