CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion
Abstract:
To track the 3D locations and trajectories of the other trafﬁc participants at any given time, modern autonomous vehicles are equipped with multiple cameras that cover the vehicle’s full surroundings.
Yet, camera-based 3D object tracking methods prioritize optimizing the single-camera setup and resort to post-hoc fusion in a multi-camera setup.
In this paper, we propose a method for panoramic 3D object tracking, called CC-3DT, that associates and models object trajectories both temporally and across views, and improves the overall tracking consistency.
In particular, our method fuses 3D detections from multiple cameras before association, reducing identity switches signiﬁcantly and improving motion modeling.
Our experiments on large-scale driving datasets show that fusion before association leads to a large margin of improvement over post-hoc fusion.
We set a new state-of-the-art with 12.6% improvement in average multi-object tracking accuracy (AMOTA) among all camera-based methods on the competitive NuScenes 3D tracking benchmark, outperforming previously published methods by 6.5% in
AMOTA with the same 3D detector.
Project page: vis.xyz/pub/cc-3dt.
Introduction
Locating and tracking dynamic objects reliably in 3D is vital to safe autonomous driving systems.
Advanced techniques with complex pipelines generally use multiple 3D sensor modalities ranging from stereo cameras to LiDAR and radar.
Conversely, a high-quality surround-view camera system provides a low-cost alternative to expensive 3D active sensors to capture a vehicle’s surroundings with a panoramic view.
Many modern consumer vehicles are equipped with such multi-camera systems.
Yet, it is still challenging to effectively utilize temporal and cross-camera association of moving objects in 3D. Such an approach can help to track objects in the long term as cameras can locate them from different viewpoints.
Existing camera-based 3D object tracking methods are designed for single-camera setups [1, 2, 3].
Since those camera-based approaches follow the tracking-by-detection paradigm [4], the major difference between them lies in the association of detections over time [5, 6, 7, 8].
For instance,
Zhou et al.
[2] relies on 2D motion to associate objects over time, while Chaabane et al.
[3] utilizes object appearance.
In contrast, Hu et al.
[1] combines appearance and 3D motion information.
Even though these recent methods perform well on the temporal association of object tracks, they fail to associate object tracks in 3D across multiple cameras installed on the vehicle.
Therefore, we propose an effective cross-camera fusion method to the multi-camera 3D tracking problem.
We merge 3D detections and their appearance information from multiple cameras into a single 3D tracking component that performs both temporal and cross-camera association.
Given
[Figure]
the cross-camera association, we leverage a cross-camera motion model to improve the quality of single-frame 3D detections even when an object has just appeared in a particular camera’s ﬁeld of view (FOV), leading to highly accurate tracking with much smoother trajectory of objects in 3D. We call the resulting panoramic 3D object tracking method CC-3DT.
The ﬂexible model can work with various 3D object detection methods.
We test CC-3DT on multiple challenging datasets, namely NuScenes [9] and Waymo Open [10].
CC-3DT can associate objects both temporally and across cameras, leading to 17% relative improvement over the single-camera baseline on NuScenes.
CC-3DT can exploit the synergy between cross-camera association, leading to longer cross-camera trajectories, and improved object motion modeling, leading to further improved object association.
Consequently, we observe signiﬁcant improvements in both AMOTP and AMOTA scores.
Leveraging the cross-camera information not only provides smoother object trajectories but also offers a new state-of-the-art for camera-based 3D object tracking on the competitive NuScenes [9] benchmark, outperforming existing camera-based methods by a large margin.
Related Work
Camera-based 3D detection.
Driven by the success of convolutional neural networks (CNNs) in object detection [11, 12, 13, 14, 15], recent works on camera-based 3D object detection have been using CNNs to regress the 3D parameters of objects from an image [16, 17, 18, 19, 20, 21].
In contrast, another line of works [22, 23, 24] lifts the image to a 3D point cloud using single image depth prediction and detects objects in the 3D point cloud, similar to LiDAR-based 3D object detectors [25, 26, 27].
Lately, transformer networks have been explored for end-to-end object detection [28].
Inspired by this, Wang et al.
[29] propose to use a transformer network for 3D object detection from multiple cameras.
Our work extends beyond single image 3D detection by performing temporal reﬁnement of the 3D bounding box parameters, improving upon the initial detection result.
2D multi-object tracking.
Recent works on MOT follow the tracking-by-detection paradigm [4], where per-frame object detections are associated using different cues, such as 2D motion [30, 5, 31, 32, 33, 2], visual appearance [34, 35, 6, 8, 36, 3, 37, 38] or 3D motion [39, 40, 41, 42, 43].
Recently, researchers have explored the use of transformers in MOT [44, 45, 46].
These works take inspiration from Carion et al.
[28] and use the query-based detection process for temporal query propagation to track objects across time with a transformer.
In addition, there is a broad range of literature on multi-camera MOT [47, 48, 49, 50, 51].
Those works focus on scenarios with highly overlapping, stationary surveillance cameras in e.g. smart city applications [52].
Their aim is to overcome occlusion problems in monocular MOT systems.
On the other hand, our work focuses on multi-camera tracking in the context of autonomous driving, where the cameras usually have little overlap and are mounted on a moving platform.
Camera-based 3D multi-object tracking.
In the realm of 3D MOT, there are many methods that rely on LiDAR sensors since they provide both accurate 3D localization and 360-degree coverage of the environment [39, 53, 54, 55, 56, 57].
On the contrary, camera-based 3D MOT has been approached by only a handful of works [1, 2, 3].
Those methods are conceptually similar to 2D
MOT systems, employing tracking-by-detection using motion or appearance cues for association.
While sharing the properties of 2D MOT systems, the camera-based approaches cannot beneﬁt from the 360-degree coverage of the LiDAR-based 3D MOT methods.
Nevertheless, none of the methods above consider cross-camera association explicitly.
Concurrent to our work, Zhang et al.
[58] propose MUTR3D, a method that combines Wang et al.
[29] with the ideas in [44, 45, 46] into a transformer-based multi-camera 3D tracking method.
However, our work differs signiﬁcantly from
MUTR3D in that we do not rely on a transformer for joint detection and association, but fuse the detections from multiple cameras into a single tracking component.
Moreover, our method outperforms MUTR3D signiﬁcantly in experimental evaluation, even with a weaker 3D detector.
Method
We ﬁrst introduce our panoramic 3D tracking pipeline inspired by Hu et al.
Afterward, we detail our cross-camera association and motion modeling.
Our method is illustrated in Figure 1.
3.1 Overview
Assume we are given a set of input images I t from cameras M and the corresponding camera locations in world-frame ξ t in a sequence of time steps T .
We aim to ﬁnd the set of tracks T = { τ 1 , . . .
, τ N } corresponding to the objects in the scene.
Each track i ∈ [1 , N ] consists of a set of states (cid:104)
τ it , τ it +1 , ... (cid:105) depending on the time steps that the object is visible in.
Each state τ it = (cid:104)
f τ i t , b τ i t , c
τ i t (cid:105) at a speciﬁc time t ∈ T consists of an appearance embedding f τ i t ∈ R 256 , a 3D bounding box b τ i t ∈ R 7 and a conﬁdence value
c
τ i t ∈ [0 , 1] .
The 3D bounding box is deﬁned as b τ i t =
[ x, y, z, θ, l, w, h ] where ( x, y, z, θ ) deﬁne the 3D center of the object and its orientation around the vertical axis in 3D world-frame, and ( l, w, h ) deﬁne the object dimensions.
3D estimation and merging.
We use a 3D estimation network to extract detections D mt from an input image I mt of camera m at time t .
We lift all detections D mt from all cameras m ∈ M to world space with ξ mt and merge them with non-maximum suppression (NMS) in 3D, obtaining multi-camera detections D t .
The detections d ∈ D t share the same state space with tracks T , i.e. d t = (cid:104) f dt , b dt , c dt (cid:105) .
Data association.
We associate the detections D t with active tracks T t at time t by greedy assignment given an afﬁnity matrix
A ( T t ,
D t ) ∈ R ||T t ||×||D t || that combines appearance, location, and velocity correlation.
We compute the appearance correlation based on the appearance embedding similarity in [37, 38] as
[Equation]
The location correlation is based on the absolute difference between the predicted and observed 3D bounding box parameters as
[Equation]
where r is a constant scalar.
The motion correlation considers the absolute difference between the predicted motion V τ t = −−−−−→ [Equation] P
[Equation] t and observed pseudo motion V [Equation] P
[Equation] P [Equation] and the centroid distance | P τ t − P d t | as a
[Equation]
+ [Equation] a pseudo where , a centroid =
[Equation]
w cos is deﬁned as cosine similarity of two motion vectors to favor centroid distance if the motion vectors are in the same hemisphere and pseudo motion otherwise, so that we do not match a pair with conﬂicting motions.
Note that we transform object center P into world-frame using ξ to account for ego-motion.
Given the individual afﬁnities, A ( T t , D t ) is computed as
[Equation]
where w deep is a constant that weighs appearance and a hybrid of motion and location correlation.
Motion model.
We employ two LSTM networks Φ pred and Φ update to handle the extrapolation and update of trajectories given track histories and new observations, respectively.
Before data association, we use Φ pred to propagate the previous object state b t − 1 one time step into the future to compute the location and motion correlation.
We input the previous n − 1 = 5 object velocities v τt − n : t −and the previous LSTM hidden state h
pred t − 1 into the network, where an object velocity is deﬁned as v τt =
b τt − b τt − 1 .
The network outputs the predicted velocity ˆv τt , h pred t =
Φ pred ( v τt − n : t − 1 , h pred t − 1 ) .
After data association, we use Φ update to estimate a velocity reﬁnement given the predicted velocity ˆv τt , observed velocity v (cid:48)
τt , conﬁdence of the associated detection c τt and previous hidden state
h pred t − 1 .
The velocity reﬁnement v τt , h update t =
Φ update ([ ˆv τt , v (cid:48)
τt , c
τt ] , h update t − 1 ) in turn reﬁnes the object state b τt =
b τ t − 1 + v τ t .
Note that for both networks, we ﬁrst use a linear projection to encode each input into a 64 -dimensional vector which we omit for ease of notation.
Track management.
In order to handle newly appearing, disappearing, and reappearing objects during inference, we keep an online track memory where each track τ ∈ T is assigned one of three possible states { active , inactive , dead } .
At each time step t , all tracks currently marked as active or inactive are used to calculate their afﬁnity with the current detections D (cid:116) .
If a detection d t ∈ D (cid:116) matches with a track τ ∈ T , d t , it will be assigned to τ .
If a track τ has no matching detection, it will transition into state inactive .
If τ did not have a matching detection for L time steps, τ transitions into state dead and will no longer be considered.
Note that we propagate the position of inactive tracks forward in time with Φ pred to properly match tracks in cases of occlusion.
Each detection without matching track spawns a new track in active state.
3.2 Cross-camera association
Hu et al.
[1] employs a separate tracker for each camera m ∈ M .
In particular, the data association of each camera happens independently, as well as the track management and motion modeling.
Since each camera has only a limited FOV, there are many disappearing and re-appearing tracks, making the association prone to error and the trajectories less stable.
In addition, object identity will not be preserved across camera views.
We explore how a panoramic multi-camera tracker can overcome those limitations with a few simple modiﬁcations to the tracking pipeline.
First, we investigate how performance is affected if we use NMS in 3D to naively combine the existing tracks from single-camera trackers.
We call this approach detect → track → merge.
Next, we introduce the detect → merge → track paradigm, i.e. instead of aggregating the tracking result of camera-dependent trackers, we ﬁrst aggregate the detection results from all cameras m ∈ M and subsequently apply global data association, motion modeling, and track management.
The advantages of this are threefold.
(i)
In the regions where the cameras have an overlapping FOV, the multi-camera tracker has access to more detection candidates, meaning that an object that is highly truncated in one camera can be better recognized in a neighboring camera.
(ii) Cross-camera data association results in long trajectories that span across multiple cameras, which allows for a richer track history and thus better motion modeling.
(iii) Our data association scheme leverages appearance, location and velocity information to follow tracks more effectively across cameras compared to 3D NMS based merging of single-camera tracking results
3.3 Cross-camera motion modeling
A crucial advantage of our panoramic tracker is that we can exploit longer trajectories across camera views for motion modeling.
This leads to an improved multi-frame reﬁnement via Φ update and more accurate location prediction via Φ pred , which in turn beneﬁts the data association since we can expect better estimates for the afﬁnities a loc ( τ t , d t ) and a motion ( τ t , d t ) .
In addition, we propose to integrate cross-camera trajectories into the training process of the motion model.
The intuition is that due to object truncation, single-view 3D localization at image borders is usually brittle.
Thus, if we train the motion model on these trajectories, it can handle those situations better.
In contrast to Hu et al.
[1], who create the trajectory training data per camera and train on the union of the single-camera trajectories, we create a cross-camera trajectory dataset that matches ground truth trajectories spanning multiple cameras to the aggregated set of detections from all cameras.
To match detections to a ground truth trajectory, we use birds-eye-view (BEV) distance instead of
Intersection-over-Union (IoU) of 2D bounding boxes in Hu et al.
To train Φ pred and Φ update , we randomly sample a ground truth trajectory and extract the matched ground truth and detection pairs in a random temporal window ∆ t of ﬁxed size, possibly spanning across multiple camera views.
We use the set of detections to simulate inference, i.e. we obtain the predicted velocities ˆv ∆ t from Φ pred given the track history and the reﬁned velocities v ∆ t from Φ update given the predicted velocity, the observed velocity, and the detection conﬁdence.
We use the ground truth trajectory to calculate the ground truth velocities ˜v ∆ t and the trajectory loss
[Equation]
where δ ( · , · ) is the Huber loss [60].
Moreover, we add a regularization term that encourages Φ pred to predict a smooth trajectory over ∆ t :
[Equation]
Finally, we calculate the overall loss L motion as a weighted combination of the above terms
[Equation]
, ˜v ∆ t ) = L traj ( v ∆ t , ˆv ∆ t , ˜v ∆ t ) + w linear L linear ( ˆv ∆ t ) .
Experiments
Datasets.
We evaluate our method on two popular 3D MOT benchmarks.
First, we experiment on the NuScenes benchmark [9] which consists of street scenes captured in varying scenery and geographic areas in Boston and Singapore.
The scenes are captured from a moving vehicle with a multiple sensors attached.
It provides six different camera views with a 360-degree coverage of the environment.
In total, the dataset contains 1000 sequences sampled at 2 Hz.
Second, we evaluate our method on the Waymo Open benchmark [10], which has similar characteristics to the NuScenes benchmark.
However, the sampling rate is 10 Hz, and the ﬁve cameras mounted on the vehicle cover only about a 5 / 8 azimuth angle of the horizon.
Evaluation metrics.
We follow the evaluation protocol in NuScenes using the metrics in Weng et al.
The main metrics are average multi object tracking accuracy (AMOTA) and average multi object tracking precision (AMOTP) which are integrals over MOTA / MOTP [61] scores computed via n-point interpolation.
AMOTA is deﬁned as AMOTA =
[Equation]
where MOTA r = max (0 , 1 − IDS r +FP r + FN
r
+ − (1 − r )P r P ) .
Speciﬁcally, MOTA r is a recall normalized MOTA and IDS r , FP r ,
FN r are number of ID switches, false positives and false negatives at recall r .
P is the total number of ground truth positives.
AMOTP is deﬁned as AMOTP = 1 n − 1 (cid:80) r ∈{ 1 n − 1 , 2 n − 1 ,..., 1 } (cid:80)
[Equation]
d
[Equation]
t
[Equation]
t TP t where d i,t is the BEV distance between track i and its ground truth track at time t and TP t denotes the number of matched ground truth tracks at time t .
Implementation details.
We train the 3D estimation network using ResNet-101 [62] as backbone with batch size of 32 for 24 epochs with random image ﬂipping as data augmentation on 8 V
GPUs.
The initial learning rate is set to 0 .
01 with a linear learning rate warm-up in the ﬁrstiterations.
We decay the learning rate by a factor of 10 at epochs 16 and 22 , respectively.
We use
SGD [63] with a momentum of 0 .
9 and a weight decay of 0 . 0001 .
For NuScenes, we use 1600 ×resolution and train on the training set only.
On Waymo Open, we scale the images to 1280 ×resolution and use batch size of 16 during training.
We decay the learning rate by a factor of 5 at epochs 8 , 12 , 16 , 20 , 22 , respectively.
To merge 3D detections across cameras, we use NMS with a 3D IoU threshold of 0 . 1 .
To generate the cross-camera trajectory dataset, we use a matching threshold of 2 meters BEV distance.
We train Φ pred and Φ update with batch size of 128 forepochs.
We use w linear = 0 .
001 and Adam [64] with AMSGrad and a weight decay of 0 . 0001 . [Table] [Table]
4.1 Ablation studies
Cross-camera association.
In Table 1, we compare the pipeline in Hu et al.
[1], naive merging of single camera results (detect → track → merge) and our pipeline described in section 3.2 using the LSTM motion model, a Kalman Filter baseline (KF3D) and no motion model.
Note that all individual components are consistent while only changing the high-level pipeline to provide a fair comparison.
We observe that naive merging helps the overall performance and results in a gain of 1 . 7 % in AMOTA using the LSTM motion model.
With our cross-camera association (detect → merge → track), we observe another 1 .
9 % gain in AMOTA while the IDS are cut in half, highlighting the effectiveness of cross-camera association.
Note that while KF3D has slightly less IDS than LSTM, its recall level is also lower, i.e. IDS are lower due to lower recall not better association.
Interestingly, the gain of cross-camera association is very small when not using a motion model ( 0 . 1 %), supporting our hypothesis that the motion model beneﬁts greatly from cross-camera association.
Moreover, the inﬂuence of the motion model on the ﬁnal tracking performance is the highest when using cross-camera association, improving from 24 .
9% to 28 .
3% AMOTA.
Our results underline the importance of cross-camera association to achieve consistent tracking results.
Cross-camera motion modeling.
Table 1 also shows that motion modeling improves through the use of cross-camera trajectories, i.e. we observe a higher gain in AMOTA of 3.4% (LSTM) and 2.0% (KF3D) versus using no motion model compared to only 2.6% (LSTM) and 1.1% (KF3D) when using the pipeline in Hu et al.
The effect of training the motion model with cross-camera trajectories (see section 3.3) is illustrated in Table 2.
We observe another improvement of 0.6% in
AMOTA compared to utilizing cross-camera trajectories only during inference.
Moreover, Tableshows that the total number of trajectories for training increases, which can be attributed to the fact that for a trajectory to be considered for training, it needs to be at least || ∆ t || time steps long.
Crosscamera trajectories help to increase training data diversity.
In sum, we achieve a gain of 4.2% in
AMOTA, equating to a 17% relative improvement compared to the single-camera baseline.
Qualitative comparison.
Figure 2 illustrates CC-3DT results compared to the baselines in Table 1.
While our proposed CC-3DT preserves object identity and exhibits a smooth, accurate trajectory for the object moving across camera views, the baselines struggle with object localization and object association.
Speciﬁcally, QD-3DT [1] cannot preserve the object identity while naive merging (detect → track → merge) results in poor localization, showing the beneﬁt of our method. [Table] [Figure] k ( t )
[Figure]
ac k L e f t ( t + 1 ) B ac k L e f t ( t + 2 ) [Figure] 3D estimation network training scheme.
Following [29] and different from [1], we optimize the 3D estimation network training scheme by increasing the learning rate of the 3D detection head by a factor of 10.
As shown in Table 2, this improves the overall performance by 2 . 2% in AMOTA.
Note that IDS increase due to increased recall of our detector.
4.2 Comparison to state-of-the-art
Comparison on the NuScenes benchmark.
We evaluate our method the NuScenes benchmark in
Table 4.
On the validation set, we outperform the state-of-the-art method MUTR3D by 1.7% with a weaker detector, by 6.5% in AMOTA with the same detector (DETR3D [29]), and by 13.5% with a stronger detector (BEVFormer [66]).
Our AMOTP is signiﬁcantly lower (1.361 vs. 1.498) with the same detector, meaning that our trajectories are more precise.
We also have the lowest IDS, meaning that our association works best.
The testing set results corroborate our ﬁndings, i.e. we outperform
MUTR3D by as much as 12.4% in AMOTA and achieve the lowest AMOTP and IDS.
We set a new state-of-the-art of 42.9% and 41.0% AMOTA on the validation and testing sets, respectively.
[Table] Comparison to MUTR3D. Table 3 provides additional comparison to MUTR3D using different motion models.
MUTR3D reports results with a Kalman Filter baseline [67] and a feed-forward network (FFN).
While the FFN can only improve marginally upon the Kalman Filter baseline, our motion model signiﬁcantly outperforms the Kalman Filter baseline (KF3D).
Also, our cross-camera association is more effective than the transformer-based one in MUTR3D (lower IDS).
Note that while MUTR3D with Kalman Filter has the lowest IDS, the detection recall is also much lower.
[Table]
Comparison on the Waymo Open benchmark.
We compare with [1] on the validation split of
Waymo Open.
The comparison in Table 5 shows that our method performs better in MOTA than
QD-3DT, while also having a lower Mismatch ratio, meaning that we have both better overall and association performance.
We outperform QD3DT signiﬁcantly on each of the 3D IoU thresholds.
Note that we do not evaluate with an IoU threshold of 0 .
7 , as it is too restrictive for camerabased methods to achieve a meaningful score.
4.3 Limitations
Our method could fail if the predicted 3D detection is amiss.
Speciﬁcally, in cases of high occlusion or truncation, the object’s 3D properties are often ambiguous to the observer, e.g. an object’s 3D center could even be out of the camera FOV.
This is especially pronounced for big or close objects that are often only partially visible.
While combining information from multiple time steps in our motion model alleviates those issues, integrating this information during the detection stage could further improve the results.
These challenges can also inﬂuence the classiﬁcation performance of the 3D detector.
If an object is highly occluded or truncated, it is often challenging to infer its category from a single image, especially for semantically similar classes ( e.g. bus and truck).
However, if the category is estimated incorrectly, object identity may not be preserved.
Conclusion
This paper presents an effective 3D object tracking method in a multi-camera setting.
The key idea we present is to merge 3D detections of multiple cameras into a single 3D tracking component that handles both temporal and cross-camera association.
By leveraging cross-camera association, we observe improved motion model quality, leading to enhanced object association and smoother trajectories across camera views.
As a result, the suggested approach outperforms existing methods by a large margin achieving state-of-the-art results on the widely-used NuScenes benchmark.
Acknowledgments
This work is supported by Ministry of Science and Technology of Taiwan (MOST 110-2634-F002-051).
We gratefully thank National Center for High-performance (NCHC) for the storage and computational resource.