Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?
Abstract
Recent developments in monocular multi-object tracking have been very successful in tracking visible objects and bridging short occlusion gaps, mainly relying on data-driven appearance models.
While we have significantly advanced short-term tracking performance, bridging longer occlusion gaps remains elusive: state-ofthe-art object trackers only bridge less than 10% of occlusions longer than three seconds.
We suggest that the missing key is reasoning about future trajectories over a longer time horizon.
Intuitively, the longer the occlusion gap, the larger the search space for possible associations.
In this paper, we show that even a small yet diverse set of trajectory predictions for moving agents will significantly reduce this search space and thus improve long-term tracking robustness.
Our experiments suggest that the crucial components of our approach are reasoning in bird’s-eye view space and generating a small yet diverse set of forecasts while accounting for their localization uncertainty.
This way, we can advance state-of-the-art trackers on the MOTChallenge dataset and significantly improve their long-term tracking performance.
This paper’s source code and experimental data are available at https://github.com/dendorferpatrick/QuoVadis .
Introduction
Multi-object tracking (MOT) is a long-standing research problem with applications ranging from real-time dynamic situational awareness for robot navigation [21, 15, 77, 44, 62, 10], traffic monitoring [69], studying animal behavior [52] and monitoring biological phenomena [3].
State-of-the-art MOT methods [75, 8, 4, 82, 74, 65] combine regression [75, 4] and combinatorial optimization [8] in conjunction with identity re-identification (ReID) models [32, 59, 8, 75, 66, 4] to track objects in the image space.
Such approaches have been very successful for tracking visible objects and bridging short-term occlusions.
However, as can be seen in Figure 1b, long-term tracking remains an open challenge: state-of-the-art methods successfully bridge 50% of occlusions within one second, falling below 10% when the occlusion extends for more than 3 seconds.
This is often not reflected in standard benchmarks [15, 21, 69, 77], as long-term occlusions are statistically rare.
In the past, combining ReID models with simple motion models has been immensely helpful [15] for short-term tracking.
Nonetheless, as the occlusion time becomes longer, the set of possible associations grows exponentially with the increasing gap length [54].
This combinatorial complexity hinders the ability of visual-based ReID models to disambiguate between objects.
Consequently, we believe that ReID models are insufficient to resolve long-term occlusions.
However, continued efforts to develop stronger appearance models will remain an important research direction in vision-based
MOT.
Tracking moving pedestrians during occlusions is challenging, and simple linear motion models fail since human motion is complex and driven by non-observable factors such as goals, intent, or simply preferences.
Therefore, we propose an alternative in this work: using long-term trajectory forecasting in order to prune down the combinatorial search space of feasible trajectory continuations.
As the main contribution of this paper, we carefully study what is needed to leverage trajectory forecasting for multi-object tracking, as we have recently witnessed rapid progress in learning-based
[Figure]
[Figure] [Figure]
I D
[Figure]
c a ll
[Figure]
Forecasting
[Figure]
trajectory forecasting [55, 38, 23, 1].
However, these methods operate in a fully-observed, metric bird’s-eye view (BEV) space, effectively disentangling the effect of the perspective projection on reasoning about motion.
By contrast, monocular MOT methods only observe a projection of the visible portion of our 3D space.
Our analysis reveals that we can bridge this gap by localizing trajectories in BEV-space, but crucially, the localization of 2D bounding boxes in BEV must be temporally coherent .
We achieve this by estimating a single homography per sequence in a data-driven manner.
Forecasting methods can reason beyond simple linear extrapolations, predict multiple possible future outcomes, and account for social interactions.
But are these all necessary ingredients for bridging complex and long-term occlusions?
Our study suggests that the key ingredient is to estimate a set of forecasts that can possibly cover several diverging future paths with only a handful of samples and account for prediction uncertainty.
Our trajectory forecasting approach can be applied to improve the long-term tracking capabilities of existing object tracking methods.
In particular, by applying our framework on top of the state-ofthe-art method [80] of the MOTChallenge benchmark, we improve the performance on HOTA on
MOT17 by 0 . 09 pp and MOT20 by 0 .
10 pp and further decrease the number of IDSW by 93 and 36 , respectively.
We hope our conclusions will encourage the community to continue investigating how 3D reconstruction and trajectory forecasting improve single-camera long-term tracking.
We summarize our main contributions as follows: we (i) present a study on how we can reconcile two related fields of research on vision-based trajectory forecasting and monocular multi-object tracking.
Our study reveals that the core component of this interplay is temporally coherent reasoning about motion in 3D space.
We (ii) utilize a synthetic MOT dataset to study how to localize objects in 3D BEV space in a manner that facilitates robust reasoning about plausible future motion and which are the core forecasting components needed to bridge longer occlusion gaps; Finally, (iii) we demonstrate that we can generalize our conclusions from synthetic sandbox to real-world monocular
MOTChallenge sequences and demonstrate that our recipe can be used to improve long-term tracking performance for several object trackers.
Preliminaries
This section discusses the fundamentals of vision-based multi-object tracking and trajectory forecasting, the current state-of-the-art, and analyzes failure cases.
2.1 Multi-object Tracking
Monocular multi-object tracking (MOT) is the task of localizing objects as bounding boxes in image sequences and assigning them an identity-preserving unique ID.
State-of-the-art methods decompose the problem into object detection and detection association.
Quantifying tracking errors.
The detection aspect of the task is commonly quantified by counting per-frame detection errors over the sequence.
To quantify association errors, we count identity switches (IDSW) ( i.e. , wrong ID swaps or re-initializing a ground-truth track with a different tracking
ID) and identity transfers (IDTR) ( i.e. , incorrectly linking two different objects with the same tracking
ID).
While a successful association over occlusion gaps decreases the number of IDSW, a wrong association between tracklets leads to an IDTR instead.
Recently introduced HOTA [41] metric separately evaluates object detection and temporal association aspects of the tracking task.
Temporal association is quantified via association accuracy (AssA) term, that quantifies association recall (AssRe) and association precision (AssPr).
The AssRe term accounts for IDSW errors, while AssPr accounts for IDTR errors.
Are all identity errors created equal?
Correct track association of objects that undergo longer occlusion gaps is especially challenging because the appearance and position of an object may drastically change.
In MOT datasets [15, 21] the majority of occlusions are short occlusions ( ≤ 2 s ).
Hence, solving rare long occlusions ( > 2 s ) does not significantly impact the model performance.
As a result, long-term tracking is commonly overlooked in the literature.
This can be seen in Figure 1b: state-of-the-art methods bridge less than 10% gaps beyond three-second-long occlusions.
Prior work.
Early tracking methods focus on combinatorial optimization [79, 27, 36, 71, 49] and hand-crafting visual and motion-based descriptors [48, 33, 11], especially beneficial in the era of unreliable object detectors.
State-of-the-art methods for monocular visual MOT are data-driven and primarily rely on appearance.
Regression-based methods [4, 75, 82] can localize objects even when object detections are missing, often used in conjunction with ReID models to bridge short occlusions.
However, regression models fail when an occluded person appears at a distant image position.
For solving long-term occlusion, discrete optimization methods, combined with end-to-end learning based on graph neural networks [8, 70, 78], construct large graphs stretching over multiple seconds leading to high computational costs and complexity.
Motion has always played an essential role in visual tracking [5, 20, 4], especially beneficial in 3D where it is dis-entangled from projective distortion [34, 50, 26].
The interplay between reasoning in 3D for monocular pedestrian tracking and linear motion models was first investigated in [29].
Identity preservation is important in several applications, ranging from video editing, safety camera analysis, and social robots interacting with humans to autonomous driving.
We only have access to a single RGB camera in several application scenarios.
Exceptions are autonomous driving datasets [21, 10] that generally provide 3D sensory data, together with 3D track information.
However, only a handful of object tracks contain occlusion gaps longer than 2 s : 0 .
6% in BDD100K [77] and 4% in widely-used KITTI tracking [21] dataset.
Therefore, autonomous driving datasets are, at the moment, not well suited for studying long-term tracking.
Instead we conduct our experiments and analysis using MOTChallenge [15] dataset, where 19 .
4% of tracks undergo long ( > 2 s ) occlusions gaps.
We hypothesize that bridging long-term gaps requires understanding the projection geometry and motion models that can reason about plausible diverging future paths and non-linear motion.
2.2 Trajectory Forecasting
Pedestrian trajectory forecasting has been studied independently of the closely related task of object tracking.
Forecasting is challenging because (i) human behavior and, therefore, future motion underlies the effect of complex social and scene interactions and latent navigating intent.
Moreover, (ii) entire scene geometry is usually not directly visible to the observer, and in general, it is difficult to localize past trajectories precisely.
To this end, existing models use standard datasets [53, 35, 38, 55] and study forecasting in idealized conditions: given an accurate bird’s-eye view of the scene and perfectly-localized past trajectories to predict trajectory continuations in metric space.
Quantifying forecasting accuracy.
Forecasting performance is measured in metric space as Ldistance between the prediction and ground-truth trajectory (as final displacement error, FDE, or average displacement error, ADE) wrt.
top-k forecasts (commonly k = 20 ).
We note that this approach mainly incentivizes high forecasting recall and neglects forecasting precision which is important for the application of forecasting methods [13].
[Figure]
Prior work.
Early forecasting methods were deterministic, firstly based on physical models [25], and later on data-driven LSTM-based encoder-decoder networks [1] methods, focusing on modeling social [23, 1, 2] and scene [56, 31] interactions.
The forecasting task is inherently uncertain, and we need to express the stochasticity in the model.
With learning a distribution of possible future trajectories, generative models [23, 56, 31, 2, 14, 13] have emerged as state-of-the-art prediction methods.
Recent efforts have been explicitly focusing on conditioning forecasting on estimated pedestrian goal/intent [14, 43, 42] and estimating multimodal posterior distributions [38, 13] that yield diverse trajectories that cover different plausible directions.
These deep neural network approaches can model complex and non-linear trajectories beyond simple linear models.
Can we bring the two worlds together, and if so, how ?
Furthermore, which of the aforementioned aspects of forecasting methods ( i.e. , stochasticity, non-linearity, multimodality, diversity, accounting for interactions) are important in the context of multi-object tracking?
These are the questions we discuss in the following sections.
Methodology
In this section, we present our method for long-term multi-object tracking based on trajectory forecasting in bird’s-eye view (BEV) scene representation.
Simply applying trajectory prediction to multi-object tracking is not trivially possible, as object trajectories observed in the image space break multiple assumptions of real-world trajectory prediction.
While trajectory prediction works in bird’seye view coordinates, the motion and size of objects in image space depend on the camera’s intrinsic parameters, orientation, and position.
In addition, we face temporal (limited length of observation), association (association errors along with observation), and measurement (imprecise localization of objects) uncertainties of the trajectories.
Contrarily, objects are represented as bounding boxes in the image instead of single 2D positions for the object tracking task.
To bridge the gap between prediction and tracking, we must find a transformation from the image to the real space.
We assume objects move on a planar ground to formulate such a transformation.
Thus, the bottom-center points of detection bounding boxes p can be mapped to a 2D BEV coordinate x via an initially unknown homography transformation H that relates the homogeneous coordinates as x ∝ H · p .
Overview.
Given a monocular video sequence captured from a stationary camera from arbitrary viewpoint, we first estimate the homography H , which maps the image plane to the 3D world groundplane for the whole sequence (Section 3.1).
Then, we incorporate our model into an online tracker that takes a monocular tracker output and localizes tracks and detections in BEV space (Figure 2 a ) using the estimated homography.
Next, we forecast lost tracks in BEV space (Figure 2 b ) using our trajectory forecasting network (described in Section 3.2).
Finally, we integrate forecasts into the online tracker (Section 3.3) while accounting for the uncertainty in estimated forecasts, and match new detections to existing tracks to resolve short- and long-term occlusions (Figure 2 d ).
3.1 Data-driven Homography Estimation
For combining monocular object tracking and forecasting, we first need to transform object detections and tracks from the image sequence into points and trajectories in a bird’s-eye view representation.
[Figure]
Given a set of 2D object detections represented as bounding boxes localized in the image plane, we aim to find a homography H that maps their bottom-center positions to their corresponding 2D BEV coordinates.
In Figure 3, we outline our homography estimation method.
We first train a monocular depth estimator [6] on a synthetic dataset [19] to reconstruct a 3D point cloud (with estimated or known intrinsics) of the first frame of a static sequence.
Then, we leverage the semantic segmentation network [73] to mask, select, and fit a plane to the ground pixels.
We estimate the normal vector of the ground plane in 3D and align the plane to the XY plane.
Then, we project ground points along the z -axis, leaving us with a pairwise correspondence between ground pixels in the image and a 2D position in BEV, as needed to estimate the homography between the two planes.
We also linearize the homography transformation for pixel positions close to the plane’s horizon to prevent the transformation from diverging (for more information, see Appendix A.1).
Static camera.
We compute the homography only for the first frame of the sequence and use it throughout the sequence, making our pedestrian localization robust to temporal fluctuations of the depth estimator.
Moving camera.
For moving camera sequences, we also need to account for the egomotion of the camera, which we estimate between consecutive frames as follows.
First, we compute a framedependent homography H t for each frame.
Then, we compute pairwise pixel-correspondences between (masked) ground pixels using optical flow [12] and compute a translation vector between the two point sets (lifted to 3D via H t ).
Empirically, we observe that estimating only translation (without rotation) yields more robust egomotion estimates.
3.2 Forecasting
Localization of object tracks in BEV enables us to leverage data-driven forecasting models beyond simple linear motion to reason for future trajectories.
However, these models expect ground-truth fixed-size past trajectory observations, while our projected trajectories are noisy and of varying lengths.
As discussed in Section 2.2, prediction models are optimized for metrics that incentivize a large number of predictions and minimize L 2 distance to ground-truth trajectories.
It is thus unclear how different proposed concepts translate into real-world tracking scenarios.
We, therefore, identify the main design patterns proposed in the forecasting community and verify their impact directly in the context of forecasting to track .
Preprocessing.
Forecasting models encode trajectories using an LSTM encoder-decoder [1] architecture, which takes a fixed-size observed trajectory as input and predicts a future trajectory.
We construct input trajectories from temporally consecutive detections of the same track ID, localized in
BEV.
To account for localization noise, we smooth the noisy observations using the Kalman filter and linearly extrapolate trajectories into the past to get trajectories of the required fixed-size input length.
Trajectory forecasting design patterns.
In our experimental setup, we build on the LSTM encoderdecoder architecture [1] and include the following key design patterns recently emerging in the forecasting community.
Stochasticity.
Stochastic trajectory predictors enable us to sample multiple plausible future trajectories to account for the uncertainty in future positions.
We follow the approach by [23] and learn a generative GAN [22] model and train it with a best-of-many [7] loss.
As a result, the network
[Figure]
internally learns an observation-conditioned distribution of future trajectories, from which we can sample.
Social Interactions.
Social interactions impact future motion: pedestrians adapt their trajectories on-the-fly to avoid collisions.
Several methods [23, 1, 2, 31] account for interactions in the forecasting literature.
These methods leverage pooling [23], attention [56], or graph neural networks [31] to provide social context ( i.e. , trajectories of surrounding agents) to the trajectory decoder.
To answer whether modeling social interactions matters for tracking, we implement Social GAN (S-GAN) [23], which uses pairwise interaction features between neighboring pedestrians by using max-pooling before they are passed to the decoder.
Multimodality and Diversity.
While the aforementioned generative models learn a distribution over trajectories, they need to sample many trajectories to cover all modes present in the scene, as learning a multimodal posterior with a single GAN is difficult [64].
To predict the scene’s main modes with as few samples as possible, we implement a multi-generator GAN network [13], extending the presented GAN architecture by training multiple decoder heads, where each decoder learns to focus on a particular model.
As a result, we get a set of plausible but maximally separated predictions by sampling from these different generators.
3.3 Tracking via Forecasting
We assume we have an online object tracker capable of tracking visible objects ( e.g. , bounding box regression-based tracker [4]).
As long as tracks are being updated with new detections, we consider them active and keep them in the active set S A .
Once we cannot associate a detection, a track becomes inactive and is stored in the set of inactive tracks
S I .
For each frame t the tracker outputs a set of tracks O =
( o 1 , . . .
, o M ) with o i =
( ID i , b i , f i ) where ID ∈ N + represents the track identity, b ∈ Rdenotes a bounding box in pixel space (see Figure 2a), and f ∈ R D represents a D -dim feature vector encoding the appearance information obtained from a pre-trained convolutional network [24].
We localize bounding boxes in BEV coordinates x ∈ R 2 using our estimated homography H .
Quo Vadis?
If an object track becomes inactive ( i.e. , temporally lost), we move the active track into the memory bank and predict k trajectories of length τ max in BEV space using the trajectory forecasting model as described in Section 3.2.
As long as the track is inactive and not yet matched, we move along the predicted trajectory and do not predict an entirely new trajectory in each frame.
Filtering and removing predictions.
No prediction can live forever.
When we use stochastic trajectory predictors with multiple predictions, we need to limit the number of inaccurate or obsolete predictions to decrease the chance of false re-association.
In practice, we limit the lifetime of a prediction to a maximal lifetime of τ max and try to filter out unlikely forecasts.
We use spatial and social context to determine the freespace [29] in which objects should be visible to the camera.
We assume that visible objects eventually are detected and, therefore, remove prediction branches that should be visible for more than τ vis frames.
We consider an object as visible if neither scene nor other pedestrians occlude the object.
In particular, this means that the predicted BEV position lies in an area of the projected ground mask (shown in
Figure 2c) and has no bounding box overlap ≥ 0 .
25 with any other object detection, closer to the camera.
Relative order can be determined based on bottom bounding box coordinates for amodal detections.
If all predictions from an inactive track are removed even before τ max , we also remove the entire track.
Matching predictions with new detections.
Given the trajectory forecasts, we match them with new detections via bi-partite matching, following the standard practice in tracking [81, 68, 39, 5].
This boils down to computing association costs c ij between the predictions of an inactive track i and new un-associated detections j :
[Equation]
where ∆ IoU is the IoU score between the two bounding boxes, ∆ L 2 is the Euclidean distance between the prediction i and a detection j in BEV, and ∆ App represents the cosine distance between visual features f i and f j .
τ L 2 , τ IoU , and τ App denote thresholds for the matching.
Therefore, we determine an association in BEV metric space and in the image domain using IoU bounding box overlap between the forecast and detected box.
Matching tracks based on spatial distance in real space leads to high recall and reduces the number of ID switches (IDSW) but may also lead to an increase in ID transfer errors (IDTR), especially in crowded scenes with many new detections close to each other.
While forecasting significantly narrows combinatorial search space for associations, verifying potential associations with an appearance model is still beneficial in practice.
To decrease the number of wrong associations, we require a minimal visual similarity τ App and minimum IoU overlap of the bounding boxes τ IoU for close objects.
These thresholds serve as a filter of visually incompatible matches but do not add to the value of the cost function for the matching.
In essence, we obtain a pre-selection of potential matching candidates by using the trajectory forecast and filter those if the appearance drastically deviates between the last observation and the new detection.
Experimental Evaluation
In this section, we first discuss our evaluation test-bed, followed by an experimental study on bird’seye-view (BEV) trajectory reconstruction (Section 4.1).
Then, we analyze different forecasting design patterns applied to the domain of object tracking in BEV space and discuss the relevance of different model modules for tracking (Section 4.2).
Afterward, we demonstrate how our approach can be used to improve several vision-based MOT methods on static sequences and to justify our design decisions.
Finally, we show that our forecasting model can be used to establish new state-of-the-art on the real-world MOT17 and MOT20 datasets (Section 4.4).
For visualization of our tracking method, we refer the reader to Appendix C.
Datasets.
We evaluate our trajectory prediction framework on different publicly available pedestrian tracking datasets, namely synthetic MOTSynth [19] and two real-world MOT17 and MOT datasets [15].
MOTSynth is a large synthetic dataset for multi-object tracking.
It providesdiverse sequences with various viewpoints, lighting, and weather conditions.
Importantly, it provides ground-truth depth information and 3D key points for pedestrians, allowing us to study the suitability of different methods for BEV trajectory reconstruction.
MOT17 [47] and MOT20 [16] are real-world tracking datasets commonly used to benchmark pedestrian tracking models.
We use these datasets to evaluate our method on real-world recordings.
For our experiments, we utilize the commonly used split of the MOT17 training set, where the first half of each sequence is used for training and the second half for the evaluation [37, 80, 72].
Metrics.
For measuring the quality of the bird’s-eye view reconstruction, we indirectly evaluate the quality of the reconstruction by evaluating the forecasting and tracking performance.
To compare different models for trajectory forecasting, we report the standard L 2 final displacement error (FDE) for top-k predictions for 2 s and 4 s prediction horizons (see Section 2.2).
For multi-object tracking evaluation, we report higher-order tracking accuracy (HOTA) [41], with a focus on the association aspect of the task.
To this end, we also report AssA, AssPr, and the number of
ID switches IDSWs.
Additionally, we report IDSW when the tracker loses an object and re-initiates a new track for the same object when it re-appears.
We call these errors as ID lost .
For metric discussion, we refer to Section 2.1.
Metrics labeled with either S (short) or L (long) only consider prediction or occlusion lengths shorter or longer than 2 s , respectively
Hyperparameters.
For all experiments, we use the following parameters for the matching of detections with inactive tracks: τ L 2 = 2 . 5 m, τ App = 0 .
8 , τ IoU = 0 .
The maximal lifetime of prediction is τ max
= 6 s and maximal visibility τ vis = 1 s before it is removed.
We refer the reader to
Appendix B for further information on implementation details.
Object trackers.
We study and ablate our method on eight high-ranked state-of-the-art trackers of MOTChallenge and refer to them as baseline .
We use BYTE [80], JDE [68], CSTrack [37],
FairMOT [81], TraDes [72], QDTrack [51], CenterTrack [82] and TransTrack [63] for an evaluation [Table] on the MOT17 validation set and BYTE and CenterTrack on the MOT20 training dataset.
These trackers use ReID similarity and/or simple motion cues for bridging short-term occlusions
[Table]
4.1 Bird’s-Eye View Estimation
This section discusses different approaches to obtaining scene BEV representations of detected objects in the image for forecasting.
We work with static sequences of the MOTSynth dataset (that provides depth maps used for evaluating and training a monocular depth estimator).
With this, we test a linear motion model to gap occlusions of different durations, which we obtain by running a
CenterTrack [82] baseline tracker.
We evaluate the ratio of successful matches between the target and the linear prediction and count a match to be successful if the IoU of the predicted bounding box is larger than 0 .
5 or the L 2 distance in metric space is lower than 2 m .
We get the predicted bounding box by translating the last observed bounding box by the predicted displacement in the image.
Baselines.
We compare motion in (i) BEV and (ii) pixel space.
We evaluate different approaches to localize trajectories: (a) using ground truth (GT) 3D coordinates orthographically projected to BEV (oracle), (b) the proposed homography estimation as described in Section 3.1, and (c) directly using learned monocular depth estimates and resulting point clouds, followed by orthogonal projection of points these representing an object.
Conclusions.
As seen in Figure 4, GT 3D (oracle) based motion estimates solve 89 .
3% of the gaps, suggesting that the motion in the synthetic dataset is dominantly linear.
Our proposed datadriven homography estimation approach only drops by 15% compared to using ground-truth 3D keypoints.
By contrast, estimating linear motion in pixels space only resolves 50 .
2% , and using per-frame monocular depth estimates 43 .
1% of the occlusion gaps.
This is likely because such depth estimates are not temporally stable.
As can be seen in Figure 4b, this performance is especially apparent for longer occlusion gaps.
Furthermore, we forecast motion in pixel space but transform the prediction into BEV for matching.
While increasing performance compared to exclusive forecasting and matching in pixel space, we find that the results are inferior to predictions in BEV due to the distortion of projecting real motion into the image plane.
We conclude that our proposed homography transformation is suitable for forecasting.
4.2 Trajectory Prediction Models
In this section, we evaluate different forecasting models and components (as discussed in Section 3.2).
We compare a constant-velocity model (Kalman filter) in BEV and pixel space, an identity (static) model, and a stochastic GAN predictor generating k = 3 and k = 20 samples.
Furthermore, we test predicting social interactions with GAN and the multimodal trajectories with MG-GAN in BEV space.
𝑡=0.7𝑠 [Table] =15𝑠
[Table]
[Table]
Forecasting.
We observe in Table 1 that the linear model performs well for short-term windows ( 0 .
69 FDE S ), suggesting that linear motion is suitable for short occlusions.
We also do not find a significant difference between GAN w/o social module (S-GAN).
While FDE error suggests vanilla
GAN ( k = 20 ) yields the best forecasting results ( 0 .
65 FDE S and 0 .
99 FDE L ), but this configuration [Table]
leads to the lowest association precision ( 71 . 31 ) and HOTA score (
. 81 ) wrt.
tracking performance.
This result suggests a misalignment of evaluation metrics used in forecasting and tracking; a better forecaster in terms of ADE/FDE does not necessarily lead to a better tracker.
This is a known drawback of ADE/FDE metrics, which essentially measure only recall and not the precision of the forecasting output.
Furthermore, this shows the careful trade-off between the number of predictions k and the recall/precision of the predictions and tracking results.
Tracking.
To investigate the effect on long-term occlusions, we focus on the change of ID lostL for short ( t occl ≤ 2 s ) and long ( t occl > 2 s ) occlusion gaps.
As can be seen in Table 1, even the static motion model solves 8 .
4% (ID lostL ), as many occluded objects do not move.
By modeling linear motion (Kalman filter in pixel space), we can improve short-term re-association for 0 .
59 pp (long-term IDSW) over the static model.
We focus the discussion on long occlusion gaps.
In terms of the generative model, we observe that interaction-aware S-GAN ( 8 . 64% ID lostL ) is on-par with vanilla
GAN ( 8 . 57% ID lostL ) for k = 3 ; interestingly, both are below linear Kalman filter (BEV) performance, suggesting that these models suffer from low precision.
Only MG-GAN, explicitly trained to generate multimodal trajectories, outperforms the linear model ( 17 .
43% ID lostL )
and significantly outperforms vanilla GAN with only three samples.
These conclusions generalize to tracking metrics.
4.3 Tracking Evaluation
In this section, we study the impact of forecasting models on the valuation set’s tracking performance.
First, we discuss the impact of different design decisions on matching strategy, as explained in
Section 3.3.
Trajectory matching.
First, we ablate the matching cost function (Equation (1)).
As can be seen in Table 2, we find that a combination of L 2 and IoU without any threshold τ leads to the highest decrease in terms of ID lost ( − 28 .
02% ) and overall highest association recall (AssRe) ( 61 . 17 ).
However, this is at the cost of decreasing association precision (AssPr) ( − 4 . 09 ).
We obtain the highest AssPr ( 73 . 36 ) by only relying on L 2 matching and thresholding, however, at the loss of
AssRe ( − 0 . 74 ).
Adding appearance-based τ App and IoU τ IoU thresholds provide the best trade-off and overall highest AssA ( 54 . 29 ) and HOTA score ( 54 . 27 ) while still recovering 21% of lost trajectories.
Validation results.
In Table 3, we present the performance of different state-of-the-art trackers on the static sequences of MOT17 -val and MOT20 -train
(trained on MOT17 ), equipped with our trajectory forecasting model.
As can be seen, our model brings stable improvements over all the key metrics: HOTA, AssA, and IDSW.
Our trajectory prediction model consistently reduces IDSW for all models.
This is also shown in Figure 1b where we demonstrate that our forecasting model improves
ID recall significantly for occlusion times > 1 s .
While our focus was on sequences with stationary viewpoints, we show that our model is also applicable to sequences with moving cameras by estimating the camera’s egomotion as described in Section 3.1.
In Table 4 we present results on the moving sequences of the MOT17 validation set (excluding sequence MOT17-05 for which the quality and consistency of our depth estimator was too low to construct time-consistent homographies).
As can be seen, we improve 6 out of 8 trackers wrt.
HOTA score and even improve CenterTrack [82] by 3.07 pp.
We visualize the traversed BEV map of sequence MOT17-07 in Figure 5.
4.4 Benchmark Evaluation
In this section, we apply our method to state-of-the-art tracker ByteTrack [80] and, by improving its long-term tracking capabilities, establish a new state-of-the-art on the MOT17 & MOT20 benchmarks.
We evaluate our method in the private detection regime, as these trackers use private detectors.
In Table 5, we compare our QuoVadis to the base tracker ByteTrack [80] and compare both to MOT benchmark published top-performers.
We improve performance on key metrics overall top methods.
Notably, we reduce the number of identity switches by 93 compared to [80] and establish a new
[Table] [Table]
[Table]
state-of-the-art in terms of HOTA ( 63 . 14 ).
We observe similar trends on MOT20 , where we improve over the base tracker ByteTrack [80] by +0 .
5 in terms of IDF1 and reduce the number of identity switches by 36, similarly establishing a new state-of-the-art ( 61 . 48 HOTA).
Remarks and Limitations
The paper primarily focuses on the conceptual work of building an entire pipeline from video to tracks studying different forecasting paradigms, and showing the benefit of leveraging trajectory forecasting in BEV for the tracking task.
Nevertheless, we want to outline further remarks and limitations of our work.
Model complexity.
Our model is complex, consisting of multiple sub-modules, as constructing object trajectories in BEV space based on a monocular video and trajectory forecasting are challenging problems.
We foresee that future work will improve the end-to-end integration and efficiency of the algorithms.
Bird’s-eye view reconstruction.
A vital part of our approach is an accurate homography transformation that allows us to project the objects in the image into the 3D ground plane.
However, the homography transformation depends on the quality of depth estimates, which makes the overall approach sensitive to errors in 3D localization.
Future work will benefit from further development of time-consistent depth estimators.
Furthermore, the presented trajectory prediction models do not yet account for the BEV localization uncertainty, which results from errors in the transformation or the simplified assumption of the ground plane.
These limitations show the need to develop trajectory forecasting models that account for the localization uncertainties of the upstream tasks.
Conclusion
This paper presents a study on how to bridge the gap between real-world trajectory prediction and single-camera tracking.
Throughout our paper, we identified challenges and solutions to leveraging real-world trajectory prediction to benefit single-camera tracking.
In particular, we focus on resolving the re-identification of objects after long-term occlusions.
Here, we start from the first principles, questioning motion representation in pixel space and using a combination of models to construct a more accurate BEV representation of the scene.
We find that the key component is a forecasting approach reasoning about multiple feasible future directions with a small set of multimodal forecasts.
We can substantiate our conclusion by achieving new state-of-the-art performance on the MOT17 and
MOT20 datasets.
Ultimately, we have showcased a novel way of combining state-of-the-art trajectory prediction models and multi-object tracking task.
We have outlined a new way of thinking about motion prediction in tracking and motivating the beneficial symbiosis of both tasks.
We hope that both fields start moving towards each other and incorporate the requirements and needs of each other.
Acknowledgments.
This research was partially funded by the Humboldt Foundation through the
Sofja Kovalevskaja Award.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See(c)
Did you discuss any potential negative social impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] https://github.com/dendorferpatrick/QuoVadis (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Supplementary.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Supplementary. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See 3.
(b) Did you mention the license of the assets?
[No] The dataset is freely available for research purposes (c) Did you include any new assets either in the supplemental material or as a URL?
[No] No new assets were introduced (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Quo Vadis: Supplementary Material
The supplementary material complements our work with additional information on the bird’s-eyeview reconstruction in Appendix A.
Furthermore, we provide implementation and training details on different components and networks used in our method in Appendix B. Finally, we present visual examples as visualizations and videos in Appendix C.
A Information on Bird’s-Eye View Reconstruction
The paper presents our approach to constructing a bird’s-eye-view (BEV) representation for a static tracking sequence.
Here, we extend the explanation by adding a description of moving cameras and how we linearize the homography transformation for farther objects to avoid enormous distances and unrealistic velocities.
A.1 Linearization of Homography
To get a bird’s-eye-view (BEV) representation of the tracking scene, we estimate the homography
H between the image and the ground plane.
Hence, the homogenous pixel positions transform accordingly to Equation (2) as follows:
[Equation]
This approach assumes that objects move on a perfect plane and object’s position in the image is represented as the bottom mid-point of the object’s bounding box.
Depending on the perspective transformation of the camera, we find that minor changes in pixel value lead to enormous distances in BEV.
Given a homography matrix:
[Equation]
the BEV coordinate y is computed as:
[Equation]
As the denominator in Equation (4) is approaching zero, the y -coordinate grows hyperbolically.
This behavior is undesired for trajectory prediction because these large jumps in the object’s position result in unrealistic velocities for the object.
Therefore, we define a threshold for which we linearly extrapolate the transformation such that the transformed distance between two neighboring pixel points is maximal 0 .
2 m as shown as a red line in Figure 6b.
This formulates the condition as
[Equation]
We call the p y value for which the inequality Equation (5) is equal, the linearization threshold p Ty .
The pixel point where the denominator of Equation (4) becomes 0 is called the horizon because no point on the plane is projected on a lower point in the image.
To prevent this hyperbolic growth for image points closer to the horizon, we linerarize Equation (4) around p Ty and apply the linear transformation for all p y ≤ p Ty as shown in Figure 6b.
Thus, we stabilize the distance between two points to prevent very unrealistic velocities, which would make the transformed values pointless.
To transform from pixel space to BEV and back, we also inverse the linearized transformation to get a one-to-one mapping.
B Implementation Details
In this section, we provide additional information on the implementation of our method and its key components.
The source code is available at https://github.com/dendorferpatrick/
QuoVadis .
B.1 Synthetic Training Data
For training the trajectory predictor (Appendix B.2) and the depth estimator (Appendix B.3) we use the MOTSynth dataset [19] which provides ground truth 3D positions of objects and image depth information.
We use the split suggested by Fabbri et al.
[19] with 576 sequences in the training set and 192 in the validation set.
[Figure]
B.2 Trajectory Predictor
For our trajectory model, we use the implementation of MG-GAN [13].
For studying the effect of modeling social interactions on tracking, we implement a social max-pooling module following
S-GAN [23].
Model.
The trajectory prediction model generates a set of K future trajectories { ˆ Y ki } k =1 ,...,K with t ∈ [ t obs + 1 , t pred ] given the input trajectory X i with t ∈ [ t 1 , t obs ] for each pedestrian i .
We use t obs = 8 observation steps and t pred = 12 prediction steps as default for training the model.
However, input and output length can vary depending on the observed tracks during inference for the tracking model.
Forthe multimodal MG-GAN implementation, we use n G = 3 generators from which we sample one prediction from each generator during inference.
Training.
We construct trajectories of the MOTSynth data with 8 observation and 12 prediction steps, each step being 0 .
4 s .
The entire model is trained in a GAN framework using a prediction model and a discriminator network.
We train the network on the entire train dataset over 200 epochs, with a learning rate λ = 10 − 3 , and using a batch size scheduler [58].
B.3 Depth Estimator
Depth estimation is a crucial part of the BEV estimation in our model.
Therefore, we use a vision transformer-based [18] network [6] for monocular dense depth estimation.
Model.
The transformer-based model regresses the depth prediction as a linear combination of depth range bins of adaptive size.
The network encoder-decoder extracts visual features from the image, which are passed to the mVit block.
mVit is a lightweight vision transformer based on [18].
The model applies an MLP on top of the mVit’s output, predicting the size of the bins for the depth range.
The encoder computes the weights of the bins by passing the features through multiple convolutional layers with a final softmax non-linear activation function.
Training.
The network trains on the synthetic MOTsynth dataset to leverage a large number of tracking scenes of varying perspectives, weather, and light conditions.
To better generalize to realworld data, we augment the scenes with ground reflections by mirroring surfaces in the image.
This results in better performance, especially for the indoor MOT sequences, with ground reflections.
We find the model trained on synthetic data performs well on real data even without fine-tuning.
To increase the default model depth map resolution from 640 × 480 to 960 × 576 we grow the transformer positional embedding vector size from 500 to 1000.
We trained the model using AdamW optimizer [40] with weight decay 10 − 2 .
Following [57], the maximum learning rate λ max was set to 3 .
5 × 10 − 4 with linear warm-up from 14 λ max → λ max for the first 30% of the iterations followed by cosine annealing to 3 4 λ max .
We trained the model for 20 epochs with an image resolution of 960 × 576 on a training split of MOTSynth dataset with the batch size of 8 on 4 RTX 8000 for one week.
Then, we trained the model for 30 epochs with an image resolution of 960 × 576 on the full MOTSynth dataset with a batch size of 8 on 4 RTX 8000 for ten days.
[Figure]
B.4 Image Segmentation
We run a pre-trained Detectron2 [73] segmentation network to get the segmentation masks of the tracking scene images.
Explicitly, we use the pre-trained COCO Panoptic Segmentation model with
Panoptic FPN [73].
The model outputs semantic labels for 134 COCO classes and panoptic object ids, which are irrelevant to our task.
In our model, we use segmentation labels to mask ground pixels of the scene.
Therefore, we combine the following COCO classes to our ground class: pavement , road , platform , floor , floor − wood , grass , sand , dirt .
B.4.1 Optical Flow
We estimate the optical flow using the implementation [12] of an attention-based GMA model[28].
We use the standard MMFlow configuration for the GMA pre-trained model on a mix of the datasets [28, 17, 45, 9, 46, 30].
While the model was pre-trained on images with size (768, 368), we resized the MOTChallenge images to (960, 540) at test time.
C Visual and Qualitative Results
This section shows a visual example of the difference between a BEV and a 2D image space prediction.
Furthermore, we want to point to the additional scene videos also provided in the Supplementary material.
2D versus 3D. In Figure 7 we show the trajectory prediction of our MG-GAN projected into the image (Figure 7a), the prediction in BEV (Figure 7b), and trajectory prediction in pixel space using a
Kalman Filter (Figure 7c).
We find the problem of the model reasoning in pixel space and cannot account for the effect of the camera perspective.
As a consequence, this results in unrealistic motion in image space.
In contrast, we see in Figure 7a that our model predicting in BEV understands the spatial structure of the scene and is, therefore, able to predict the correct trajectory for the object and resolves the long-term occlusion.
Example Videos.
Additionally to the written supplementary material, we provide brief video clips of different MOT17 validation and test sequences with ByteTrack and Center Tracks.
In Figure 8, we give a brief description over the format of the provided video sequences.
We show our predictor output on the left side, the baseline tracker output in the middle, and the online
BEV prediction and reasoning on the right side.
For our model, we show the tracker detection and predictions in BEV, including their projection in the image.
D Information on computation of ID Recall
In the introduction, we present the performance of the baseline trackers compared to our trajectory forecasting model on how well they can re-associate tracks after occlusion from occlusions.
We measure the performance as a fraction of ground-truth tracks detected and assigned correctly before and after occlusion.
As the first step, we need to identify the ground-truth occlusion regions for every sequence.
We use the visibility scores of objects and threshold those into a binary visibility flag, stating whether an object is visible in a given frame.
Then, we apply a minimum rolling window on the visibility flags to get connected components and to smooth the deviations of the visibility flag values. The rolling
[Figure]
window also includes visible frames before and after the occlusion, where the actual IDSW may happen.
We compute the frame ids where an occlusion starts and ends by extracting all connected components, with the visibility flag being 0 .
We only consider components where the object is visible before and after the occlusion.
Finally, we check for every tracker if the tracker detected an object at the start and the end of the occlusion component and if the track ids between the beginning and start match.
We use τ vis = 0 .as visibility threshold and a temporal window size ws = 5 as hyperparameters.