Paper: Learning Pseudo 3D Representation for Ego-centric 2D Multiple Object Tracking (ICLR-2024)

Reviewer comment (Experiments):
Not clear in which dataset the ablation was performed? Is it for both or just one? It should be done for both datasets.

Author response:
We sincerely thank you for the professional comments. We are inspired and hope our discussion brings more insights.
### W1: Ablation study on KITTI
- We conducted ablation study on Waymo open dataset, because it is a more large-scale dataset than KITTI. Besides, there is no validation set on KITTI. The previous works that conducted ablation study on KITTI split each training sequence into two halves, and use the first half frames for training
and the second for validation. The overlapping videos cause inaccurate (too high) results that cannot reflect the actual performance well.
- We also supplement the ablation study on KITTI.
  
|Method|HOTA↑|MOTA↑| FP↓| FN↓ |IDSW↓|
|-|:-:|:-:|:-:|:-:|:-:|
|Baseline| 77.6| 86.2| 6.3%|7.4%| 0.1%|
|+low-quality dets| 78.1|86.5| 7.5%|5.9%| 0.0%|
|+GNN |78.3 |86.5| 7.6%| 5.8%| 0.0%|
|+3D representation|78.6 |86.8| 7.4%| 5.8%| 0.0%|

Reviewer comment (Reproducibility):
It is not stated on which device the inference times are measured. Is it on the same GPU on which it was trained? One GPU or more? How far is it from working in real time? It is not clear and SLAM related applications require working in real time.

Author response:
### W3: Inference time
- We measure the latency on NVIDIA A40 GPU, the same as it was trained on. We use a single GPU to measure the latency. 
- The inference latency is mainly dominated by the object detector, for about 150ms per frame. The tracking stage (feature extraction and object association) takes about 80 ms. If using a real-time object detector, like the speed-optimized YOLO-series detector (YOLOX_x 17.3 ms, YOLOv8x 3.53ms), we believe it will be a (near) real-time system for autonomous driving (sensor input is 10Hz for most autonomous driving systems). Besides, please note the 3D reconstruction is only conducted in pseudo-label generation stage, and it will not affect the inference speed.

Reviewer comment (Experiments):
Why do authors use the specified object detection method instead of something newer like DETR or YOLOv8?

Author response:
### Q1: Object detector
CenterNet is a commonly used detector in MOT, so we chose it as our base detector for a fair comparison with SOTA methods. (Because the detector is not the core concern in our method.)

We also conduct additional experiments on a more modern detector to show the performance with a better detector. We show the results with the YOLOX detector (also widely used recently, especially for ByteTrack-series trackers.)
The detection performance is very well on Waymo open dataset. 
||AP VEHICLE |AP PEDESTRIAN |AP CYCLIST|AP ALL|
|-|:-:|:-:|:-:|:-:|
|YOLOX|64.22|78.36|57.91|66.83|

Based on it, we can achieve better tracking results.
|Method| Backbone| MOTA ↑ |IDF1 ↑| FN ↓ |FP ↓ |ID Sw. ↓ |MT ↑ |ML ↓|
|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|QDTrack (Fischer et al., 2022)| ResNet-50 |55.6 |66.2 |514548 |214998 |24309 |17595 |5559|
|P3DTrack (Ours)| DLA-34 |55.9 |65.6| 638390| 74802| 37478| 13643 |8583|
|P3DTrack_YOLOX (Ours)| Modified CSP v5|59.2|67.6|549228|90342|51913|16475| 7362|

Reviewer comment (Experiments):
How does the system work at night when visibility is reduced? Does the ability of 3D reconstruction remain the same or does it decrease?

Author response:
### Q2: At night
When the visibility is low at night, the SfM system will be negatively affected.  However, it only slightly affects pseudo label generation. The network learning 2D boxes and 3D representation is less affected. We show a video on the anonymous project page.

Reviewer comment (Novelty):
The novelty of the paper is limited. The main idea is to generate pseudo 3D object labels from monocular videos so that it can be used to train the model to obtain 3D location / representation. There are main issues and details need to address about the process of generating these pseudo labels. Thus, in terms of the methodology for MOT, there is not much new development in this paper, e.g. GNN-based aggregation, association model learning and appearance using reID feature, etc. are the existing techniques in MOT literature.

Author response:
### W1: Limited novelty
- Our main novelty and contribution lie in the 3D representation. Specifically, We focus on how to obtain 3D labels, how to generalize the label to other objects, and how to use the (not so accurate) 3D features in MOT. The experiments also show that our 3D representation yields a substantial improvement compared to traditional 2D representation and other pretrained 3D features. As for technical contribution, to obtain 3D labels, we design the two-stage object clustering (IntraPC and InterPC). To generalize the label to other objects, we propose an image-based 3DRL module learning from all 2D labels and static 3D labels, and verify that this learning module can generalize the static labels to dynamic objects. To measure the degree of inaccuracy, we introduce uncertainty-based 3D loss.
We believe this learning paradigm can inspire the MOT community to pay attention to mining the 3D information hidden in the video that does not require manual annotation. 
- The feature aggregation and association modules are not the main contribution in our paper. For the online "Tracking by Detection" paradigm, most methods share a very similar association module, i.e., encoding the appearance/motion features, aggregating the object features, matching between detections and tracklets. This pipeline design can even back to DeepSORT published in 2017, and further adopted and slightly modified by the following work GMTracker [A], ByteTrack [B], BoT-SORT [C], etc. In this paper, we only utilize the best practices in the data association stage to obtain better performance and do not focus on the novel design in this part.
- About the details in 3D pseudo label generation, we try our best to solve your concern in the following.

Reviewer comment (Novelty):
The impact of the paper is limited given the ego-centric datasets with LiDAR data widely available.

Author response:
### W2: Limited impact
- We agree that most **existing** ego-centric datasets have LiDAR sensors. However, with the development of embodied AI and vision-based autonomous driving, hand-held cameras and cameras on vehicles without LiDAR will be widely used. For example, using DJI drones, driving Tesla autonomous-driving vehicles, and biking with a GoPro camera will provide more available data.  We believe our algorithm will have great potential in these fields.
- Besides, we also report the additional results on the general non-ego-centric dataset MOT17 test set. The results show that even though training 3D representation on only half of the videos that have ego-motion, we still obtain performance improvement, especially compared with pretrained depth-based tracker QuoVadis [D]. Please note in this Table, we change our base detector to YOLOX to align the detector performance with SOTA methods.
  
|Method| MOTA↑| IDF1↑| HOTA↑| FP↓| FN↓| IDSw↓|
|-|:-:|:-:|:-:|:-:|:-:|:-:|
|ByteTrack| 80.3| 77.3| 63.1 |25491| 83721 |2196|
|QuoVadis|80.3|77.7|63.1|25491|83721|2103|
|BoT-SORT| 80.5	|80.2	|65.0	|	22521	|86037|1212|
|P3DTrack (Ours)|80.6 |79.8 |64.9 |22119 |86061 |1197|

[A] He et al. Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking. In CVPR 2021.

[B] Zhang et al. ByteTrack: Multi-Object Tracking by Associating Every Detection Box. In ECCV 2022.

[C] Aharon et al. BoT-SORT: Robust Associations Multi-Pedestrian Tracking. arXiv 2206.14651.

[D] Dendorfer et al. Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking? In NeurIPS 2022.

Reviewer comment (Theory):
How can the model learn if there is only loss to supervised static object? The output o^t_j can be any values.

Author response:
### Q2: Learning from static object
Just as mentioned in response to Q1, the 3DRL module learns object's 3D representation from a single image. Like monocular 3D object detector, on the head of CenterNet, we regress the object's 3D position ($H\times W\times 3$) from $H\times W$ feature map. However, the supervision is only on the 2D center of static objects. As mentioned in response to Q1, we learn class-agnostic 3D representation from monocular image, and in an image, moving and static objects are not different in appearance. Moving and static objects share the same prediction head, so $o^t_j$ will not be any values for moving objects.

Reviewer comment (Evaluation):
In table 3, there is an increase of ID Sw when using Baseline + 3D representation. How do you explain this behaviour? Is it suppose for 3D representation to help reduce ID Sw.

Author response:
### Q4: Increase of ID Sw
- It is because in the last row, the object recall is much higher than the baseline. Please refer to the FP and FN metrics in Table 3.
ID Switches can only be compared under similar recall performance. In general, the more objects considered in tracking, the more ID Switches are reported.
Besides, IDF1 is also the metric to reflect the identification performance. We obtain 5.8 IDF1 improvement (68.1 vs. 62.3).
- 3D representation can help reduce ID Sw. Please refer to Table 4. In this Table, we only change the 2D/3D representation without modifying other modules and thus maintain a similar recall, so it can reflect the influence of our 3D representation better.