Paper: Towards Category Unification of 3D Single Object Tracking on Point Clouds (ICLR-2024)

Reviewer comment (Novelty):
You should discuss relative works on your key idea (e.g., encoding shape- and size-changed geometric information). Relative works include [a] (adaptive region learning), [b] (dynamic ball-query selection for size, dynamic foreground and background learning), [c], etc. [a] Pyramid r-cnn: Towards better performance and adaptability for 3d object detection. [b] DBQ-SSD: Dynamic Ball Query for Efficient 3D Object Detection. [c] RBGNet: Ray-Based Grouping for 3D Object Detection.

Author response:
According to your insightful comment, we have discussed related works on our key idea to give a better understanding of our work. Our core idea is to encode adaptive group configurations to accommodate shape- and size-changed geometric information from cross-category data. We appreciate your observation regarding the connection between the mentioned references and our core idea, specifically the dynamic receptive field. We have added a subsection, as shown in the $\textbf{“2.3 Dynamic Grouping” subsection of the main manuscript on page 3, line 136}$.

Reviewer comment (Experiments):
The comparison of latency should include other methods that have been reported, e.g., [c], [d], etc. In addition, the recent SOTA SOT methods should also be included. [c] Beyond 3d siamese tracking: A motion-centric paradigm for 3d single object tracking in point clouds. [d] Implicit and Efficient Point Cloud Completion for 3D Single Object Tracking.

Author response:
According to your comment, we have conducted a comparison of latency between our unified models and eight state-of-the-art methods (including the methods you mentioned), aiming to give a comprehensive assessment of the proposed methods in comparison to existing methods. In addition, we have also included a comparison of model parameters and FLOPs. Our methods introduce slight overhead compared to the baseline methods. For a detailed comparison and analysis, please refer to the $\textbf{“B.4 Computational Cost Analysis” subsection of the Appendix on page 16, line 585}$.  

To better demonstrate the effectiveness of the proposed category-unified models, we have compared the recent SOTA methods, such as TAT [ACCV2022], STNet [ECCV2022], DMT [T-ITS2023], CXTrack [CVPR2023] and SyncTrack [ICCV2023] for comparison, as shown in the $\textbf{“4.2 Comparison with State-of-the-art Methods” subsection of the main manuscript on page 7, line 295}$.

Reviewer comment (Experiments):
Why not verify on Waymo dataset?

Author response:
In the community, KITTI and NuScenes are often employed for a comprehensive evaluation of a tracker. Specifically, tracking models are typically trained and tested on each category within these two datasets. In contrast, Waymo is often utilized to assess the generalization capabilities of the models. A common practice is to test a model pretrained from KITTI on the Waymo dataset. To further validate the generalization of our unified model and enhance the quality of our work, we have evaluated the proposed category-unified model on Waymo, following common experiment setup. Detailed experiments and analyses are provided in the $\textbf{“B.3 Results on Waymo Open Dataset” subsection of the Appendix on page 16, line 571}$.

Reviewer comment (Experiments):
Can Adaformer be extended to general detection?

Author response:
Thanks for your meaningful comment. We believe that the concept of AdaFormer holds potential for extension to general detection. To the best of our knowledge, 3D point cloud-based object detection can be broadly categorized into indoor and outdoor (autonomous driving scenarios) detection. From a model perspective, detection methods can be further classified into those relying on point representation and those utilizing voxel representation. Currently, in indoor scenes, methods based on point representation dominate, while in outdoor scenes, voxel representation-based methods prevail due to the computational challenges posed by large-scale scenarios for point representation-based methods. Our AdaFormer is a point representation-based feature extraction network. Its design philosophy may be extended to point representation-based detection methods, introducing dynamic receptive fields to enhance detection accuracy. For detection methods based on voxel representation, our approach may also offer insights, such as dynamically adjusting voxel sizes to enhance feature representation for objects with diverse shapes and sizes.

Reviewer comment (Evaluation):
Are there any inference speed and FLOPs comparison for proposed method?

Author response:
Thanks very much for your positive feedback, we have polished our paper according to your valuable suggestions. We have incorporated a comparison of inference speed and FLOPs to enhance the quality of our work. We have also presented the model parameters. The proposed category-unified models introduce a tolerable additional inference cost and computational overhead compared to baseline methods (For example, P2B [48 Fps, 4.30 GFLOPs, 1.34M parameters] v.s. SiamCUT [36 Fps, 4.51 GFLOPs, 2.06M parameters]), mainly incurred by the proposed AdaFormer backbone. Our unified model inputs and learning objective do not incur any additional overhead during the testing phase. Detailed comparisons and analyses are provided in the $\textbf{“B.4 Computational Cost Analysis” subsection of the Appendix on page 16, line 585, which is highlighted in \textcolor{red}{red} in the revised manuscript}$.

Reviewer comment (Experiments):
The proposed approach can track objects across all categories using a single network with shared parameters. But as I mentioned above, there is no training details about the unified training. Is the proposed only trained on the full KITTI and then test on it (the same for Nuscenes)? or the combination of KITTI and Nuscenes are used? Please all more illustration in the paper.

Author response:
(1) Details of unified training.
In our experimental configuration, the proposed unified model is trained on the full KITTI and then test on it, the same for NuScenes, not the combination of KITTI and NuScenes. More specifically, the experimental setup commonly used in the community is to train and test each category individually within each dataset. To be a fair comparison, our unified model is trained on all training data (cross-category data) of a dataset, and then tested on all test data (diverse categories) of that dataset. In addition, during the training phase, we directly use data from all categories to train a unified model, without any additional data augmentation techniques. We appreciate the insightful comment and have made improvements to the manuscript by supplementing training details, as shown in the $\textbf{“A More Implementation Details” section of the Appendix on page 13, line 497}$. 

Further discussion: There are notable differences among various datasets. For example, NuScenes displays more pronounced sparsity compared to KITTI and Waymo. We posit that cross-dataset training requires addressing challenges related to data distribution. This may include dealing with diverse background and motion state distributions, such as differences in pedestrian behavior between KITTI and NuScenes. The concept of joint cross-dataset and cross-category training will enable a single model to track all objects across all categories, irrespective of data distribution, which further enhances the model's generalization capabilities. In the future, we will keep it tune and further improve our unified model.

Reviewer comment (Experiments):
The main concern in this paper is about the lack of Waymo dataset evaluation and missing recent approaches for comparison. Please also include the latest references below for comparison, in order to better verify the effectiveness of the proposed approach.

[1] Temporal-aware Siamese Tracker: Integrate Temporal Context for 3D Object Tracking. ACCV 2022.  
[2] 3D Siamese Transformer Network for Single Object Tracking on Point Clouds. ECCV 2022.  
[3] CXTrack: Improving 3D Point Cloud Tracking with Contextual Information. CVPR 2023.  
[4] A Lightweight and Detector-Free 3D Single Object Tracker on Point Clouds. IEEE Transactions on Intelligent Transportation Systems. 2023.

Author response:
(2) Evaluation on Waymo.
According to your comment, we have evaluated the proposed method on Waymo. Unlike KITTI and NuScenes, the community often conducts generalization experiments on Waymo dataset. Following this, we test the KITTI pre-trained model on Waymo. We also compare our category-unified models with existing SOTA methods, including the recent methods you mentioned. The experimental results demonstrate that the proposed category-unified model not only generalizes well across diverse categories but also exhibits a certain level of generalization to datasets with varied data distributions. Detailed experiments and analyses are provided in the $\textbf{“B.3 Results on Waymo Open Dataset” subsection of the Appendix on page 16, line 571}$.

(3) Recent methods comparison.
To better verify the effectiveness of the proposed approach, we have included all the recent methods you mentioned and the latest SyncTrack [ICCV2023] for comparison, and added the corresponding comparison analysis, as shown in the $\textbf{“4.2 Comparison with State-of-the-art Methods” subsection of the main manuscript on page 7, line 295}$. In addition, we also compare our category-unified models with recent SOTA methods on Waymo, as shown in the $\textbf{“B.3 Results on Waymo Open Dataset” subsection of the Appendix on page 16, line 571}$.

Reviewer comment (Experiments):
The proposed approach can track objects across all categories using a single network with shared parameters. But as I mentioned above, there is no training details about the unified training. Is the proposed only trained on the full KITTI and then test on it (the same for Nuscenes)? or the combination of KITTI and Nuscenes are used? Please all more illustration in the paper.

Author response:
(1) Details of unified training.
In our experimental configuration, the proposed unified model is trained on the full KITTI and then test on it, the same for NuScenes, not the combination of KITTI and NuScenes. More specifically, the experimental setup commonly used in the community is to train and test each category individually within each dataset. To be a fair comparison, our unified model is trained on all training data (cross-category data) of a dataset, and then tested on all test data (diverse categories) of that dataset. In addition, during the training phase, we directly use data from all categories to train a unified model, without any additional data augmentation techniques. We appreciate the insightful comment and have made improvements to the manuscript by supplementing training details, as shown in the $\textbf{“A More Implementation Details” section of the Appendix on page 13, line 497}$. 

Further discussion: There are notable differences among various datasets. For example, NuScenes displays more pronounced sparsity compared to KITTI and Waymo. We posit that cross-dataset training requires addressing challenges related to data distribution. This may include dealing with diverse background and motion state distributions, such as differences in pedestrian behavior between KITTI and NuScenes. The concept of joint cross-dataset and cross-category training will enable a single model to track all objects across all categories, irrespective of data distribution, which further enhances the model's generalization capabilities. In the future, we will keep it tune and further improve our unified model.

Reviewer comment (Experiments):
The main concern in this paper is about the lack of Waymo dataset evaluation and missing recent approaches for comparison. Please also include the latest references below for comparison, in order to better verify the effectiveness of the proposed approach.

[1] Temporal-aware Siamese Tracker: Integrate Temporal Context for 3D Object Tracking. ACCV 2022.  
[2] 3D Siamese Transformer Network for Single Object Tracking on Point Clouds. ECCV 2022.  
[3] CXTrack: Improving 3D Point Cloud Tracking with Contextual Information. CVPR 2023.  
[4] A Lightweight and Detector-Free 3D Single Object Tracker on Point Clouds. IEEE Transactions on Intelligent Transportation Systems. 2023.

Author response:
(2) Evaluation on Waymo.
According to your comment, we have evaluated the proposed method on Waymo. Unlike KITTI and NuScenes, the community often conducts generalization experiments on Waymo dataset. Following this, we test the KITTI pre-trained model on Waymo. We also compare our category-unified models with existing SOTA methods, including the recent methods you mentioned. The experimental results demonstrate that the proposed category-unified model not only generalizes well across diverse categories but also exhibits a certain level of generalization to datasets with varied data distributions. Detailed experiments and analyses are provided in the $\textbf{“B.3 Results on Waymo Open Dataset” subsection of the Appendix on page 16, line 571}$.

(3) Recent methods comparison.
To better verify the effectiveness of the proposed approach, we have included all the recent methods you mentioned and the latest SyncTrack [ICCV2023] for comparison, and added the corresponding comparison analysis, as shown in the $\textbf{“4.2 Comparison with State-of-the-art Methods” subsection of the main manuscript on page 7, line 295}$. In addition, we also compare our category-unified models with recent SOTA methods on Waymo, as shown in the $\textbf{“B.3 Results on Waymo Open Dataset” subsection of the Appendix on page 16, line 571}$.