Paper: CO-MOT: Boosting End-to-end Transformer-based Multi-Object Tracking via Coopetition Label Assignment and Shadow Sets (ICLR-2024)

Reviewer comment (Experiments):
The performance is not stable across different datasets. For instance, the tracker cannot even achieve comparable performance with existing trackers such as GTR/PA3Former/GRTU on MOT17, which is by far the most widely used and trusted benchmark within MOT community. While authors claim that MOT17 is rather a small dataset to train transformer-based trackers, other ways such as using synthetic dataset [a] could be one option. Also the proposed tracker performs worse than the major baseline MOTRv2, which is also transformer-based, in terms of IDF1 and HOTA. In this case, it is a bit hard to justify that authors are making a direction towards improving MOTRv2. In fact, I would stay away with a rather heavier tracker (Transformer with multiple layers and many parameters) that gives even worse performance on MOT17 than light-weight tracking-by-detection/regression based trackers that require way less data to train, while still work better on major tracking benchmarks.

Author response:
1.1  Due to the limited time of the rebuttal, we will supplement the results adding synthetic dataset.

1.2 MOTRv2 is transformer-based, but due to the addition of YOLOX preprocessing, it has a larger computational load and is not a pure End-to-end method. The original intention of this paper is to maintain the End-to-end structure and computational load of MOTR and achieve performance comparable to MOTRv2. The strong performance in Table 2 has well supported our motivation on multiple datasets.

1.3  The table below lists the performance of known End-to-end methods on the MOT17 test. It can be seen that End-to-end tracking has been prevalent in the past several years, and its performance has gradually improved. Currently, End-to-end MOT methods are still in the early stages of development, and we believe that there will be significant improvements in the future. The value of a method should not be denied just because it has not achieved state-of-the-art performance on a particular dataset. For example, although DETR and Deformable DETR have lower performance on the COCO dataset compared to other non-Transformer detection methods, their contributions to subsequent methods cannot be ignored.

| |Source|HOTA| DetA| AssA| MOTA| IDF1|
|---|---|---|---|---|---|---|
|MeMOT          | CVPR’22|56.9| - |55.2| 72.5| 69.0|
|TrackFormer    |CVPR’22| - | - | - |65.0| 63.9|
|***MQT***            |arXiv’22|  - | - | - |66.5| 65.2|
|***DNMOT***          |arXiv’23|58.0|-|-|75.6|68.1|
|MOTR           |ECCV’22|57.8| 60.3| 55.7| 73.4| 68.6|
|MeMOTR         |ICCV’23| 58.8| 59.6 |58.4| 72.8| 71.5|
|***MOTRv3 (ConvNeXt-B)***| arXiv’23 |60.1| 58.7|62.1|75.9| 72.4|
|CO-MOT         |-      | **60.1**| 59.5| 60.6| 72.6| **72.7**|

**Table 2 (c)**:  Comparison to End-to-end methods on the MOT17 test dataset.  The methods with ***bold italic*** fonts are newly introduced. CO-MOT has achieved the highest HOTA(60.1). Although the HOTA performance of MOTRv3 is the same as CO-MOT, it uses a more powerful backbone (ConvNeXt-B).

Reviewer comment (Theory):
For the idea of using shadow set, too many engineering tricks such as min, max, operations are used. It would be much better if authors could come up with some mathematical formulations as a technically sound approach to replace these simple heuristics, so as to better justify the use of this method. For example, IMHO, how about borrowing ideas from particle filter, to use importance sampling to maintain a set of queries to track each object?

Author response:
Thank the reviewer for the provided advice to make this work more technically solid. "min" and "max" used to select the samples for training are heuristic but effective. More exploration such as importance sampling for sample selection will be conducted in our future work.

Reviewer comment (Theory):
For the tracking queries and the detection queries, they both self-attend each other and then cross-attend the image feature tokens. Intuitively, all the queries can perceive the whole image, and it is not good enough to split them into two parts, which may degrade the detection results.

Author response:
In the multi-object tracking (MOT) task, it is necessary to keep tracking the targets that are tracked in the previous frame, while also detecting newborns that appear in the current frame. The tracking queries and detection queries play different roles in these tasks. During training, they are supervised by different supervision, where the detection queries use Hungarian matching, and the tracking queries use TALA. During inference, the tracking queries and detection queries are concatenated together and indiscriminately passed through multiple Transformer decoders. Thus, all the queries can perceive the whole image, as the reviewer points out, but both tracking and detection queries must exist for MOT.

Reviewer comment (Experiments):
Why not evaluate the proposed method on the crowded dataset MOT20? In crowd scenes, the tracking query has the similar semantic information with the detection query, and may fail to assign correct label in crowd scenes.

Author response:
As the End-to-End solution has just emerged in the past year, there are not many methods evaluated on MOT20 that we could find. Here are the ones in Table 2 (d). Notably, our approach achieves 57.5% HOTA, which is the state-of-the-art in End-to-end tracking methods.

| |Venue |HOTA| DetA| AssA| MOTA| IDF1|
|---|---|---|---|---|---|---|
|MeMOT | CVPR’22 | 54.1| \ | 55.0 | 63.7| 66.1| 
|TrackFormer|CVPR’22| 54.7| \ | \ |  68.6| 65.7 |
|CO-MOT| -| **57.5** | 50.5 | **65.7** | 60.1 |	**70.5**|

**Table 2 (d)**: Comparison to existing methods  on the MOT20 test dataset.

Reviewer comment (Experiments):
In Table 1, the authors point out that the detection result of MOTR can be improved by removing the tracking queries. How about the proposed CO-MOT? If CO-MOT is only used as a detection task, and whether the tracking performance can be further improved?

Author response:
The following table lists the mAP indicators, which are consistent with the DetA indicators in Table 2(a). The detection performance of CO-MOT is far superior to MOTR but slightly inferior to MOTRv2. 

Tracking tasks not only include detecting new targets but also need to track targets that have already been tracked, maintaining the identity of the targets between consecutive frames. Therefore, CO-MOT cannot be used only as a detection task, otherwise, it would not be able to associate targets between previous and subsequent frames.

| method | mAP|
|---|---| 
|(a) MOTR | 42.5 |
|(b) MOTRv2|  70.7 |
|(c) CO-MOT|  69.1|

**Table 6**:  The detection performance (mAP) of MOTR (v2) and CO-MOT on DanceTrack validation dataset.

Reviewer comment (Experiments):
It is better to evaluate the effect of the number of decoders L on the tracking performance.

Author response:
We ablate the number of decoders and illustrate the results in Table 7. The results show that the HOTA consistently improves as the number of decoders increases, while other metrics significantly increase or are comparable. To make a fair comparison with MOTR and consider the efficiency, we choose six decoders by default in all the experiments.

|  L |HOTA| DetA| AssA| MOTA| IDF1|
|---|---|---|---|---|---|
|L=2|49.0  |  51.9 |   46.6 | 53.8 |   53.2|
|L=4|58.2  | 71.7  |  47.5 | 80.8| 60.7|
|L=6|61.0  |  73.9  |  50.6| 82.7 |62.6|
|L=8|61.5 |  77.0 |   49.3 | 86.6| 62.6 |

**Table 7**:  Ablation study on the number of decoders L. As L increases, the computational load increases, and the HOTA tracking performance improves, but the growth rate of HOTA significantly decreases.

Reviewer comment (Presentation):
Pease reorganize the formats of the references, and make them consistent.

Author response:
We reorganize the formats of the references according to the official regulations in the manuscript.

Reviewer comment (Experiments):
Missing experiments on more models: In Section 3.2, this paper mentions that we revisit the Tracking Aware Label Assignment (TALA) used to train end-to-end Transformers such as MOTR and TrackFormer for MOT. However, all the ablations are conducted on MOTR. I cannot find evidence that COLA and shadow work on other models like TrackFormer. The authors should provide results of TrackFormer on two datasets at least to make the proposed solutions more solid.

Author response:
In the supplementary materials, we provide the results of COLA and Shadow Set on Trackformer. We do not make any modifications to the TrackFormer framework and used the default hyperparameters provided by TrackFormer. The result shown as follows demonstrates the effectiveness of COLA (+6.4\%) and Shadow Set (+9.3\%) when applied to TrackFormer. Codes will be released upon acceptance.

| |COLA|	Shadow|	HOTA|
|---|---|---|---|
|(a) TrackFormer| | | 41.4 |
|(b) ours | ✓ | | 47.8 (+6.4) |
|(c) ours  |✓ | ✓ |50.7 (+9.3) |

**Table 4**:  The ablation experiments of applying CO-MOT to TrackFormer on the DanceTrack validation set. As components are added, the tracking performance improves gradually.

Reviewer comment (Evaluation):
Comparison of failure cases: The failure cases shown in Figure 5 are the common cases of all the Transformer-based approaches. The authors should provide the failure cases of previous approaches and denote which kind of case has been solved by CO-MOT.

Author response:
The Figure 7 illustrates the failure cases of MOTR and denotes which kind of case has been solved by CO-MOT. MOTR has poor detection and tracking performance. First, as shown in Figure 7 (a), it fails to detect the person in time under the extreme case of bending over. Second, as shown in Figure  7 (b), due to the tiny visual features when a person stretches out their hand, the detection box is inaccurate, and the model misidentifies it as multiple people. Third, as shown in Figure  7 (c), the tracking identity switches after the human body is obscured. However, all of the above cases can be solved by CO-MOT, showing the extraordinary performance of MOT.

Reviewer comment (Presentation):
In Figure 4, do the FLOPs of the MOTR include the YOLOX detector? What are the specific numbers of parameters for different approaches? The specific number of parameters should be indicated in Figure 4. For example, the number can be placed in circles.

Author response:
We show the specific numbers in the following table and indicate them in Figure 4.

| | params| FLOPs| Inference FPS| HOTA|
|---|---|---|---|---|
|MOTR| 40M | 173G| 19| 54.2|
|MOTRv2| 139.1M | 454.9G | 14| 69.9|
|CO-MOT| 40M | 173G| 19 | 69.4|

**Table 5**: The parameters, model size, and FLOPs of MOTR, MOTRv2, and CO-MOT.

Reviewer comment (Presentation):
The layout of the article is quite chaotic. Figure 1 is used as an example in the method introduction, but it is too far from the text information. For another example, in Section 3.2 of the introduction of strategies and methods, formulas are mixed in a large paragraph without a comprehensive formula or image description, making it difficult for readers to understand. The reviewer must refer to the paper of the baseline tracker MOTR for a comprehensive understanding.

Author response:
Thank you for your constructive feedback on the paper organization. We have updated the manuscript according to your suggestions to improve its readability and comprehensibility. For example, we move Figure 1 near the method section so that readers can refer to it more conveniently. Additionally, we demonstrate the symbols of Section 3.2 in Figure 2. To highlight the technical improvements of the new approach, we add a detailed figure of MOTR (v2) and CO-MOT in Figure 6. Note that the stacked circles and squares denote the shadow sets.

Reviewer comment (Presentation):
The authors should re-organize the method introduction and include a detailed figure to highlight the technical improvements of the new approach, including both the label assignment and shadow set.

Author response:
Thank you for your constructive feedback on the paper organization. We have updated the manuscript according to your suggestions to improve its readability and comprehensibility. For example, we move Figure 1 near the method section so that readers can refer to it more conveniently. Additionally, we demonstrate the symbols of Section 3.2 in Figure 2. To highlight the technical improvements of the new approach, we add a detailed figure of MOTR (v2) and CO-MOT in Figure 6. Note that the stacked circles and squares denote the shadow sets.

Reviewer comment (Experiments):
The comparisons with more tracking approaches on the BDD100k or DanceTrack datasets are required to demonstrate the performance improvement.

Author response:
We demonstrate tracking methods as many as possible in the supplementary materials, which are evaluated on DanceTrack. The methods with ***bold italic*** fonts are newly introduced. For BDD100K, we have already shown all the methods evaluated in Table 2(b) so that no more methods are introduced.


| |Source | HOTA   | DetA | AssA | MOTA | IDF1 |
|---|---|---|---|---|---|---|
***SORT***          |ICIP’16 |47.9| 72.0 |31.2| 91.8 | 50.8|
***CenterTrack***    | ECCV’20|41.8| 78.1| 22.6| 86.8| 35.7|
TransTrack     |arXiv’20| 45.5| 75.9| 27.5| 88.4| 45.2|
***FairMOT***        |IJCV’21| 39.7| 66.7| 23.8| 82.2| 40.8|
QDTrack        | CVPR’21|54.2| 80.1| 36.8| 87.7| 50.4|
***TraDeS***         | CVPR’21| 43.3| 74.5| 25.4| 86.2| 41.2|
ByteTrack      | ECCV’22| 47.7| 71.0| 32.1| 89.6| 53.9|
OC-SORT        | CVPR’23| 55.1| 80.3| 38.3| 92.0| 54.6|
***GTR***            |CVPR’22| 48.0| 72.5| 31.9| 84.7| 50.3|
***C-BIoU***         | WACV’23 |60.6| 81.3| 45.4| 91.6| 61.6|
***MT IoT***         | arXiv’22| 66.7| 84.1| 53.0| 94.0| 70.6|
DiffusionTrack |arXiv’23|52.4|82.2 |33.5| 89.3| 47.5|
MOTRv2+        | CVPR’23| 69.9| 83.0| 59.0| 91.9| 71.7|
|---|---|---|---|---|---|---|
MOTR           |ECCV’22| 54.2| 73.5| 40.2| 79.7| 51.5|
 ***DNMOT***          |arXiv’23|53.5|-|-|89.1|49.7|
MeMOTR         |ICCV’23| 63.4| 77.0| 52.3| 85.4| 65.5|
MeMOTR*        |ICCV’23| 68.5| 80.5| 58.4| 89.9| 71.2|
***MOTRv3 (Res50)***  |arXiv’23|68.3| -|-|91.7| 70.1|
CO-MOT         |-      | 65.3| 80.1| 53.5| 89.3| 66.5|
CO-MOT+        |-      | **69.4**| **82.1**| **58.9**| 91.2| **71.9**|

**Table 2 (a)**:  Comparison to existing methods on the DanceTrack test set. The last few lines are all end-to-end tracking methods, among which CO-MOT has achieved the highest HOTA.