Paper: Transferring Labels to Solve Annotation Mismatches Across Object Detection Datasets (ICLR-2024)

Reviewer comment (Writing):
I had problem following the scope of the paper.

Author response:
In the following, we answer the reviewer's concern with the scope of the paper:
1. **Confusion with the introduction and Figure 1**: We introduce the label translation problem as a general and prevalent type of mismatches among object detection datasets. While resolving arbitrary label mismatches is extremely non-trivial, we consider our streamlined version of addressing specifically bounding box mismatches as the necessary first step in this novel line of research. Fortunately, our metric of concern (target performance) provides strong evidence that solving the simpler problem remains effective. We would appreciate if the reviewer could expand on any points that they feel we haven't addressed.

Reviewer comment (Presentation):
The paper introduces a very general alignment setting in Introduction + Figure 1, which is not matched with what is really performed. I had to make several loops between Method & Introduction + Figure 1 to understand what is going on.

Author response:
In the following, we answer the reviewer's concern with the scope of the paper:
1. **Confusion with the introduction and Figure 1**: We introduce the label translation problem as a general and prevalent type of mismatches among object detection datasets. While resolving arbitrary label mismatches is extremely non-trivial, we consider our streamlined version of addressing specifically bounding box mismatches as the necessary first step in this novel line of research. Fortunately, our metric of concern (target performance) provides strong evidence that solving the simpler problem remains effective. We would appreciate if the reviewer could expand on any points that they feel we haven't addressed.

Reviewer comment (Novelty):
"1) We consider scenarios where the class label spaces are matched but the annotation protocols are different, therefore, leading to annotation mismatches (see Section 3). 2) We optimize the target performances, while multi-dataset detection optimizes the average performances of all datasets considered." => The authors should provide convincing arguments + results on why this narrowed-down scope of the mismatch problem is significant.

Author response:
2. **Narrowing down the scope of the mismatch problem**: 
    - We argue that optimizing only the target dataset performance is not narrowing down from multi-dataset object detection. Instead, they have distinct values depending on the application. Optimizing target dataset performance is particularly important when there is a specific target domain of interest, such as in Sim2Real. The performance on simulated data is relatively unimportant and only serves the purpose of improving target performance on real-world data.
    - While we narrowed down the mismatch problem to a shared class label space, LGPL can handle the setting when two datasets have only partially overlapped categories, by translating the overlapped categories and leaving the other as is.

Reviewer comment (Theory):
"In this work, we assume that (1) the class labels are the same between the source and the target labeling functions and (2) the source labeling function either detects or over-detects all the objects specified in the target labeling function." => Again, the validity of these assumptions should be justified.

Author response:
3. **Assumptions in Section 4.1**: The assumptions are made based on the observation from seven datasets surveyed in Section 3. 
    - To further justify the second point “the source labeling function either detects or over-detects all the objects specified in the target labeling function”, we manually collect the gold translated labels and perform TIDE analysis to understand the types of translation required (See the details in Section 5.1 and Section A.3). The results in Table 9 show the low value of “Missed error” and high value of “Background error”, which supports the need to fix background errors.

Reviewer comment (Experiments):
"We optimize the target performances, while multi-dataset detection optimizes the average performances of all datasets considered." => Why is it not better to have a multi-dataset version?

Author response:
4. **A multi-dataset version**: As pointed out in the second point (`Narrowing down the scope of the mismatch problem`), performing multi-dataset object detection has distinct application values that differ from our goals in this work.

Reviewer comment (Writing):
"label translation" is misleading. With the general coverage in the Introduction, I expected language translation to be performed in the method. "Recall from our main assumption that label translation does not require generating a correct class label, but only determining whether the object should be labeled as well as the corresponding bounding box." => I have difficulty calling this translation.

Author response:
5. **The term "label translation"**: We thank the reviewer for the thoughtful concerns. The confusion with language translation is unfortunate, and we are keen to hear suggestions on how this might be mitigated in future.

Reviewer comment (Experiments):
Experimental evaluation is limited. It is not clear why YOLOv3 is chosen, compared to its newer versions or other one-stage detectors?

Author response:
**Empirical evaluation**

The reviewer asked why we chose YOLOv3 instead of newer variants or one-stage detectors. In fact, we specifically choose three different types of object detectors: two-stage object detector (Faster-RCNN), one-stage object detector (YOLOv3), and transformer-based object detector (Def-DETR). 

We choose YOLOv3 out of three YOLO variants in MMdetection due to its stable training. But in any case, we feel that we have adequately demonstrated the robustness of our technique to different object detectors. If the reviewer would like to see the performance on another detector, please kindly provide us with a reference and we will aim to incorporate these results into a later version.

We would also want to emphasize that we have tested on four different scenarios with three different architectures as well as ablations with model-centric alternatives such as domain adaptation and foundation model alternatives such as SAM.

Reviewer comment (Evaluation):
Most importantly, as bbox localization changes through the proposed method, the paper should quantify localization directly and evaluate performance with such a measure. I would recommend the localization term of the Localization Recall Precision metric.

Author response:
**Suggestions of LRP metric**

We appreciate the reviewer's great suggestion of the LRP metric and we will include the LRP metric in our camera-ready version.

Reviewer comment (Experiments):
After the rebuttal, the authors have been able to partially address my concerns about the significance of the taxonomy and the experimental evaluation. For taxonomy to be considered a contribution, I expect more analyses & justifications. The experiments with a SOTA model should also be more comprehensive as YOLOv8-nano is hardly a SOTA model. However, the authors did the best of what could be done in a short amount of time and I find the extended discussion on the taxonomy and the new results promising. Therefore, I would like to increase my original recommendation.

Author response:
**Justifying the taxonomy**: We are happy to pursue any suggestions for how we may better validate the taxonomy. Our revision clarifies the following points.


The taxonomy is proposed not to exhaustively list out every possible annotation mismatch, but to provide a deeper understanding to the problem and we expect future works can build on top of the taxonomy. For example, currently in Table 7, we compare 14 dataset pairs and characterize which mismatches exist with each pair.


It is challenging to measure how many examples suffer from certain types of mismatches since ideally, we would need to manually count the number/type of mismatches for each pair of datasets. However, this is beyond scope since it is comparable to relabeling the entire datasets. Instead in Table 8 and 9, we have manually relabeled 300 images from nuScenes, MVD, and nuImages according to each other's label styles. The taxonomy relates the annotation mismatches with the difference in bounding boxes. For example, Table 7 and the TIDE analysis in Table 9 together reveal that the cross-modality (CM) mismatch in nuScenes $\rightarrow$ nuImages leads to severe bounding box shift, but relatively little over-detection issues. 


**Stronger one-stage object detector and SOTA object detector**: We agree that applying LGPL on SOTA object detectors may potentially yield smaller gains, as it is natural to observe a ceiling effect as the base object detectors get stronger. However, we emphasize that this work is not about building a SOTA algorithm, but showing that our pre-processing can *consistently yield some gains essentially for free*.


**New experiments with stronger one-stage object detector**: We have performed new experiments using [YOLOv8-nano](https://github.com/open-mmlab/mmyolo/tree/main/configs/yolov8) [1], as our evaluating object detector for nuScenes $\rightarrow$ nuImages. The findings here align with our results in Section 5.3: 1)LGPL *consistently* outperforms all the baselines out-of-the-box and 2) Translating labels via LGPL leads to higher-quality object detectors.


|                | YOLOv8-nano Downstream-mAP | YOLOv8-nano Downstream-AP$^{75}$ |
|----------------|----------------|----------------|
| No-translation | 20.5          | 19.7               |
| PL             |  16.3         | 16               |
| PL \& NF       | 20.7          | 20.8               |
| LGPL           | 25           | 25.2               |


To finish these runs in time, we only trained on 1 random seed, reduced the image resolution to [1024, 512], and used default hyperparameters for all settings. Our original experiments had 3 seeds, a resolution of [1500, 800], and extensive hyperparameter grid searching. We plan to complete these results on YOLOv8 in the camera-ready version.


[1] YOLOv8, https://github.com/ultralytics/ultralytics

Reviewer comment (Experiments):
The assumption is too strong that the source and the target datasets have the same categories. In multi-dataset training, it is a common case that different datasets not only have overlapped categories but also have unique categories. I believe it's more important to handle unique categories that are annotated in one dataset but not in the others. That's because unannotated objects in the other datasets will be regarded as background by mistake. It's better to have a solution to handle both overlapped and unique categories.

Author response:
**Handling unique categories that are in one dataset but not in the others**


This is a great point. Indeed, LGPL can handle the setting when two datasets have only partially overlapped categories, by translating the overlapped categories and leaving the others as is.

To specifically translate unique classes, we note that LGPL is model/training-agnostic and can be applied with any other data augmentation or training algorithm. For example, if the target dataset has unique categories, we can use any off-the-shelf semi-supervised learning to learn the unique categories. If the source dataset has some unique categories, we can just drop those labels.

Reviewer comment (Experiments):
RoI head is able to refine various region proposals to the target-style boxes. Is it necessary to train a RPN as box generator that generates source-style boxes? It's better to compare the proposed solution with a only-RPN solution. That is, train a RPN (probably together with the two-stage architecture) with the target dataset and use it on different source datasets so that we don't need to train RPNs for different pairs of source-target datasets.

Author response:
**Necessity of training a RPN that generates source-style boxes**

We interpret the reviewer's suggestion as: "Using the RPN trained on the target dataset so that the training/inference is not tied with any source dataset."

We highlight that the pseudolabeling baseline is exactly training the RPN together with RoI on the target dataset. However, as shown in Table 1, pseudolabeling under-performs in the label translation problems, often with worse performance than "No translation".

Reviewer comment (Experiments):
Evaluations on widely used object detection datasets are missing. How about COCO, Objects365, and OpenImages? It seems that the experiments only include detection datasets for driving scenarios.

Author response:
**Evaluation on other detection datasets**

We expect our strategy to work on other common detection data sets like COCO and Object365 since our method and problem are not restricted to driving data. We focus on driving data sets because this is the real-world motivation of our work (e.g., see our Internal-Dataset) where the practice involves combining multiple data sets from different vendors [1] or mixing synthetic data [2]. We also emphasize that we considered four scenarios from seven different data sets and arrived at consistent results.

Reviewer comment (Experiments):
The proposed label translation seem to require training between every source and target datasets, which is costly to scale up.

Author response:
**Label translator is source-target specific**



We agree that a universal label translator that would not require retraining on the source and the target datasets would be impactful and impressive. However, we emphasize that we are the first one that formally introduce the label translation problem. We feel it is necessary for this work to streamline the problem setup. Furthermore, implementing a universal label translator is non-trivial (e.g., off-the-shelf foundation models are not always effective as we show in Table 5). 
We hope to extend this framework to a universal translator in future work.

Reviewer comment (Novelty):
Although the introduced research problem is interesting. The technical contributions of this paper is somewhat limited. It would be better to further highlight the novelty and technical contributions.

Author response:
**Novelty and technical contributions**

We revised the paper to clarify our novelty and the contributions in the Introduction and the Conclusion. 


To clarify our novelty and technical contributions:

1. **Novelty**: We are the first to introduce and formalize the label translation problem, to the best of our knowledge. We demonstrate that this problem is prevalent with an extensive study of several object detection datasets, by categorizing a set of four common types of annotation mismatches. 
2. **Technical/Methodology**: We propose a simple but effective label translation algorithm, which, although motivated by pseudolabeling, expands upon this naive baseline. Unlike pseudolabeling, LGPL effectively leverages source dataset bounding boxes and class labels during label translation. 
3. **Empirical**: We extensively validate our approach on seven datasets and four source-target combinations. Our approach *consistently* outperforms all the baselines in every setting. Importantly, we compare against strategies like supervised domain adaptation [1], demonstrating that our data-centric approach is a new alternative to the standard method for addressing image/label mismatch.

Reviewer comment (Experiments):
In Figure 4, the proposed method is compared to an unsupervised domain adaptive object detection method and an unsupervised image domain adaptive method. In my understanding, the two domain adaptive methods don’t have any labels in the target domain. It is not strange the proposed method achieves superior performance, as it uses more annotations. I’m wondering why the proposed method is not compared to some weakly supervised cross-domain object detection methods, such as "H2FA R-CNN: Holistic and Hierarchical Feature Alignment for Cross-Domain Weakly Supervised Object Detection, CVPR’22", where all the image-level annotations are available.

Author response:
**Comparison with supervised domain adaptive object detection method [1]**

We apologize for the confusion on this experiment. In Figure 4, we indeed compare the proposed approach with supervised domain adaptation (SDA) [1]. Prior art [1] adopts popular unsupervised domain adaptation (UDA) approaches to the supervised settings, where both the source and the target labels are accessible. We adopt two variants S-DANN and S-CycConf from [1]. Note that we prefix the supervised version with "S". 

SDA approaches proposed in [1] directly align instance features given the source and the target labels, while cross-domain weakly supervised object detection (CD-WSOD) either aligns instance-level features via pseudolabels [2,3] or aligns image-level features [4]. We choose SDA as our baselines since SDA avoids noisy pseudolabels and is able to align instance-level features between the source and the target domain effectively.

We appreciate the reviewer's careful review. We have fixed the submission to clarify the comparison in Figure 4 and added a brief background to the supervised domain adaptation in Section C.1.

Reviewer comment (Experiments):
The class-wise threshold $\sigma_c$ is ad hoc. How to choose the value? Is it sensitive to the final results?

Author response:
**Details of choosing class-conditional thresholds $\sigma_c$**

Our revision clarifies this point in Appendix D.1.

We choose $\sigma_c$ by quantizing the confidence score [5] for each class and applying it to all the label translators. Annotations falling into the last bin or with a confidence score lower than $0.001$ are dropped. In this way, the classes that are more challenging get the lower thresholds and vice versa. We leave further investigation of the class-conditional threshold to future research.

Let $v_i$ be the value of the $i^{th}$ bin. We performs sensitivity test w.r.t. $\sigma_c$ on nuScenes $\rightarrow$ nuImages:

| Label translator                  | Faster-RCNN Downstream-mAP |
|-------------------|----------------|
| LGPL with $\sigma_c = v_1$  | 42.6 $\pm$ 0.1               |
| LGPL with $\sigma_c = v_2$  | 42.02               |
| LGPL with $\sigma_c = v_5$  | 42.05               |

From the experiments above, the choices of the class-conditional threshold do have impacts on the downstream-mAP but all of them still outperform the baselines.