Paper: Alice Benchmarks: Connecting Real World Re-Identification with the Synthetic (ICLR-2024)

Reviewer comment (Novelty):
The contribution is not enough. I fully understand the importance of the benchmark. But as far as I am concerned, the paper’s biggest significance is the “un-clustered” data architecture. Some other contributions as benchmarking existing works and discussion cannot be considered as significant due to (1) no significant contribution beyond other works, and (2) no novel method is proposed to customize the new-labeled synthetic to real dataset.

Author response:
Thank you for your comments. We aim to clarify the contributions of this work to provide a clearer understanding:

First, the Alice benchmarks include an online service for evaluating domain adaptation algorithms in object re-identification, which as far as we know is the first online evaluation server in the re-ID community. This service specifically focuses on learning from synthetic data and testing on real-world data.

Second, two challenging real-world datasets are collected for person and vehicle re-ID evaluation when learning from synthetic data. Note that the "un-clustered" data architecture is not their only key characteristic. Additionally, they offer unique features: 1) The target training data more accurately reflect real-world distributions as they are not manually selected. 2) Both AlicePerson and AliceVehicle present more challenging test scenarios compared to existing datasets. 3) The test labels in these datasets are hidden, providing a unified, fair, and challenging benchmarking environment. 

Third, We conduct evaluations of some existing common domain adaptation methods on our two tasks. Results and analysis provide insights into the characteristics of synthetic and real-world data.

Fourth, our work introduces new research problems enabled by Alice, particularly in understanding the "synthetic to real" domain adaptation challenge.

Compared to existing works, our paper systematically evaluates and discusses various domain adaptation methods for learning from synthetic data. As far as we know, there is no previous work conducting this kind of evaluation.  Therefore, we respectfully disagree with the comment that  ''no significant contribution beyond other works".

Regarding the  ''no novel method being proposed to customize the new-labeled synthetic to real dataset", this paper provides a friendly editing interface where researchers can generate their own source dataset with an arbitrary number of images under **manually controllable environments**. This interface, combined with our discussions on **content-level domain adaptation methods**, lays the groundwork for future explorations in customizing synthetic data. We will study this in our future work.

Reviewer comment (Novelty):
The domain adaptation seems not suitable for synthetic to real datasets. The gain of synthetic data is easy and straightforward. Therefore, we should generate more diversified synthetic data to achieve domain generalization rather than domain adaptation. Please refer to ClonePerson and RandPerson for this kind of research.

Author response:
Thank you for your insightful comments. It is important to recognize that domain adaptation and domain generalization are distinct approaches, each with its unique applications, particularly when utilizing synthetic data for training. While it is challenging to definitively determine which approach is more suitable for synthetic to real datasets, studying the potential of domain generalization is a good idea.

In our current work with Alice, we have focused on domain adaptation, partly because acquiring some unlabeled target data is relatively low-cost, making it a feasible starting point. However, we acknowledge the potential and relevance of domain generalization in this context, as the reviewer suggested. We will extend the scope of Alice to include domain generalization in our future work. This expansion will enable us to investigate the potential effectiveness of synthetic data across both domain adaptation and generalization contexts.

Reviewer comment (Experiments):
The authors discuss the legitimacy in data-labeling, however, the main step that incurs the privacy problem is capturing data. How can your data captured in Figure 2 protect the privacy of persons and vehicles? As far as I am concerned, the passersby cannot know their cars or themselves are trained for AI models. A license from local government is not enough — it seems to be a human rights matter.

Author response:
Thank you for highlighting the critical issue of privacy. As we have noted on page 2 (bottom) of our paper, we have taken specific measures to protect the privacy of individuals in Alice. This includes manually blurring human faces and vehicle license plates to prevent identification. Additionally, the dataset we released consists solely of bounding boxes for persons and vehicles, rather than full images like in Figure 2. Moreover, our dataset, referred to as Alice in the paper, is distributed under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). This license restricts the use of Alice for non-commercial purposes, further mitigating potential privacy concerns.

We also recognize the value of using synthetic data as an effective strategy to avoid privacy issues. In this regard, Alice is intended to encourage and facilitate research in this direction. By adopting these measures, we have minimized privacy risks as much as possible while contributing to the advancement of research in the field.

Reviewer comment (Experiments):
The synthetic data in benchmark are from existing datasets, i.e., PersonX and VehicleX, while the real-world data (AlicePerson and AliceVehicle) in benchmark shows less advantage compared to current off-the-shelf reid datasets. The strong clusterability issue is the only difference in my opinion. However, for AlicePerson, I do NOT think adding distractor and junk is a good way to solve the strong clusterability issue compared. For AliceVehicle, in Table.4, I do NOT find much difference between AliceVehicle and VeRi776 on the strong clusterability issue.

Author response:
Indeed, PersonX and VehicleX serve are existing datasets, but we have significantly modified and integrated them into a new framework within Alice. This integration includes **a user-friendly editing interface**, enabling researchers to create customized source datasets. This interface allows for the generation of a varied number of images under environments that are manually adjustable, offering greater flexibility and control in dataset creation.

We respectfully disagree with this comment. Beyond the clusterability issue, our real-world datasets exhibit several significant differences when compared to existing datasets like Market-1501:

1) Target training data is more consistent with real-world distributions because they are not manually selected by humans.

2) AlicePerson and AliceVehicle are more challenging datasets because existing methods get lower improvements on them than on existing datasets. 

3) The test labels of AlicePerson and AliceVehicle are hidden, which will offer a unified, fair, and challenging benchmark. 

Meanwhile, datasets are only part of Alice Benchmarks. Alice provides:

1) an online evaluation platform which as far as we know is the first online evaluation server for person/vehicle re-identification.

2) evaluations and analysis of different domain adaptation methods, which can provide some insights into learning from synthetic data.

3) discussions of interesting questions and future directions. 

Furthermore, the Alice project is our long-term project. In our future work, we will try to include more tasks beyond the existing re-ID task by welcoming open-source collaboration, with the aim of covering more objects in the visual world. Therefore, we think the current Alice and its potential version will contribute to the community for studying learning from synthetic data and related topics. We will add these discussions in our revised paper to provide readers with a more complete understanding of the Alice Benchmarks.

Thank you for your comments. However, there seems to be a misunderstanding regarding the settings of Alice. Firstly, the inclusion of distractors and junk in AlicePerson is not intended to address clusterability issues. In image retrieval and person re-ID communities, incorporating low-quality bounding boxes, often termed "distractors," into test sets is a standard practice. For instance, Oxford5K has 25\% of distractors in its gallery images, while Market-1501 includes 2,793 distractors in its 19,735 gallery images. The reduced clusterability in Alice target training sets is caused by collecting data directly from real-world scenarios without manual selection.

Regarding the comparison of AliceVehicle and VeRi776, our paper has discussed this in the second paragraph of Section 6.3:

>differences in performance on the person re-ID dataset are sharper than on vehicle re-ID dataset due to the fact that changes in the appearances of cars are limited compared to the changes in people. Despite the randomness of the training set in AliceVehicle, the clustering will not be much weaker than that of the VeRi-776 training set, so there is no large difference in their final results

Numerically, the results may appear similar, but a closer examination reveals notable differences. For instance, when comparing improvements over the baseline IDE, most methods on AliceVehicle achieve about 50\% of the improvement they do on VeRi-776. As an example, PUL's improved mAP score on AliceVehicle is 6.65\%, roughly half of its 11.87\% improvement on VeRi-776. This suggests that AliceVehicle presents more challenging conditions than VeRi-776. 

We will add these discussions to our revised paper.

Reviewer comment (Experiments):
The UDA methods compared in this benchmark (Fig.3 & Tab.3 & Tab.4) are out of fashion, new methods should be involved.

Author response:
Thanks for your kind suggestion. In our current paper, we have focused on evaluating the most commonly used DA methods to establish a \red{reliable benchmark}. These methods have been frequently cited and utilized in recent re-identification (re-ID) literature. Meanwhile, we also compared different baseline methods in which MSINet (CVPR2023) is a very recent work. Following the suggestion of the reviewer, we will try to add new methods in the revised paper.

Reviewer comment (Novelty):
Beside the introduce of the strong clusterability issue, I can not find any new things in this benchmark. Please see the weakness for details.

Author response:
We respectfully disagree with this comment. Beyond the clusterability issue, our real-world datasets exhibit several significant differences when compared to existing datasets like Market-1501:

1) Target training data is more consistent with real-world distributions because they are not manually selected by humans.

2) AlicePerson and AliceVehicle are more challenging datasets because existing methods get lower improvements on them than on existing datasets. 

3) The test labels of AlicePerson and AliceVehicle are hidden, which will offer a unified, fair, and challenging benchmark. 

Meanwhile, datasets are only part of Alice Benchmarks. Alice provides:

1) an online evaluation platform which as far as we know is the first online evaluation server for person/vehicle re-identification.

2) evaluations and analysis of different domain adaptation methods, which can provide some insights into learning from synthetic data.

3) discussions of interesting questions and future directions. 

Furthermore, the Alice project is our long-term project. In our future work, we will try to include more tasks beyond the existing re-ID task by welcoming open-source collaboration, with the aim of covering more objects in the visual world. Therefore, we think the current Alice and its potential version will contribute to the community for studying learning from synthetic data and related topics. We will add these discussions in our revised paper to provide readers with a more complete understanding of the Alice Benchmarks.

Reviewer comment (Evaluation):
Despite not being quite clear at this point, it would be nice if authors could provide some discussion on the potential limit that synthetic data can benefit the learning and inference on real-world data, specifically whether there are some inherent barriers.

Author response:
Thank you for your valuable suggestion. In this paper, we primarily focus on exploring the domain gap between synthetic and real-world data, which is thought as a key barrier in this field. Additionally, in Section 7, we have provided some related discussions of other possible barriers. For instance, under the topic "Challenges of organizing synthetic datasets and improvement directions," we highlight the limitations caused by the lack of synthetic assets. In the section "Generalization of conclusions drawn based on the Alice benchmarks," we discuss the applicability of conclusions derived from synthetic data to real-world scenarios.

Moreover, an additional challenge in using synthetic data is the difficulty of synthesizing data for specific tasks, such as in the field of medical imaging. However, the research community is showing a promising trend towards developing more diverse synthetic datasets including a broader range of objects. For example, the Infinigen dataset presented at CVPR2023 shows the expanding capabilities of synthetic data. 

Following your suggestion, we will incorporate these additional insights and discussions into the revised version of our paper, thereby providing a more comprehensive understanding of the potential and limitations of synthetic data in learning and inference.

Reviewer comment (Experiments):
The authors should consider the impact of different synthetic data generation strategies on domain adaptation effects, such as using different graphics engines, different attribute distributions, different noise levels, etc.

Author response:
Thank you for your suggestion. In our paper, we have included some discussions regarding the impact of various synthetic data generation strategies on the effectiveness of domain adaptation.

For example, Section 6 discusses the impact of varying attribute distributions, specifically comparing random attribute selection and attribute descent methods, on domain adaptation results. Our results indicate that re-ID domain adaptation models, when trained on synthetic data refined by attribute descent, show improved performance. Additionally, Table 6 presents the FID scores comparing target real-world data with synthetic data, with and without attribute adjustment. These results reveal that synthetic data with learned attributes have a smaller domain gap with real-world data compared to randomly attributed data.

Furthermore, in Section E, we explore other potential strategies for effectively utilizing synthetic data. For example, it is mentioned that 
> the characteristics of a good training set are still largely unknown. In this regard, having a smaller domain gap between training and testing sets might not be the only objective. Other indicators such as appearance diversity or even noise are worth investigating.

While these possible directions are interesting, it is beyond the scope of a single paper to explore every potential strategy in depth. Therefore, our current work begins by benchmarking common strategies and offering insightful discussions to lay a foundation for future exploration.

On the topic of graphics engines, such as Unity and Unreal, the paper `A Review of Synthetic Image Data and Its Use in Computer Vision' indicates that the choice of engine is less impactful than the quality of 3D models. Hence, our discussion emphasizes the significant role of 3D model creation in generating high-quality synthetic data, rather than the effects of different engines.

In conclusion, the impact of various data generation strategies, as the reviewer mentioned, opens a wide range of possibilities. This board field cannot be thoroughly explored within one paper. Our current work provides in-depth discussions of the widely used strategies and highlights potential future directions. It will promote further work and discussions around the effective use of synthetic data. We will continue to investigate these directions and add the aforementioned discussions to our revised paper.

Reviewer comment (Evaluation):
The quality of the generated data and labels is not clearly discussed. I recommend measuring the quality of the generated data and the impact of improving quality on the results.

Author response:
Thank you for your valuable suggestion. Regarding the quality of synthetic data, the accuracy of labels is nearly 100\% due to the automated annotation process. This process involves generating images with annotation based on the IDs of the 3D person and vehicle models. Once these 3D identities are ready, a programmed system generates the annotations, thereby minimizing the risk of incorrect labelling. As a result, the label quality of our synthetic data is highly reliable, which is why we do not discuss the label quality much in the paper. 

To measure the quality of the generated images, we report the FID between real-world and synthetic data generated with Random and Attribute Descent (Attr. desc) methods in Table 6. The FID is widely used in evaluating the quality of GAN-generated data, which can help us understand the quality of synthetic data generated by different methods. For example, the FID between Market-1501 and synthetic data with Attr. desc is 83.24, lower than the FID of 104.78 for Market-1501 against synthetic data with random attributes. This difference in FID scores indicates that the data generated through Attr. desc is of higher quality than that generated using random attributes.

Additionally, Section 6.2 of our paper discusses the effectiveness of content-level domain adaptation, which enhances the quality of synthetic data by mimicking the distribution of real data. Moreover, the results shown in Fig. 3 highlight the impact of data quality improvement on re-ID results. Synthetic data generated using attribute descent demonstrate improved re-ID results compared to randomly generated data. 

Following your recommendation, we will include more detailed discussions on the impact of data quality in the revised version of our paper.

Reviewer comment (Presentation):
The title of this paper is biased. It focuses on the person/vehicle re-identification, while the title uses object re-identification. The scope of the title is too large. Although the authors claim that more objects will be added later, for the purposes of this article, it is limited to the field of person/vehicle re-identification.

Author response:
Thank you for the helpful reminder. We acknowledge the point of the reviewer that the current scope of the Alice project primarily centres on person/vehicle re-identification. We will revise the paper's title to more accurately reflect its current focus.