Paper: Defect Spectrum: A Granular Look of Large-Scale Defect Datasets with Rich Semantics (ICLR-2024)

Reviewer comment (Experiments):
Although the paper added rich semantic information to the dataset, its detection performance was not very good. Existing unsupervised defect detection methods have already surpassed an AUPR score of 0.7 at the pixel level, which means that existing defect detection methods can achieve an IoU of over 70 without using defect data. Some methods that use a large amount of normal data and a small amount of abnormal data have even higher performance.

Author response:
**Inclusion of Anomaly Detection (AD) Methods**: In our benchmark, we focused on evaluating semantic segmentation methods, intentionally excluding unsupervised AD methods. The reason is that current AD methods are not equipped to provide masks for different defect types, which is crucial in our approach. We compute the mean Intersection over Union (mIoU) by averaging across different defect types, a metric where AD methods fall short as they cannot differentiate between these types.
Regarding the concern about 'poor detection performance,' our use of mIoU for multi-class segmentation sets a higher standard than the binary classification task's Average Precision at a Recall (AUPR). Although our mIoU values are lower than AUPR scores reported in AD papers, they reflect a more robust defect inspection capability, as demonstrated in **Appendix Section. E** of our updated paper. Additionally, AD methods struggle with unaligned images, such as those in the VISION dataset, where variance is high.

Reviewer comment (Novelty):
I think the motivation behind the paper is very valuable, but perhaps the authors need to consider the differences from existing unsupervised anomaly detection methods.

Author response:
**Inclusion of Anomaly Detection (AD) Methods**: In our benchmark, we focused on evaluating semantic segmentation methods, intentionally excluding unsupervised AD methods. The reason is that current AD methods are not equipped to provide masks for different defect types, which is crucial in our approach. We compute the mean Intersection over Union (mIoU) by averaging across different defect types, a metric where AD methods fall short as they cannot differentiate between these types.
Regarding the concern about 'poor detection performance,' our use of mIoU for multi-class segmentation sets a higher standard than the binary classification task's Average Precision at a Recall (AUPR). Although our mIoU values are lower than AUPR scores reported in AD papers, they reflect a more robust defect inspection capability, as demonstrated in **Appendix Section. E** of our updated paper. Additionally, AD methods struggle with unaligned images, such as those in the VISION dataset, where variance is high.

**Future Directions and Community Involvement**: Our work aims to define a more complex defect inspection task in a practical manner and present augmented datasets for exploration. We leave the design of specific methods to the broader research community. Our paper's goal is to establish a foundation for researchers to adapt and leverage state-of-the-art methods within our proposed benchmark.

Reviewer comment (Reproducibility):
This paper is more suitable for a demos track, which only displays its rough progression of dataset refinement, but lacks technique detail presentation. The important details about auxiliary annotation tool and diffusion generator for defect image generation are not clearly clarified. If this refined dataset and annotations are private (not publicly available), the contributions to the development of defect inspection are very limited.

Author response:
**Open Source Issue**: We completely understand your concern regarding the open-source of our dataset. We have now made these resources publicly through this anonymous link: https://defect-spectrum-authors.github.io/defect-spectrum/, ensuring that our contributions to the field of defect inspection are substantial and accessible to the research community.

Reviewer comment (Experiments):
The comparative experiments between the refined dataset and the baseline dataset are missing in Tab.3. The results in Tab.3 can not illustrate the advantages of the refined dataset.

Author response:
**Comparison between Refined and Baseline Dataset**: To address your concern about the comparative experiments. Since both the training set and testing set were being refined, it is hard to compare the baseline-trained model quantitatively using the refined testing set or the source testing set. Thus, we qualitatively compared the segmentation model trained on the baseline dataset and our refined dataset, as shown in the **Appendix section. D** of our updated paper. The results with our annotation demonstrate 1) a more granular segmentation mask; 2) a more comprehensive detection (including those that remain unlabeled in the baseline dataset); 3) captures a wide range of defective types with better granularity while having rich semantics (different defective classes).

Reviewer comment (Evaluation):
The results of VISION dataset in Tab.4 is somewhat trival. The authors should analyze these results. The improvements of refined dataset should be clearly clarified, such as how many synthetic data are supplemented and the performance improvement relative to the increment of dataset size.

Author response:
**Clarification on Table 4 and Synthetic Data Utilization**: We appreciate your comments on Table 4 and the need for clarification regarding synthetic data. Regarding the results in Table 4, we would like to clarify that we have selected the best baseline to perform our performance boost experiment. We think although the improvement is comparatively smaller than the increase in the MVTec dataset, it still demonstrates the effectiveness of the synthetic data. We also extended the experiment to compare using three other baselines: M2F, MiT-B0, and DeepLabV3+, the experiments can be found in the table below (also see **Appendix Section. A** in the updated paper). 

Performance (mIoU) comparison between models trained with and without synthetic data. The bolded text indicates results with synthetic data.
| Model        | MVTec                     | VISION                    | Cotton                     |
|--------------|---------------------------|---------------------------|----------------------------|
| DeepLabV3+   | 51.58/**55.55**           | 52.33/**53.46**           | 48.73/**58.58**            |
| Mask2Former  | 45.70/**50.16**           | 54.12/**55.47**           | 64.09/**65.39**            |
| MiT-B0       | 46.45/**56.21**           | 49.62/**50.75**           | 50.52/**55.86**            |

**Generation Data Ratio Experiment**: Further, as shown in Figure. 8, our experiments include a detailed analysis of the generation data ratio, where we examine the impact of varying proportions of synthetic data on model performance. This provides a comprehensive view of how synthetic data supplementation affects defect detection capabilities. We are sorry for the confusion it brought and will adjust it by extending the experiment as we mentioned in the response to reviewer 2(qXne), the results for the extended synthetic data ratio experiment can be found in **Appendix Section. B** of our updated paper.

Reviewer comment (Experiments):
Although the dataset contains descriptive captions, they are not used in the defect inspection baselines. It is not clear with me why these descriptive captions are essential and how they are used in practical systems? To me, it seems that captions are not as accurate as the provided semantic masks. And in what way can these captions improve the defect inspection system?

Author response:
**Importance of Image-Text Pair Datasets**: Image-text pair datasets are crucial for training Multimodal Large Language Models(MLLMs), which have shown their dominant capabilities in a large range of tasks. Our descriptive captions offer additional contextual information that can be leveraged by these models. We look forward to empowering MLLMs with the ability of defect inspection through this effort.

Reviewer comment (Experiments):
Figure 8 looks quite interesting, but it seems a bit incomplete. In Fig. 8, there is a clear trend that when adding more synthetic samples, the performance of both approaches is boosted. Thus, It would be more interesting if results of further increasing the ratio of synthetic data can be shown.

Author response:
**Extended Synthetic Data Ratio Experiments**: To address your concerns, we have extended our experiments to include scenarios with higher ratios of synthetic data, shown in the **Appendix Section. B**(Please refer to the updated paper). Specifically, we have now included results for systems trained with 200% and 300% synthetic data ratios. When using synthetic data that is 200% of the size of the original training set, there is an enhancement in the performance, but results in greater variance. Additionally, the performance starts to decrease after reaching 300%. This is due to the overly-used synthetic data disrupting the original data distribution.

Reviewer comment (Experiments):
It seems the the sampling strategy for generated samples could be improved. In the paper, it seems that the quantity of generated samples is at a fixed ratio to the real samples. In this sense, if there is a lot of real data for certain object, then model would generate a lot of synthetic data, and vice versa. This is a bit counter-intuitive to me. It seems more reasonable that for those objects with limited number of real data, we should generate more synthetic data to mitigate the overfitting problem.

Author response:
**Sampling Strategy**: As an immediate response to your suggestion, we conduct an experiment where we employ a uniform sample strategy – using a fixed number (e.g., 150) of samples in the training set for each object. So those with limited data gain more synthetic data and vice versa. As shown in the table below (also see **Appendix Section. C**), with this sampling strategy, baselines such as MiT-B0 show a great leap in performance.

|       | Baseline: **DeepLabV3+** | With Synthetic | Baseline: **MiT-B0** | With Synthetic |
|--------|--------------------------|----------------|---------------------|----------------|
| **Mean** | 51.58             | **54.87**      | 46.45           | **56.21**      |

Reviewer comment (Experiments):
unclear motivation of the choice of the base benchmarks

Author response:
**The reason to choose MVTEC, VISION, DAGM, and Cotton datasets**:
**MVTEC Dataset**: Renowned for its high quality, the MVTEC dataset is a well-established benchmark in the field. Its high-resolution images and large-scale nature provide a robust testbed for models, enabling detailed defect detection. 

**VISION Dataset**: as the largest scale dataset among those we evaluated, VISION has high variance and high-resolution images. This dataset challenges models with a wide array of defect types and imaging conditions, making it an excellent benchmark for assessing a model's versatility. 

**DAGM Dataset**: Unique in its use of special imaging techniques, DAGM provides images that are far from natural representations. This characteristic is crucial for testing a model's ability to generalize and perform in less conventional or more challenging imaging environments. 

**Cotton Dataset**: As a real-world industry dataset produced directly in factory settings, the Cotton dataset offers high variance and practical relevance. It simulates real-world industrial conditions, thereby providing a realistic assessment of a model's performance in actual deployment scenarios.

Reviewer comment (Experiments):
insufficient experiments

Author response:
**Extended experiments on other baselines**: The decision to select the best baseline aims to demonstrate Defect-Gen's capability in the most demanding scenarios, particularly where baseline performance is already high. This approach highlights the robustness and effectiveness of our method, even under challenging conditions. To further underscore the efficacy of Defect-Gen, we conducted comprehensive experiments using three distinct segmentation models: Mask2Former, MIT-B0, and DeepLabV3+, as detailed in the table below (also see **Appendix Section A** in the updated paper). The results from these models reinforce our findings, demonstrating that Defect-Gen not only excels with top-performing methods but also significantly enhances the performance of less advanced models. This broad applicability underscores the versatility and practical utility of our approach.

Performance (mIoU) comparison between models trained with and without synthetic data. The bolded text indicates results with synthetic data.
| Model        | MVTec                     | VISION                    | Cotton                     |
|--------------|---------------------------|---------------------------|----------------------------|
| DeepLabV3+   | 51.58/**55.55**           | 52.33/**53.46**           | 48.73/**58.58**            |
| Mask2Former  | 45.70/**50.16**           | 54.12/**55.47**           | 64.09/**65.39**            |
| MiT-B0       | 46.45/**56.21**           | 49.62/**50.75**           | 50.52/**55.86**            |

Reviewer comment (Evaluation):
I have some questions regarding the quantitative evaluation results of multiple defect segmentation methods across the different Defect Spectrum reannotated datasets presented in Table 3. The table shows that the average performance of these methods is quite similar, despite their varying performance on individual defect image types. This raises doubts about the suitability of this benchmark for assessing the discrimination ability of different models.

Author response:
**Concerns on the quantitative evaluation result**: The concern about the similar average performance of various methods across different datasets, despite their varying performance on individual defect image types, is a valid point. However, if we look at the mean IOU report in Table 3, the min and max MIOU reported are 45.40 and 51.58. The segmentation results on the CityScapes dataset across various baselines also demonstrated similar MIOU results between 76.9 to 81.6, except for the traditional UNet(69.1).  As for the varying performance on individual classes, this is due to the different granularity that exists in different defect types, making some of the defective regions comparatively smaller than other types.

Reviewer comment (Evaluation):
There are limitations to the quantitative evaluation of the generation quality of the proposed data generation method. In Table 4 of section 4.4, only the performance of the best baseline is reported, which may not be sufficiently persuasive.

Author response:
**Extended experiments on other baselines**: The decision to select the best baseline aims to demonstrate Defect-Gen's capability in the most demanding scenarios, particularly where baseline performance is already high. This approach highlights the robustness and effectiveness of our method, even under challenging conditions. To further underscore the efficacy of Defect-Gen, we conducted comprehensive experiments using three distinct segmentation models: Mask2Former, MIT-B0, and DeepLabV3+, as detailed in the table below (also see **Appendix Section A** in the updated paper). The results from these models reinforce our findings, demonstrating that Defect-Gen not only excels with top-performing methods but also significantly enhances the performance of less advanced models. This broad applicability underscores the versatility and practical utility of our approach.

Performance (mIoU) comparison between models trained with and without synthetic data. The bolded text indicates results with synthetic data.
| Model        | MVTec                     | VISION                    | Cotton                     |
|--------------|---------------------------|---------------------------|----------------------------|
| DeepLabV3+   | 51.58/**55.55**           | 52.33/**53.46**           | 48.73/**58.58**            |
| Mask2Former  | 45.70/**50.16**           | 54.12/**55.47**           | 64.09/**65.39**            |
| MiT-B0       | 46.45/**56.21**           | 49.62/**50.75**           | 50.52/**55.86**            |

Reviewer comment (Experiments):
I am curious about the reasons behind choosing to integrate and reannotate those four datasets from the diverse benchmarks listed in Table 1.

Author response:
**The reason to choose MVTEC, VISION, DAGM, and Cotton datasets**:
**MVTEC Dataset**: Renowned for its high quality, the MVTEC dataset is a well-established benchmark in the field. Its high-resolution images and large-scale nature provide a robust testbed for models, enabling detailed defect detection. 

**VISION Dataset**: as the largest scale dataset among those we evaluated, VISION has high variance and high-resolution images. This dataset challenges models with a wide array of defect types and imaging conditions, making it an excellent benchmark for assessing a model's versatility. 

**DAGM Dataset**: Unique in its use of special imaging techniques, DAGM provides images that are far from natural representations. This characteristic is crucial for testing a model's ability to generalize and perform in less conventional or more challenging imaging environments. 

**Cotton Dataset**: As a real-world industry dataset produced directly in factory settings, the Cotton dataset offers high variance and practical relevance. It simulates real-world industrial conditions, thereby providing a realistic assessment of a model's performance in actual deployment scenarios.