Paper: SelfClean: A Self-Supervised Data Cleaning Strategy (ICLR-2024)

Reviewer comment (Experiments):
The SSL encoders which are the foundation of this paper are all trained with default or near-default parameters. Given that those parameters were tuned for ImageNet in the original papers, why shouldn't we worry that the encoders in this work are trained to varying levels of quality, confounding the results?

Author response:
That is a very fair observation, which we purposefully neglected as our goal was not to show the optimal performance that can be achieved with significant time spent on hyperparameter tuning. 
However, we expect the quality of the encoders to positively correlate with the performance of SelfClean, thereby suggesting that even better results may be achieved by tuning hyper-parameters in a dataset-specific way.
This is now noted in the Limitations paragraph of section 6.

Reviewer comment (Experiments):
One of the three types of data fault considered in the paper is *near duplicates*. Given that the paper considers many medical use cases, this seems potentially problematic. What about adjacent B-scans in an OCT volume? Or adjacent tissue sections in an H&E block? Or longitudinal X-rays from a patient? All of these should be near-duplicates, but may be quite important.

Author response:
We fully agree with the observation and the examples of near duplicates that should not be removed.
In our paper we only give the definition of near duplicates without formulating a recommendation for their treatment, because no general suggestion can be made as correctly pointed out.
However, SelfClean enables practitioners to find relations among data samples, including longitudinal assessments and spatially close images, even if such information was not included in the metadata. 
This can be extremely useful for instance when merging data from different clinics in the same geographical region or when performing data splitting without leakage. 
We also observed for example that SelfClean may be able to identify dermatologic pictures of the same limbs from the front and from the back. 
When appropriately used, in our opinion, this information greatly adds to the value of the method, as we now explicitly mention in the Recommended use paragraph of section 6.

Reviewer comment (Writing):
One of the three types of data fault considered in this paper is *irrelevance*. This seems like a loaded and insufficiently defined term. Isn't relevance in the eye of the beholder? When does it make sense to have a sample that is correctly labeled but "irrelevant" to the classification task at hand? Wouldn't it be obvious from the label that such a sample is irrelevant?

Author response:
This observation is well justified as our definition of irrelevant samples is dense, and we could not unpack it because of space constraints. 
The answer is that there are indeed examples that are clearly correctly labeled but irrelevant. 
We encountered this for instance with histopathology images that were crawled together with the corresponding skin lesion: they indicate the correct disease, but they are just not suitable to be classified in the same way as, for example, dermatoscopy pictures. 
Likewise, in FOOD-101N we found abundant images of menus labeled with the correct category of food, which are arguably not amenable to the same classification as pictures of the food itself.

Reviewer comment (Experiments):
Adding X-ray images to a dataset of images from a different domain seems like a contrived task of low difficulty. (One could probably tell an X-ray image from a photograph of a skin lesion using classical image processing methods.) It seems like the "real" use case would be more like adding X-ray images of one disease to a dataset consisting of X-ray images from a very different disease. More broadly, it seems like concept granularity is an important missing piece in this work - see e.g. the granularity-depending usefulness of SimCLR features described in [cole2022does].

Author response:
The comment on granularity is very valuable, and it is included now in the Limitations paragraph in section 6. 
Throughout our experiments, we identified many datasets that have simple types of irrelevant samples, for instance, the histopathology or menu images as mentioned above.
We therefore argue that identifying even this type of simple issue is very valuable, especially when they are rare and one does not know what to look for.

Reviewer comment (Theory):
This is not an accurate definition for SSL in general: "...self-supervised learning (SSL), which produces meaningful latent spaces by training features to be invariant to augmentations." Consider, say, BYOL or MAE.

Author response:
Thank you for pointing this out. 
We have revised this sentence.

Reviewer comment (Writing):
The text discussing the merging strategy for irrelevant samples is a bit hard to follow.

Author response:
Thank you for this helpful comment. 
We will have another pass at this section and revise it.

Reviewer comment (Evaluation):
It would be nice if e.g. the metrics in Table 1 could be translated into an estimate of how much human effort is saved. This would help us understand what percentage improvement is necessary to make a meaningful difference between two methods.

Author response:
It is indeed possible to measure precisely how much human effort is saved.
However, to evaluate this in a way which only depends on the ranking requires a new metric.
More precisely, we would suggest the fraction of time spent confirming negative samples for a given recall, averaged over recall.
This is exactly zero for a perfect model which ranks all problems before valid samples, exactly one for the worst model which ranks all problems, and equal to the fraction of valid samples for a random model.
The results are still pending and will be included in the next days.

Reviewer comment (Evaluation):
The "Comparison with human annotators" section was a little hard to parse for me. Consider making the metrics and outcomes clearer. Also, "more often than random" is a pretty low bar - if something stronger can be said, it should be said.

Author response:
Thank you for pointing it out. 
We have reformulated the sentence.

Reviewer comment (Experiments):
It might be mentioning in the main paper that all of the experiments are done on ViTs (and are therefore confounded with architecture).

Author response:
Thanks for pointing this out. 
We have added a statement about the choice of encoder architecture in Section 3.

Reviewer comment (Novelty):
Lack of novelty. The described methods of clustering, distance thresholding and intra/extra class distance ratios in the embedding space trained on the target dataset have been widely known and applied in the space. Combining these approaches is not sufficient for justifying novelty.

Author response:
We are not aware of published work that leverages self-supervised features learned on the target dataset to find multiple data quality issues and presents a detailed evaluation of the data cleaning procedure.
If specific, prior relevant references are provided, we will reconsider our work's novelty.

Reviewer comment (Experiments):
Lack of proper evaluation setup. The experiments are primarily performed using the synthetic label noise which is also devised by the paper.

Author response:
We thank the reviewer for highlighting that the comparison with other methods is focused on synthetic datasets. 
The reason to take this as a starting point is that evaluating data cleaning is very difficult in real scenarios. 
The experiments in Appendix I, the comparison with metadata in Appendix G.2, and the many qualitative examples shown in Appendix M are evidence that SelfClean also works on natural contamination.
However, to address this concern, we have now extended the comparison with metadata of Appendix G.2 to include competing approaches in Table 6, albeit resorting to subsampling in the case of near duplicates because of time constraints.
Furthermore, in a follow-up work that is now accepted for publication, we analyzed four additional small to medium-size datasets with increasing levels of curation finding AUROCs of 86.2%, 99.9%, 100%, and 97.7% for irrelevant samples, 99.9%, 100%, 99.9%, and 100% for near duplicates, and 82.7%, 93.5%, 99.2%, and 98.8% for label errors.
For full disclosure, these numbers may be positively biased due to the details of the verification procedure, but still constitute strong evidence that the method performs well.
Finally, we once more highlight that our work is not limited to "label noise" only.

Reviewer comment (Writing):
The paper is hard to follow.

Author response:
We are sorry about this finding and keep this comment in mind during revision.
As the other reviewers (97xm, zncP, psYA) unanimously agree that the paper is well written, we kindly ask you to point out specific parts that would in your opinion profit from a better formulation.

Reviewer comment (Experiments):
Dataset size and data distribution would largely impact the hyper-parameters (alpha, q etc) used for identifying near duplicates, outliers etc. It is not clear how well the proposed heuristics would generalize for other datasets or other applied settings.

Author response:
This is a very valid point, which however we address by considering 10 datasets spanning different sizes (from 200 to 300,000 samples) and domains (from dermatology to X-ray scans to food and celebrity faces). 
We demonstrate that the same heuristics yield good rankings for candidate issues across all of these datasets. 
Furthermore, we are able to fix the hyper-parameters mentioned in the question, which are only used for fully automatic cleaning, to the same values throughout. 
Additionally, we report a detailed study on the influence of these hyper-parameters in Appendix H, sections H.3 and H.4.
Thus, there is evidence that the proposed heuristics generalize to several different domains and settings.

Reviewer comment (Evaluation):
Page 9: "Recommended use" section. The paper's proposal is to use SelfClean to identify the errors and fix them using human in the loop. However, the process would still introduce sampling bias originating from SelfClean. This should be noted.

Author response:
We fully agree with the observation that bias is not entirely eliminated, even in our recommended use. 
We touch on this theme in the mentioned "Recommended use" section ("The conflict between resolving data quality issues and the veto against the examination of evaluation data, mentioned in the introduction, has no easy resolution. We suggest the following compromise [...] as an improvement to the current situation."). 
To highlight the point even more, the paragraph now includes a dedicated remark.

Reviewer comment (Experiments):
What was the rationale for choosing the current baselines? Below is the suggested baseline which uses pre-trained embeddings and vector similarity to identify label noise using Markov Random Fields (most similar to the proposed approach) — Sharma et al, "NoiseRank: Unsupervised Label Noise Reduction with Dependence Models", ECCV, 2020

Author response:
We are grateful for this suggestion and have included NoiseRank in our comparison.
In Appendix G.1, you can find the results of the method in our synthetic scenario, and in Table 6 of Appendix G.2 the results of the comparison with metadata.

Reviewer comment (Writing):
The paper is way too dense and hard to follow. Proposed novelties also need to be clearly noted in the abstract.

Author response:
Thank you very much for raising this point. 
We reformulated the abstract to make it clear that features are learned in the context of the dataset only, that the combination of simple heuristics we identify is surprisingly effective, and that we perform a thorough evaluation which shows competitive results and practical utility.

Reviewer comment (Theory):
All the methods are based on the notion that distance between images represents meaningful factors of variation. The closer the images are in the latent space, the more similar they should be, both in terms of appearance and semantically.

Author response:
We agree with this comment.
Indeed, in a related project, we observe that pictograms of dermatological diseases can be used to retrieve real skin lesion images by exploiting the learned embedding space.
However, we would like to stress that in our case the hierarchy is built in the specific context of the dataset at hand.
Thus, we expect samples that are easily distinguishable from the majority to be separated from them by a greater distance.

Reviewer comment (Theory):
While SSL probably captures most factors of variation (and usually more than supervised learning), it is still not possible to control the hierarchy in terms of distance in the latent space between the factors. For example, if an image is a sketch of a dog, it is hard to tell if it will match more closely a real picture of a dog or a sketch of a cat using self-supervised features.

Author response:
We agree with this comment.
Indeed, in a related project, we observe that pictograms of dermatological diseases can be used to retrieve real skin lesion images by exploiting the learned embedding space.
However, we would like to stress that in our case the hierarchy is built in the specific context of the dataset at hand.
Thus, we expect samples that are easily distinguishable from the majority to be separated from them by a greater distance.

Reviewer comment (Evaluation):
Images from underrepresented groups or from rare classes will be removed or wrongly flagged as anomalies, which is a significant issue in medical imaging.

Author response:
Thank you for suggesting this experiment which you can now find in Appendix G.3.
Before delving into the details, it is important to note that the danger of systematically removing underrepresented groups is considerably higher in the fully automatic mode.
Fully automatic mode must be used with a lot of care to avoid undesired effects such as the ones you describe.
This is in line with our recommendation in Section 6, where we suggest to rely on the manual cleaning mode to mitigate these effects and highlight important features such as minority groups.

In Appendix G.3, we investigate if rare attributes of a dataset are systematically considered more likely to be irrelevant.
As suggested, we include some demographic attributes of CheXpert but also consider skin types in DDI and Fitzpatrick17k.
We compare the alignment of these features with the irrelevant sample ranking using AP and AUROC, similar to the comparison with metadata in Appendix G.2.
The results show that overall no systematic bias driving underrepresented groups in the earlier rankings is present, as the average precision is very close to the non-informed baseline i.e. the group's prevalence.
While this is by no means conclusive proof that biases are prevented, it illustrates that at least in a few example situations, irrelevant samples can be detected without disfavoring minority groups.