Paper: Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI (ICLR-2024)

Reviewer comment (Experiments):
Hardness characterization is an important concept in various fields such as computer vision works like object detection and segmentation, as well as natural language processing and reinforcement learning. However, the proposed HCM evaluation framework in this paper can only handle image classification.

Author response:
We clarify that we primarily studied HCMs in the context of image classification, as most HCMs (10/13) have been developed in this context. While our current evaluation predominantly employs image datasets, we also incorporate tabular datasets ("Covertype" and "Diabetes130US") to demonstrate the generalizability of H-CAT across different data modalities. This inclusion of diverse datasets is a step towards extending the applicability of HCMs beyond image classification. We hope that the extensibility of our framework will encourage future work to cover even more domains as suggested by the reviewer.

Reviewer comment (Theory):
In the Introduction, it is stated that "These 'hard' samples or data points can significantly hamper the performance of ML models." However, from an active learning perspective, can't we consider hard samples to be more informative and better samples with higher annotation efficiency?

Author response:
Thank you for pointing out the important distinction between the concepts of "hard" samples in active learning and in data-centric AI/hardness characterization. We wish to clarify the different contexts in which samples are termed as “hard” between the two areas of the literature. 

- **Active Learning**: "hard" samples refer to those instances where the model struggles to make accurate predictions, yet the samples are _correct_. These samples are valuable because they are informative. However, we note these samples are NOT inherently flawed or erroneous. 
- **Data-Centric AI and Hardness Characterization**: In contrast to active learning, in data-centric AI and hardness characterization, "hard" samples typically denote samples that have inherent issues or are erroneous, such as being mislabeled. Hence, the focus is on identifying and addressing these problematic samples to improve data quality. Consequently, we wish to detect and rectify the actual issues with the samples themselves, rather than to use them for informative purposes.

Given this distinction, our work aligns with the second context, focusing on identifying and characterizing samples that are inherently problematic or erroneous. We thank the reviewer for bringing up this point.

Reviewer comment (Experiments):
On page 3, in the section "Data characterization," it is mentioned that difficult or easy samples are classified into separate easy or hard subsets. However, instead of categorizing sample difficulty into two binary sets, wouldn't it be better to dynamically control the learning process by assigning weights or using other methods based on continuous difficulty values? Additionally, excluding OOD/outlier and atypical samples from model training could potentially enhance the model's generalization capability. So, couldn't excluding them altogether from training be a suboptimal choice?

Author response:
We agree with the reviewer that an important direction is how the types of samples could be used to dynamically control the learning process — for instance, a curriculum learning approach as suggested by the Reviewer in contrast to, for instance, pruning/sculpting them. These methods represent an interesting direction of how to leverage the hardness scores to enhance the performance of a downstream model.

However, it is important to emphasize that these approaches, while important, are orthogonal to the primary focus of our work. Our research in H-CAT is centered on understanding, evaluating, and benchmarking the capability of HCMs to detect different types of hardness in samples. The core objective is to provide a comprehensive and systematic framework to evaluate the effectiveness of various HCMs in identifying hard samples across a range of hardness types.

While we acknowledge the utility of the approaches suggested by the Reviewer in the context of model training, our research does not delve into the application of detected hard samples in training strategies. Instead, it focuses on the foundational aspect of accurately identifying these hard samples, which is a prerequisite for any subsequent application or strategy, including curriculum learning or sample pruning.

We thank the reviewer for this suggestion and have included it in **Sec 6**, wherein future work can build on H-CAT and explore this dimension of what is the best way to use the hardness scores for a downstream model.

**UPDATE:** Sec.6 now outlines work to understand the usage of hardness scores to guide future model training.

Reviewer comment (Evaluation):
In Figure 6, Spearman rank correlation scores for HCMs are reported. As far as I know, Spearman rank correlation calculates the correlation between two variables. How was the correlation computed from multiple runs in this case?

Author response:
We clarify that when we compute the Spearman correlation of scores across runs (different seeds), this is done between the hardness scores obtained for all combinations. This mirrors the approach adopted by Maini et al., 2022 and Seedat et al., 2022.

Reviewer comment (Experiments):
Sticking to just MNIST and CIFAR10 limits the usefulness of the takeways. The dataset sizes and complexity we currently operate with are very different from MNIST/CIFAR10 and I'm afraid these learnings might not generalize to larger, more complex datasets.

Author response:
### (A) Beyond MNIST & CIFAR10

We wish to clarify our choice of MNIST and CIFAR-10 was strategic as these datasets are well-established in the HCM literature and, as discussed in **Sec 5**, these datasets are “clean” without significant mislabeling. This is essential to create controlled experimental conditions for benchmarking where hardness is introduced. Furthermore, to address concerns about generalizability, we also included evaluations on more complex tabular datasets in Appendix D, such as “Covertype” and “Diabetes130US” from OpenML. This approach ensures our findings are relevant across different data modalities and sizes. 

That said, we understand the Reviewer's potential concern and have conducted an additional experiment using a medical X-ray dataset from the NIH [R1], which features high-quality images curated by multiple medical experts. This additional experiment reinforces the applicability of our findings to more complex and real-world datasets. The results shown in the updated Appendix D.7 align with our initial findings, indicating that our observations regarding various hardness types are not limited to MNIST and CIFAR-10 datasets but are also relevant to more specialized datasets such as medical images. This further validates the generalizability and practical utility of our research across diverse data domains.

**UPDATE:** Added a new Appendix D.7 with additional image experiment on a medical X-ray dataset

Reviewer comment (Experiments):
The Near OOD, far OOD, and the atypical perturbations used in the experiments are oversimplified in my opinion. In the real world, defining near/far OOD and atypical have a complicated notion of support of the sample distribution, but the use of Gaussian noise, texture changes for near OOD and exchanging MNIST with CIFAR (which are drastically different) for far OOD, and using simple invariant functions like shift and zoom for atypical does not do justice to these concepts.

Author response:
### (B) Simplicity of perturbations

We acknowledge the Reviewer's concerns about the perceived simplicity of the OOD and atypical perturbations. We highlight two aspects: (1) our methodology mathematically satisfies the definitions of these perturbations, and (2) **Figure 7 (now Figure 3 in the revised paper)** illustrates that the perturbations we used are representative of hardness types that could realistically occur in actual datasets. Moreover, while our work is indeed a first step, it represents a significant advancement from the existing state of HCM research which doesn’t consider these issues (as shown in **Table 1**). We hope that our framework serves as a foundation for more complex and nuanced studies in the future. 

Finally, a key finding is that certain **HCMs even fail on these simple perturbations**, which uncovers a fundamental limitation of the HCM field that needs to be addressed.

Reviewer comment (Novelty):
[Minor] The bulk of paper is focussed on creating a taxonomy for hardness evaluation and then building a software toolkit to evaluate existing methods for its quantification. Some initial discussions during the taxonomy creation and some of the non-obvious takeaways are instructive for the researchers in this field, but most of this work is geared towards practitioners. This isn't necessarily a negative thing, but limits the research novelty/impact of the work.

Author response:
### (C) Nature of takeaways [minor]

The paper indeed has a practical aspect of the benchmarking framework. We wish to clarify that this enables the HCM field to rigorously evaluate HCMs for the first time, which is essential for the field to progress. We believe this will help the HCM field to move from the ad hoc and qualitative state (as shown in Table 1) toward a more comprehensive and rigorous evaluation across multiple hardness types. 

Specifically, our work responds to the recent calls for more rigorous benchmarking and understanding of existing ML approaches, as highlighted by Guyon (2022), Lipton & Steinhardt (2019), and Snoek et al. (2018). We then believe this benchmark offers critical insights into the HCMs which are distilled into (1) benchmarking takeaways: to guide researchers about future development of HCMs by highlighting their strengths and weaknesses and (2) practical takeaways: to guide selection and usage of HCMs. 

Overall, this work provides insights into what types of hardness different HCMs are able to detect and shows the limitations of HCMs in detecting certain hardness types, which we hope will inspire researchers to develop new methodologies to address these limitations.

Reviewer comment (Evaluation):
As the severity of perturbations increases, do the methods used in HCMs also show an increase in the hardness score? Does this vary for learning-integrated methods vs others?

Author response:
### (D) Impact of perturbation severity


We thank the reviewer for this interesting suggestion! We have conducted an experiment for zoom shift, crop shift, and Near-OOD (covariate), where we have two severities of perturbation namely small and large. For zoom (2x vs 10x), crop shift (5 pixels vs 20 pixels), and Near-OOD (std deviation of 0.5 vs 2). 

We report the results in Fig 99, in Appendix D.8. The results show the percentage changes in hardness scores (in the direction of hardness, e.g. increase or decrease) for different HCMs for the different hardness types considered. n all cases, we see an increase in hardness score$^1$ for increased severity, with some HCM scores showcasing greater sensitivity and changes than others. In particular, we see the gradient and loss-based approaches have the greatest changes as compared to the methods computing scores based on probabilities. We thank the reviewer for suggesting this interesting experiment which has helped to shed further insights into the sensitivities of different HCMs.

For ease of access the result can also be found at this Imgur link: **https://imgur.com/tTv6lO7**

**UPDATE**: Included a new Appendix D.8 with the experiment.

----

Reviewer comment (Presentation):
The paper could be structured better. Introducing and clearly defining "hardness types" and "HCMs" earlier would help readers understand the paper better. Some important explanations and figures that support the main concepts are in the appendix, which makes it difficult for readers to access them quickly and easily.

Author response:
We thank the reviewer for the suggested paper updates of moving contents from the appendix to the main paper which we believe have improved the paper.

We have now moved Figure 7 (now Figure 3) into the main paper to readers' intuition of the differences between OOD and Atypical, as well as show that our perturbations are indeed realistic. 
Introduce hardness types in the introduction further with intuitive examples
Building on our introduction of HCMs in the intro, we move the grouping of the broad classes of HCMs (e.g. learning-based) from the Appendix into our formulation in Section 2.

**UPDATE**: moved Figure 7 (now Figure 3) into the main paper

Reviewer comment (Writing):
The paper would be more technically rigorous if it included detailed equations and unified notation for the various HCMs in the appendix. Providing a more detailed explanation of methods, AUM for example, in the main text would also make the paper more instructive.

Author response:
We thank the reviewer for the suggestion of adding the equations and unified notation for the HCMs in the Appendix, to make it a go-to-paper on hardness literature. We describe this below and have **updated Appendix A** with the following changes.

- Added the unified input-output formulation for any HCM. We have also formulated HCMs are defined by a scoring function $S$.
- We mathematically define each scoring function under this unified formulation requiring a scoring function. 
- We have added a new Table 2 in the Appendix, formalizing the meaning of the scores per HCM.

Reviewer comment (Theory):
The paper does not address the fact that hardness can change during the training process. A hard example at epoch 1 might not be hard at epoch 10.

Author response:
The inherent hardness of a sample remains consistent regardless of the training epoch. For instance, a sample that is hard due to mislabeling retains this characteristic throughout the training process. A mislabeled sample is indeed mislabeled at both epoch 1 and epoch 10. This constancy is crucial to our understanding and characterization of sample hardness. Our work on H-CAT then focuses on assessing the capability of the HCM scores to correctly identify these inherent hardnesses in the sample itself.

Reviewer comment (Novelty):
The paper insists that studying data hardness is critical for advancing data-centric AI, but it is unclear what data-centric AI is, and why HCMs are the key to advance it. A more fundamental question is, do we really need HCMs at all or we only need to create high quality datasets that eliminate corrupted examples?

Author response:
We thank the Reviewer for the opportunity to clarify the role of HCMs within data-centric AI. Data-centric AI focuses on systematic methods to enhance data quality — with hardness characterization being fundamental by identifying such hard examples in an algorithmic manner. We fully agree with the Reviewer we need to create high-quality datasets that eliminate corrupted examples — we emphasize that HCMs are one potential algorithmic tool to achieve this. Algorithmic identification is especially important when manual curation is impractical, for example at scale. Hence, we can think of HCMs being the algorithmic mechanism to achieve the data quality goals of data-centric AI. To illustrate the practical value and need for HCMs in creating high-quality datasets, we highlight that  (i) THE _Cleanlab HCM_ was used to audit ML benchmark datasets for mislabeling (Northcutt et al, 2021b), (ii) the _Data Maps HCM_ was used to audit and create high-quality RNA data in biology [R1], and (iii) the _Data-IQ HCM_ was used to audit and curate high-quality sleep staging data [R2].

Reviewer comment (Experiments):
Following the need for HCMs, a baseline of training models without HCMs is missing. There are two scenarios that should be considered as baselines: 1) training a model without HCM with the same level of corruption introduced as the other models. 2) Training a model without HCM and with only the easy examples, i.e. removing the portion of corrupted samples. This would be informative to evaluate what cases in the taxonomy really need to be investigated further with HCMs and which ones can be addressed by simply cleaning the dataset.

Author response:
We wish to clarify the goal of this work is to evaluate the detection capabilities of various HCMs in identifying different types of data hardness, rather than assessing their curation impact on downstream model performance (e.g. suggested baseline without HCMs). The reason is previous research has already demonstrated the value of using HCMs to curate datasets to train downstream models. In contrast, our work complements the existing literature which already assesses the suggested baseline and extends previous work by providing a detailed analysis of how effectively different HCMs can identify various hardness types, which is a novel and necessary perspective for the field.

Reviewer comment (Experiments):
The taxonomy is manually defined based on the interpretation of previous literature, and the datasets used in the study are intentionally clean (MNIST and CIFAR) to simulate hard samples. Perhaps data hardness should be evaluated in a data-driven way, i.e., using a real world dataset and using data science, statistics, and machine learning to find out the nature of hard samples on the dataset. This reviewer finds the approach of defining the hardness of data artificial, and perhaps not reflecting the challenges of current problems and datasets.

Author response:
We agree with the Reviewer that, in the real world, data hardness should be evaluated in a data-driven way (e.g. with data science and ML methods): this is precisely what an HCM does when used to identify and audit hard samples in a dataset. However, benchmarking and evaluating HCMs and their detection capabilities requires the creation of a controlled benchmarking environment. We note real datasets do not have ground truth of hardness; hence, to accurately assess the detection capabilities of HCMs, we need to establish a ground truth in our benchmark which we do via clean datasets (CIFAR and MNIST). We clarify this selection is a strategic decision to establish a clear baseline and understand the specific capabilities of each HCM under known conditions. Moreover, these datasets are widely adopted in the HCM literature. We contend that the use of such datasets doesn't detract from the study's relevance; instead, it ensures that the evaluation of HCMs is precise.

We also wish to clarify that indeed the hardness types defined in our framework are reflective of challenges and problems in real data. **Figure 7 (now moved to the main paper as Figure 3)** illustrates that the hardness types are representative of hardness types that could realistically occur in actual datasets.

Reviewer comment (Evaluation):
The paper points to the use of indirect measures, such as improvement in downstream performance, as an issue (Sec. 2.2). However, it seems like the metric that they use to evaluate methods and create the benchmark is downstream performance in the corresponding classification task. It was not clear what the proposed solution to this issue is in the benchmark.

Author response:
We thank the reviewer for the opportunity to clear up this misunderstanding. By 'indirect,' we refer to the HCM literature only assessing changes in downstream performance rather than the actual hardness detection task –- i.e. we don’t know if HCMs do the task they claim to do. In contrast, we address the gap in the literature as shown in Table 1 and directly evaluate the capability of HCMs to detect specific hardness types. This approach allows us to understand which types of hardness each HCM can effectively identify, rather than merely assessing model improvement on a curated dataset, which might obscure the actual capabilities of different HCMs on different hardness types, offering valuable insights for the development and application of these methods.

Reviewer comment (Presentation):
The writing style and paper formatting is too flashy and very distracting. Instead of having a natural flow, the paper has sentences in bold, boxes in colors, and other decorations that ask for too much attention, giving the impression that there is something more important to read than the argument the reader is focusing on. I found this style very hard to follow. In addition, the paper reads as if it was trying hard to sell a toolkit rather than presenting a study with solid conclusions.

Author response:
We thank the reviewer for the suggestions to help improve the paper and its readability. We have made the following changes based on your suggestions.

- Changed the citation color from blue to black to reduce the color on the page.
- Reduced the in-text bolding, except to delineate sections.
- Removed the color boxes in many places
- We have also streamlined the description of the toolkit. This has allowed space for additional insights to be moved from the Appendix to the main paper

Reviewer comment (Novelty):
Unfortunately, I don't find this study very compelling as a full research article. As it is currently, the paper seems to be more appropriate for a Datasets & Benchmarks track at NeurIPS, for instance, rather than bringing in novel insights to advance hardness characterization methodology. Maybe I'm missing what the most important conclusion is, which is what I'd like the authors to clarify.

Author response:
We thank the reviewer for acknowledging the suitability of our paper for a benchmarking track. We wish to highlight that this year the ICLR call for papers includes a track specifically for **datasets and benchmarks** — which we noted as the primary submission area for this work. With this in mind, our work responds to the recent calls for more rigorous benchmarking and understanding of existing ML approaches, as highlighted by Guyon (2022), Lipton & Steinhardt (2019), and Snoek et al. (2018). 

Our work provides four main contributions to the field: **(1) Hardness taxonomy:** we formalize a systematic taxonomy of sample-level hardness types; **(2) Benchmarking framework:** to evaluate the strengths of different HCMs across hardness types; **(3) Systematic and quantitative HCM evaluation:** prior to our study, no research had comprehensively evaluated HCMs across various hardness types nor directly evaluated HCMs' capabilities to correctly detect hardness — we do for 13 different HCMs across 8 different hardness types;** (4) Insights:** our benchmark provides novel insights into the capabilities of different HCMs when dealing with different hardness types, exposing gaps and opportunities in current HCMs and offering practical tips for researchers and practitioners. We hope the H-CAT framework will both add rigor to the HCM field and address the critical need for a structured and empirical approach to understanding and improving HCM methodologies.

We also highlight that similar analyses proposing a framework and benchmark have also previously been accepted by ICLR:
 - A Fine-Grained Analysis on Distribution Shift (Wiles et al., 2022)
- BREEDS: Benchmarks for Subpopulation Shift (Santurkar et al., 2021)
- MEDFAIR: Benchmarking Fairness for Medical Imaging (Zong et al., 2023)
- Long Range Arena: A Benchmark for Efficient Transformers (Tay et al., 2021)
- A framework for benchmarking Class-out-of-distribution detection and its application to ImageNet (Galil et al., 2023)