Paper: Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on HuggingFace (ICLR-2024)

Reviewer comment (Evaluation):
The paper uses exact keyword matching to identify corresponding subsections. Thus it's hard to know the proportion of dataset cards which covers the corresponding subsection but with different keywords.

Author response:
Notably, **84%** of the occurrences of section titles in the 7,433 dataset cards align with the 27 titles provided by the HuggingFace community. This high level of adherence demonstrates the effectiveness of exact keyword matching as an analytical method. Additionally, the fact that only 11% of dataset cards do not include any of these 27 standardized titles further strengthens the argument. This low rate of deviation indicates that the majority of dataset cards follow a consistent structure, underscoring the reliability of exact keyword matching in identifying corresponding dataset card sections.

We have added this validation procedure to *Appendix C.1 Accessing and Parsing Dataset Cards* to clarify this point.

Reviewer comment (Experiments):
How are the human annotators selected? Are they machine learning domain experts?

Author response:
In our study, the human annotators were carefully chosen, taking into consideration their expertise and background in the field of AI. Specifically, we selected PhD students with a robust research background in NLP, Computer Vision, Human-AI, Machine Learning, and Data Science. Their extensive experience with AI and utilizing datasets ensured a deep understanding of dataset documentation. To confirm the reliability of our evaluation, we randomly sampled 30 dataset cards for the annotators to assess and achieved an Intraclass Correlation Coefficient (ICC) of **0.76**, which is considered a good agreement [2]. This high level of agreement, combined with the annotators' deep expertise in AI research, substantially reinforces the trustworthiness of the annotation results.

We have added the selection criteria of human annotators to *Appendix C.2 Human-Annotated Dataset Card Evaluation Methodology and Criteria*.

**Reference**

[1] Gebru, Timnit, et al. "Datasheets for datasets." Communications of the ACM 64.12 (2021): 86-92.

[2] Koo, Terry K., and Mae Y. Li. "A guideline of selecting and reporting intraclass correlation coefficients for reliability research." Journal of chiropractic medicine 15.2 (2016): 155-163.

Reviewer comment (Experiments):
Were the authors able to find a correlation between well documented datasets and their creators? Eg, Do they come from other disciplines? Are they academic or industry researchers? Start-ups...etc

Author response:
We thank reviewer kkaF for their positive assessment of our paper and for recognizing the importance of our work. We appreciate their support and would like to address the queries.

> *Q1. "Were the authors able to find a correlation between well documented datasets and their creators? Eg, Do they come from other disciplines? Are they academic or industry researchers? Start-ups...etc"*

Our analysis of 1,982 distinct creators who contribute to the 7,433 dataset cards, reveals significant variations in dataset documentation practices based on the creators' backgrounds. The table below summarizes the results of our analysis (we have added this table as Table S3 in Appendix F of our manuscript):

| Category              | Description                                                                                                                              | Dataset Number | Adherence to Guidelines | Avg. Word Count |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -------------- | ----------------------- | --------------- |
| Industry organization | Companies (e.g.[Hugging Face](https://huggingface.co/huggingface), [facebook](https://huggingface.co/facebook))                                 | 2527           | **0.34**          | 219             |
| Academic organization | Universities, Research Labs (e.g.[Stanford CRFM](https://huggingface.co/stanford-crfm), [jhu-clsp](https://huggingface.co/jhu-clsp))           | 985            | 0.31                    | **427**   |
| Community             | Non-profit Communities (e.g.[bio-datasets](https://huggingface.co/bio-datasets), [keras-dreambooth](https://huggingface.co/keras-dreambooth)) | 1387           | 0.27                    | 190             |
| Industry professional | Engineers, Industry Scientists                                                                                                           | 985            | 0.25                    | 256             |
| Academic professional | Students, Postdocs, Faculty                                                                                                              | 672            | 0.16                    | 180             |
| All dataset card      |                                                                                                                                          | 7433           | 0.29                    | 234             |

Industry organizations, contributing 34% of the datasets and 90% of the top 100 downloads, lead in adherence to guidelines with an average word count of 219, indicating a commitment to detailed documentation. Academic organizations, though fewer in number, showcase comprehensive documentation with the highest average word count of 427. Academic professionals, however, show lower adherence to guidelines and a reduced word count, pointing to a varied relationship between the creator's sector and their approach to dataset documentation.

The information about these creators is gathered from their linked GitHub, Twitter, and personal websites on their Hugging Face profiles. Generally, datasets produced by organizations tend to be of higher quality compared to those created by individuals. Additionally, industry-generated dataset cards often surpass those from academic sources in quality.

Reviewer comment (Evaluation):
Is there a relationship between the features included in the dataset and whether there is going to be a limitations section? ie do datasets that contain more sensitive data likely to be well documented and include limitations sections?

Author response:
> *Q2. Is there a relationship between the features included in the dataset and whether there is going to be a limitations section? ie do datasets that contain more sensitive data likely to be well documented and include limitations sections?*

We thank Reviewer kkaF's intriguing question about the relationship between dataset sensitivity and documentation quality. We delved into the features of the dataset and found about 36% dataset may contain sensitive data among the 577 dataset cards that have the "Considerations for Using the Data" section.

| Category     | Number | Example                                                                                                                                                                                                                                                                                  |
| ------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Website Data | 158    | (1) "Since our dataset contains tables that are scraped from the web, it will also contain many toxic, racist, sexist, and otherwise harmful biases and texts." (2) "We are aware that since the data comes from public web pages, some biases may be present in the dataset."              |
| News Data    | 62     | (1)"News articles have been shown to conform to writing conventions" (2)"As one can imagine, data contains contemporary public figures or individuals who appeared in the news."                                                                                                         |
| Medical Data | 46     | (1)"A handful of disease concepts were discovered that were not included in MEDIC." (2)"This dataset of protein interactions was manually curated by experts utilizing published scientific literature."                                                                                  |
| Voice Data   | 11     | (1)"The Mongolian and Ukrainian languages are spelled as "Mangolian" and "Ukranian" in this version of the dataset." (2)"The dataset consists of people who have donated their voice online. You agree to not attempt to determine the identity of speakers in the Common Voice dataset." |

It is notable that as discussed in Section 5 of our paper, there are various topics being discussed in the "Considerations fo Using the Data" section, like the technical limitations and biases. Some dataset curators might focus solely on technical biases, neglecting ethical aspects (e.g., [allenai/nllb](https://huggingface.co/datasets/allenai/nllb#considerations-for-using-the-data)). Moreover, the proportion of datasets containing sensitive data that include the "Considerations for Using the Data" section is also low. For instance, a mere 9.2% of medical datasets incorporate this critical consideration. Coupled with our discovery that the "Considerations for Using the Data" section constitutes only 2.1% of the total text, this underscores a substantial need for improvement within this domain.

In response to your encouraging comments, we are motivated to continue our work and uncover more insights about dataset documentation.

Reviewer comment (Novelty):
Most finding in the paper are “common sense” i.e. there always has been a high correlation with quality of documentation and usability of a particular tool/dataset.

Author response:
**"Common Sense" Findings**

> *"Most finding in the paper are "common sense""*

While some of our findings may seem intuitive, our study uniquely provides **empirical** backing for these insights, which has not been systematically shown before. The link between documentation quality and dataset usability, while anticipated, has not been empirically confirmed on such a large-scale dataset until now. Our comprehensive analysis of 7,433 dataset cards from the Hugging Face platform, a leading ML platform, offers robust statistical support, moving our understanding from theoretical to empirical.
For instance, our study uncovers a notable lack of emphasis on crucial sections like "Dataset Creation" and "Considerations for Using the Data," which make up just 10% and 2.1% of the total text in dataset cards, respectively. These insights are not only novel but also practical, highlighting significant opportunities for enhancing dataset documentation practices.

Moreover, the scalability and reproducibility of our analysis methodology are noteworthy. It can be readily applied to other platforms, like GitHub, which similarly utilizes README files for dataset documentation. This study represents a pivotal step in conducting extensive empirical analysis of dataset documentation practices, with a focus on Hugging Face as a leading case study in the ML community. We hope to uncover and examine the prevalent norms and methods currently employed in dataset documentation. By doing so, we aim to not only underscore the vital importance of ongoing improvements in dataset documentation practices but also contribute to the responsible and ethical deployment of AI community.

Reviewer comment (Experiments):
Datasets may be simply popular because they are in a “hot” area of study.

Author response:
**Insights of Popular Datasets**

> *"Datas may be simply popular because they are in a “hot” area of study. Another example is the quality of the underlying dataset"*

The correlation between dataset popularity and documentation quality can **inform better dataset card design and practice**. While popularity may stem from being in a 'hot' area or the underlying quality of the dataset and can provide insights into what current research is focusing on and predict the future hot topic, our study offers a more nuanced perspective, exploring the relationship between dataset documentation and their popularity. This approach reveals distinct documentation patterns between popular and less popular datasets, enabling us to **identify good practices and common areas for improvement**. For instance, we observed that popular datasets often adhere better to the Hugging Face community-endorsed structure for dataset cards and exhibit higher word counts. These characteristics can serve as practical guidelines for dataset creators looking to improve their dataset cards. Additionally, we identified a consistent lack of emphasis on the "Dataset Creation" and "Considerations for Using the Data" sections across both popular and less popular datasets. This finding highlights the need for the community to create more compelling incentives, encouraging dataset creators to enhance documentation, particularly for these critical sections that are often neglected.

Reviewer comment (Experiments):
The quality of the underlying dataset may be the reason one of the datasets is very popular, which could simply be attributed to the fact that there are many studies leveraging this dataset (which in some sense is correlated with the quality of the data).

Author response:
**Insights of Popular Datasets**

> *"Datas may be simply popular because they are in a “hot” area of study. Another example is the quality of the underlying dataset"*

The correlation between dataset popularity and documentation quality can **inform better dataset card design and practice**. While popularity may stem from being in a 'hot' area or the underlying quality of the dataset and can provide insights into what current research is focusing on and predict the future hot topic, our study offers a more nuanced perspective, exploring the relationship between dataset documentation and their popularity. This approach reveals distinct documentation patterns between popular and less popular datasets, enabling us to **identify good practices and common areas for improvement**. For instance, we observed that popular datasets often adhere better to the Hugging Face community-endorsed structure for dataset cards and exhibit higher word counts. These characteristics can serve as practical guidelines for dataset creators looking to improve their dataset cards. Additionally, we identified a consistent lack of emphasis on the "Dataset Creation" and "Considerations for Using the Data" sections across both popular and less popular datasets. This finding highlights the need for the community to create more compelling incentives, encouraging dataset creators to enhance documentation, particularly for these critical sections that are often neglected.

Reviewer comment (Evaluation):
Although there are several findings with respect to what makes a dataset popular, there isn’t a conclusive suggestion for improving dataset sharing/contribution.

Author response:
**Suggestion for Improving Dataset Sharing/Contribution**


> *"It seems a although there are several findings with respect to what makes a dataset popular, there isn’t a conclusive suggestion for improving dataset sharing/contribution."*


We offer recommendations for improving dataset documentation for better dataset sharing, drawing from key findings from our research:


- **Adherence to Community-Endorsed Dataset Card Structure**: Our analysis indicates that highly downloaded dataset cards often comply with the Hugging Face community-endorsed structure. We advocate for dataset creators to consistently adhere to this structure, enhancing standardization and clarity.
- **Emphasis on Critical Sections:** We advise dataset creators to focus more on the "Dataset Creation" and "Considerations for Using the Data" sections, which are the least detailed in the current documentation. These sections are vital for dataset reproducibility and usability, and addressing them properly can mitigate potential limitations and ethical concerns.
- **Content Suggestions for Each Section:** We delve into the specifics of each section and subsection, providing suggestions for practitioners to write the sections. For instance, in the "Considerations for Using the Data" section, they can discuss both technical and social aspects of dataset limitations and impacts.
- **Inclusion of a 'Usage' Section:** Despite its absence in the Hugging Face community-endorsed template, our analysis reveals that a "Usage" section is frequently included in dataset cards and significantly enhances dataset transparency and usability. We recommend the community incorporate this section into the standard template.
- **Humain Evaluation of Dataset Card Quality:**  Through human evaluation, we discovered a strong correlation between the comprehensiveness of dataset cards and their overall quality. We suggest that creators improve thoroughness in their documentation.


Overall, improving the dataset documentation can help improve the reproducibility, transparency, and accessibility of the dataset, which can help promote the dataset sharing and contribution ecosystem. These suggestions are implied in our manuscript, encompassing sections 4 to 6, as well as in the Discussion section.

Reviewer comment (Evaluation):
The correlation between documentation length and popularity is not causal.

Author response:
**The Correlation Between Dataset Cards and Dataset Popularity**


> *"Q1. The correlation between documentation length and popularity is not causal."*
>
> * *"I’d emphasize on the quality of the documentation and what points these “popular cards” address in the written documentation there may be a more granular set of points mentioned in each that results in their popularity**"*


The analysis between the dataset documentation practices and the dataset popularity reveals a strongly positive relationship, in aspects of adherence to the template, word count, and the fill-out rate within each section and subsection. The main findings include:


- **Better Adherence to Community-recommended Dataset Card:** Popular datasets exhibit better adherence to the community-recommended documentation structure.
- **Longer Word Length:** Popular datasets feature a higher word count, particularly notable in the "Dataset Description" section.
- **Higher Fill-Out Rate across Dataset Structure:** There's a higher completion rate across all documentation sections in these datasets, with a notable emphasis on the Dataset Structure section. A well-documented structure enhances the understanding and usability of the dataset.


The table below illustrates the completion rates in different sections of the documentation (we have added this result as Figure S6 in Appendix F in our manuscript):


|                   | Dataset Description | Dataset Structure | Dataset Creation | Considerations for Using the Data | Additional Information |
| ----------------- | ------------------- | ----------------- | ---------------- | --------------------------------- | ---------------------- |
| Top 100           | 0.93                | 0.86              | 0.34             | 0.15                              | 0.91                   |
| All dataset cards | 0.47                | 0.35              | 0.16             | 0.08                              | 0.61                   |


By examining the correlation between documentation practices and dataset popularity, we gain insights into effective documentation strategies. The strengths observed in popular datasets can guide other creators, serving as a template for effective documentation. Notably, the lower completion rates in sections like "Considerations for Using the Data" in even the most downloaded datasets highlight areas needing more focus. These findings can also serve as a benchmark for the community to establish standards in dataset documentation.

Reviewer comment (Reproducibility):
Greater emphasis should be placed on the quality of the documentation and the specific points these “popular cards” address in the written documentation, as there may be a more granular set of points mentioned in each that results in their popularity.

Author response:
**The Correlation Between Dataset Cards and Dataset Popularity**


> *"Q1. The correlation between documentation length and popularity is not causal."*
>
> * *"I’d emphasize on the quality of the documentation and what points these “popular cards” address in the written documentation there may be a more granular set of points mentioned in each that results in their popularity**"*


The analysis between the dataset documentation practices and the dataset popularity reveals a strongly positive relationship, in aspects of adherence to the template, word count, and the fill-out rate within each section and subsection. The main findings include:


- **Better Adherence to Community-recommended Dataset Card:** Popular datasets exhibit better adherence to the community-recommended documentation structure.
- **Longer Word Length:** Popular datasets feature a higher word count, particularly notable in the "Dataset Description" section.
- **Higher Fill-Out Rate across Dataset Structure:** There's a higher completion rate across all documentation sections in these datasets, with a notable emphasis on the Dataset Structure section. A well-documented structure enhances the understanding and usability of the dataset.


The table below illustrates the completion rates in different sections of the documentation (we have added this result as Figure S6 in Appendix F in our manuscript):


|                   | Dataset Description | Dataset Structure | Dataset Creation | Considerations for Using the Data | Additional Information |
| ----------------- | ------------------- | ----------------- | ---------------- | --------------------------------- | ---------------------- |
| Top 100           | 0.93                | 0.86              | 0.34             | 0.15                              | 0.91                   |
| All dataset cards | 0.47                | 0.35              | 0.16             | 0.08                              | 0.61                   |


By examining the correlation between documentation practices and dataset popularity, we gain insights into effective documentation strategies. The strengths observed in popular datasets can guide other creators, serving as a template for effective documentation. Notably, the lower completion rates in sections like "Considerations for Using the Data" in even the most downloaded datasets highlight areas needing more focus. These findings can also serve as a benchmark for the community to establish standards in dataset documentation.

Reviewer comment (Evaluation):
This power-law may simply represent the underlying distribution of people interested in deep learning topics/domains.

Author response:
**Insights of Power Law Distribution**


> *"Q2. It’s not clear how the power law with respect to data usage is useful. This power-law may simply represent the underlying distribution of people interested in deep learning topics/domains (E.g. text (NLP) versus mortgage tabular data (Finance))"*


The emphasis on precise log-log scale line fitting goes beyond a simple long-tail distribution, offering nuanced insights into dataset usage. We further studied across various tasks and found that **the power law also persists within specific task domain**, despite variations in dataset counts across domains (e.g., 2128 in NLP, 217 in computer vision, 153 in audio, 137 in multimodal, 79 in tabular, with others lacking metadata). This highlights the robustness of the power-law distribution, which can reflect research trends within task domains. Understanding this distribution is crucial for comprehending how datasets are utilized in the machine learning community, aiding resource allocation, and guiding platform providers like Hugging Face in prioritizing high-demand dataset support.


We have added a figure to show the power law within different tasks (Figure S5) in Appendix F.

Reviewer comment (Evaluation):
Differences in topic domains, such as text (NLP) versus mortgage tabular data (Finance), may explain the observed power-law behavior.

Author response:
**Insights of Power Law Distribution**


> *"Q2. It’s not clear how the power law with respect to data usage is useful. This power-law may simply represent the underlying distribution of people interested in deep learning topics/domains (E.g. text (NLP) versus mortgage tabular data (Finance))"*


The emphasis on precise log-log scale line fitting goes beyond a simple long-tail distribution, offering nuanced insights into dataset usage. We further studied across various tasks and found that **the power law also persists within specific task domain**, despite variations in dataset counts across domains (e.g., 2128 in NLP, 217 in computer vision, 153 in audio, 137 in multimodal, 79 in tabular, with others lacking metadata). This highlights the robustness of the power-law distribution, which can reflect research trends within task domains. Understanding this distribution is crucial for comprehending how datasets are utilized in the machine learning community, aiding resource allocation, and guiding platform providers like Hugging Face in prioritizing high-demand dataset support.


We have added a figure to show the power law within different tasks (Figure S5) in Appendix F.

Reviewer comment (Reproducibility):
A key component in the dataset documentation is missing, e.g., an explanation of how the data is processed—perhaps this is covered in data usability.

Author response:
**Missing Key Component in Dataset Documentation**


> *"Q3. There’s a key component in the dataset documentation that is missing e.g. how the data is processed – perhaps this is covered in data usability?"*


There is a section called "Dataset Creation" in the community-endorsed dataset cards, and as the template suggests, it is divided into subsections like:


- **(1) Curation Rationale:** This subsection delves into the underlying motivation for creating the dataset, offering insight into its intended purpose and scope.
- **(2) Source Data:**
 - **(2.1) Initial Data Collection and Normalization:** Details the methodology employed in gathering and standardizing the initial data.
 - **(2.2) Who are the source language producers:** Identifies the original creators of the data, whether individuals or systems, providing context about the data's origins.
- **(3) Annotations:**
 - **(3.1) Annotation Process:** Describes the procedures and tools used for data annotation.
 - **(3.2) Who are the annotators:** States whether the annotations were produced by humans or machine-generated.


A specific example is the [ncbi_disease](https://huggingface.co/datasets/ncbi_disease#dataset-creation), comprising disease name and concept annotations from 793 fully annotated PubMed abstracts. This dataset exemplifies thorough documentation of its data collection and annotation processes, as demonstrated in the following examples:


> **Annotation process**
>
> Each PubMed abstract was manually annotated by two annotators with disease mentions and their corresponding concepts in Medical Subject Headings (MeSH®) or Online Mendelian Inheritance in Man (OMIM®). Manual curation was performed using PubTator, which allowed the use of pre-annotations as a pre-step to manual annotations. Fourteen annotators were randomly paired and differing annotations were discussed for reaching a consensus in two annotation phases. Finally, all results were checked against annotations of the rest of the corpus to assure corpus-wide consistency.


Despite the critical role of the "Dataset Creation" section in ensuring dataset reproducibility and usability, our analysis reveals a gap in documentation thoroughness. Even in the healthcare domain, where the data collection process is important, only around 23% provide comprehensive details on dataset creation. Our analysis of the varied emphasis across different sections of dataset documentation (See Section 4 and Fig 3 in our paper) also reveals that the "Dataset Creation" section is the second-least detailed section, accounting for merely about 10% of the text in the dataset cards. Given the importance of this dataset creation section in improving dataset usability and reproducibility, we recommend that dataset creators provide more comprehensive documentation of the dataset creation process in our paper.