Raison d’être of the benchmark dataset: A Survey of Current Practices of Benchmark Dataset Sharing Platforms (2022)

Abstract: This paper critically examines the current practices of benchmark dataset sharing in NLP and suggests a better way to inform reusers of the benchmark dataset. As the dataset sharing platform plays a key role not only in distributing the dataset but also in informing the potential reusers about the dataset, we believe data-sharing platforms should provide a comprehensive context of the datasets. We survey four benchmark dataset sharing platforms: HuggingFace, PaperswithCode, Tensorflow, and Pytorch to diagnose the current practices of how the dataset is shared which metadata is shared and omitted. To be specific, drawing on the concept of data curation which considers the future reuse when the data is made public, we advance the direction that benchmark dataset sharing platforms should take into consideration. We identify that four benchmark platforms have different practices of using metadata and there is a lack of consensus on what social impact metadata is. We believe the problem of missing a discussion around social impact in the dataset sharing platforms has to do with the failed agreement on who should be in charge. We propose that the benchmark dataset should develop social impact metadata and data curator should take a role in managing the social impact metadata.

## introduction benchmark datasets play a crucial role in developing the model. publicly available benchmark datasets serve as a baseline proxy to measure the model's performance and an evaluation as the machine learning (ml) and natural language processing (nlp) scholarship competes for the higher ground. recent works have started to question the validity of such benchmark datasets regarding their generalizability @xcite , documentation practices @xcite , and social impact @xcite , amongst others. @xcite focus on the way how benchmark datasets are collected and used and advocate cautious understanding of data in order to address ethical issues of using such datasets. @xcite suggest the criteria benchmarks should qualify, namely the robustness, statistical power, and considerations of social impact. however, despite the fact that the documentation of benchmark datasets and the role of the dataset sharing platform are pivotal not only in informing the users about the benchmark dataset but also soliciting a safe use, it has been relatively understudied. we believe that critically examining the current practices of dataset sharing platforms -which metadata is documented and omitted -and suggesting desiderata for data sharing platforms can serve as a practical guide for users and researchers in encouraging a safe environment. our findings show that current practices of dataset sharing platforms are highly centered on reusable purpose, which focuses on the convenience of the users in making use of the dataset. for example, it provides detailed explanations of how to load the dataset into actual development, how the test and train split are made. it was hard, on the other hand, to find the documentation of the limitations of the dataset (e.g. which societal impacts it may bring); even if there were, the concepts and definitions were often elusive. we introduce the concept of social impact metadata which is the documenting practice done in library and information science in order to advocate mitigating possible social harms. we propose desiderata for documenting benchmark datasets. beyond descriptive and administrative metadata, the documents of the metadata should also include the social impact metadata. to make it possible, we highly encourage developing the social impact metadata (e.g. demographic statistics of the data) and also emphasize a role of the data curator who is responsible for documenting in terms of the data sharing platforms. ## definitions in order to narrow down the conceptual difference that may conflict between the ml (and nlp) community and library and information science community, we introduce the definition of the key terms that will be used throughout the paper. ## social impact of benchmark dataset sharing platforms once the benchmark dataset is made available for others, data friction comes into play. data friction explains a point of resistance where data can be garbled, misinterpreted, or lost (edwards et al., 2011). as edwards et al. (2011) argue, researchers' main interest is in using data, not in describing the dataset for the benefit of invisible, unknown future users. the problem of benchmark dataset sharing arise because text-as-data and computation is no longer exclusive field of nlp and ml (monroe @xcite . the benchmark dataset can be easily used by researchers outside of nlp and ml community. for instance, now in the name of digital humanities, researchers in humanities also use computational approaches and technologies with historical text data @xcite @xcite . likewise, if the benchmark dataset is shared through the sharing platforms, ml and nlp researchers will make derivative models based on the benchmark dataset and this will lead the researchers outside of ml and nlp to indirectly impacted by benchmark dataset without knowing the social impact of the dataset. even though humanists and social scientists may not fine-tune the parameters of the model itself, their research will be impacted by how the benchmark dataset is designed and constructed. the use of pre-trained model from nlp community by researchers outside of nlp and ml can be found in the case of politeness detection model. the original idea of developing a nlp model for detecting politeness from language is from danescu-niculescu-mizil et al. (2013). as the automatic scoring of the politeness in the language has benefits regardless of the field, hoffman et al. (2017) attempted to reproduce and validate danescu-niculescu-mizil et al. (2013). in doing so, hoffman et al. (2017) applied the same model to wikipedia, which is the identical domain and found unexpected results that led them to question the quality of the dataset. their conclusion called for an investigation on research which reused the dataset that danescu-niculescu-mizil et al. (2013) developed. if the quality of dataset is spurious, then it is hard to say the following research building on the questionable dataset can avoid critics. nonetheless, the politeness corpus of danescu-niculescu-mizil et al. ( 2013 ) is now incorporated into the r package @xcite , allowing researchers from outside of ml community can easily load the package and analyze the data. there are some papers already utilized politeness corpus from outside of nlp and ml for social science research purpose @xcite . at this point, we do not know how to measure the social impact of politeness detection dataset and the derivative package will bring. for ml and nlp researchers, identifying who is responsible for assessing the social impact and alarm-ing the benchmark dataset reusers is now more than important. we can find another example of social impact of malfunctioned benchmark dataset in the recent development of a chatbot called 'lee ruda'. 'lee ruda' showed how artificial intelligence systems can jeopardize sexual minorities by exposing them to toxic communication space @xcite . if the data documentation process is not shared in the benchmark dataset sharing platforms and discussion around social impact of the dataset is not mature enough to alert reusers, the social impact of benchmark dataset can be catastrophic. in this vein, @xcite also emphasizes how naive use of the datasets may cause problems on the society by directly deploying the trained model into the society. ## current practices of benchmark dataset sharing platforms we investigated the platforms that practitioners and researchers largely accessed for datasets. this resulted in four main platforms: huggingface foot_0 , pa-perswithcode foot_1 , tensorflow foot_2 , and pytorch foot_3 . we focused on whether it provides users easy access to datasets along with its metadata. as for paperswith-code, it did not provide direct access to datasets however, it offered detailed information of data such as the papers that used (cited) the datasets. we excluded the platforms that were managed by the users themselves, such as github, as it was mostly uploaded by the data creators themselves, rather than other contributors that curated the dataset for ease of use. huggingface huggingface provides an infrastructure so ml researchers can easily leverage models and datasets. the idea of hugging-face is similar to github, where the codes and data are shared. in huggingface, it is the language model trained by different groups of researchers in nlp that is shared. of many language models available in huggingface, what made huggingface famous is transformers, which enabled loading thousands of deep learning frameworks (pytorch, tensorflow, jax) as well as language models (e.g., bert, roberta, gpt) with a single line of code. the dataset card for social impact, biases, and unknown limitations is developed to reflect the growing concern around the social impact of the ml benchmark dataset @xcite . paperswithcode paperswithcode organizes the research works from the ml community by providing three access points: tasks, datasets, and methods. paperswithcode do not house the datasets but rather provide a reference point where you can find the research worked on the specific benchmark dataset. the dataset section was organized with brief information about the dataset, relevant papers which reused the benchmark dataset, on which tasks the benchmark dataset was used, and where researchers can find the benchmark dataset. for instance, paperswithcode introduces glue dataset with additional information that it can be found from hugging face and tensorflow. ## results as one would expect, the essential role of these platforms was focused on helping the users easily fetch the dataset and use it without putting in an extra endeavor. for example, the datasets were well-curated into train and test set splits, so that users can readily reproduce, and custom it to their own task. however, when it came to sharing auxiliary information (metadata) regarding the dataset, such as its limitations, and societal impacts, a large portion of the platforms lacked providing detailed information. in this section, we introduce the metadata types used in benchmark dataset sharing platforms and summarized in ## desiderata of data sharing platforms the caveat of using benchmark dataset and visà-vis social impact is gaining attention from the ml community @xcite . however, we believe the benchmark dataset sharing platform is not currently up-to-date because the current discussion around the benchmark dataset is missing. we acknowledge that the endeavor of dataset creators is crucial in developing a safe benchmark ecosystem, however, in this work we typically focus on the data sharing platforms. from our analysis, there are many loopholes to fill. paperswithcode, tensorflow, and pytorch emphasized descriptive and administrative metadata while neglecting the importance of the social impact that the benchmark dataset can bring. we want to reiterate that even though data documentation recorded the entire process of dataset creation perfectly, data friction @xcite could happen when it was made available for others for reuse purposes. therefore, dataset sharing platforms should take initiative to inform the social impact of the benchmark datasets by critically assessing the datasets. from a data curation perspective, it is unclear who is responsible for organizing the information of social impact, biases, and other limitations. ## conclusion we believe the documentation of the benchmark dataset plays an important role as it introduces the pitfalls as well as the usage of the dataset. to this end, we examine current practices of widely accessed benchmark dataset sharing platforms -what is documented and what is omitted -. our findings suggest the need for documenting the social impact of the benchmark dataset as well as assigning the data curators for data sharing platforms to be in-charge of documenting relevant metadata.