Datasets: A Community Library for Natural Language Processing (2021)

Abstract: The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets .

## introduction datasets are central to empirical nlp: curated datasets are used for evaluation and benchmarks; supervised datasets are used to train and fine-tune models; and large unsupervised datasets are necessary for pretraining and language modeling. each dataset type differs in scale, granularity and structure, in addition to annotation methodology. historically, new dataset paradigms have been crucial for driving the development of nlp, from the hansard corpus for statistical machine translation @xcite to the penn treebank for syntactic modeling @xcite to projects like opus and universal dependencies @xcite which bring together cross-lingual data and annotations. contemporary nlp systems are now developed with a pipeline that utilizes many different datasets at significantly varying scale and level of annotation @xcite . different datasets are used for pretraining, fine-tuning, and benchmarking. as such, there has been a large increase in the number of datasets utilized in the nlp community. these include both large text collections like c4 @xcite , fine-tuning datasets like squad @xcite , and even complex zero-shot challenge tasks. benchmark datasets like glue have been central to quantifying the the advances of models such as bert @xcite . the growth in datasets also brings significant challenges, including interface standardization, versioning, and documentation. a practitioner should be able to utilize n different datasets without requiring n different interfaces. in addition, n practitioners using the same dataset should know they have exactly the same version. datasets have also grown larger, and ideally interfaces should not have to change due to this scale, whether one is using small-scale datasets like climate fever (∼1k data points), medium-scale yahoo answers (∼1m), or even all of pubmed (∼79b). finally, datasets are being created with a variety of different procedures, from crowd-sourcing to scraping to synthetic generation, which need to be taken into account when evaluating which is most appropriate for a given purpose and ought to be immediately apparent to prospective users @xcite . datasets is a community library designed to address the challenges of dataset management and access, while supporting community culture and norms. the library targets the following goals: datasets is in continual development by the engineers at hugging face and is released under an apache 2.0 license. 1 the library is available at https://github.com/huggingface/ datasets . full documentation is available through the project website. foot_1 2 related work there is a long history of projects aiming to group, categorize, version, and distribute nlp datasets which we briefly survey. most notably, the linguistic data consortium (ldc) stores, serves, and manages a variety of datasets for language and speech. in addition to hosting and distributing corpus resources, the ldc supports significant annotation efforts. other projects have aimed to collect related annotations together. projects like ontonotes have collected annotations across multiple tasks for a single corpus (pradhan and xue, 2009) whereas the universal dependency treebank @xcite collects similar annotations across languages. in machine translation, projects like opus catalog the translation resources for many different languages. these differ from datasets which collects and provides access to datasets in a content-agnostic way. other projects have aimed to make it easy to access core nlp datasets. the influential nltk project @xcite provided a data library that makes it easy to download and access core datasets. spacy also provides a similar loading interface @xcite . in recent years, concurrent with the move towards deep learning, there has been a growth in large freely available datasets often with less precise annotation standards. this has motivated cloud-based repositories of datasets. initiatives like tensorflow-datasets (2021) and torchtext (2021) have collected various datasets in a common cloud format. this project began as a fork of tensorflow-datasets, but has diverged significantly. datasets differs from these projects along several axes. the project is decoupled from any modeling framework and provides a general-purpose tabular api. it focuses on nlp specifically and provides specialized types and structures for language constructs. finally, it prioritizes community management and documentation through the dataset hub and data cards, and aims to provide access to a long-tail of datasets for many tasks and languages. ## library tour and design we begin with a brief tour. accessing a dataset is done simply by referring to it by a global identity. each dataset has a features schema and metadata. ## dataset documentation and search datasets is backed by the dataset hub foot_3 that helps users navigate the growing number of available resources and draws inspiration from recent work calling for better documentation of ml datasets in general @xcite and nlp datasets in particular @xcite . datasets can be seen as a form of infrastructure @xcite . nlp practitioners typically make use of them with a specific goal in mind, whether they are looking to answer a specified research question or developing a system for a particular practical application. to that end, they need to be able to not only easily identify which dataset is most appropriate for the task at hand, but also to understand how various properties of that best candidate might help with, or, conversely, run contrary to their purpose. the dataset hub includes all of the datasets available in the library. it links each of them together though: a set of structured tags holding information about their languages, tasks supported, licenses, etc.; a data card based on a template foot_4 designed to combine relevant technical considerations and broader context information @xcite ; and a list of models trained on the dataset. both the tags and data card are filled manually by the contributor who introduces the dataset to the library. figure 1 presents an example of the dataset page on the hub. 6 together, these pages and the search interface help users navigate the available resources. choosing a dataset given a use case, the structured tags provide a way to surface helpful datasets. for example, requesting all datasets that have the tags for spanish language and the question answering task category returns 7 items at the time of writing. a user can then refine their choice by reading through the data cards, which contain sections describing the variety of language used, legal considerations including licensing and incidence of personal identifying information, and paragraphs about known social biases resulting from the collection process that might lead a deployed model to cause disparate harms. using a dataset the data card also contains information to help users navigate all the choices, from hardware to modeling, that go into successfully training a system. these include the number of examples in each of the dataset splits, the size on disk of the data, meaningful differences between the training, validation, and test split, and free text descriptions of the various fields that make up each example to help decide what information to use as input or output of a prediction model. the data card as a living document a dataset's life continues beyond its initial release. as nlp practitioners interact with the dataset in various ways, they may surface annotation artifacts that affect the behavior of trained models in unexpected ways (gururangan et al., 2018), 7 issues in the way the standard split was initially devised to test a model's ability to adapt to new settings @xcite , or new understanding of the social biases exhibited therein @xcite . the community-driven nature of datasets and the versioning mechanisms provided by the github backend provide an opportunity to keep the data cards up to date as information comes to light and to make gradual progress toward having as complete documentation as possible. ## dataset usage and use-cases datasets is now being actively used for a variety of tasks. figure 2 (left) shows statistics about library usage. we can see that the most commonly downloaded libraries are popular english benchmarks such as glue and squad which are often used for teaching and examples. however there is a range of popular models for different tasks and languages. figure 2 (right) shows the wide coverage of the library in terms of task types, sizes, and languages, with currently 681 total datasets. during the development of the datasets project, there was a public hackathon to have community members develop new dataset builders and add them to the project. this event led 485 commits and 285 unique contributors to the library. recent work has outlined the difficulty of finding data sources for lowerresourced languages through automatic filtering alone @xcite . the breadth of languages spoken by participants in this event made it possible to more reliably bootstrap the library with datasets in a wide range of different languages. finally while datasets is designed for nlp, it is becoming used for multi-modal datasets. the library now includes types for continuous data, including multi-dimensional arrays for image and video data and an audio type. ## additional functionality and uses streaming some datasets are extremely large and cannot even fit on disk. datasets includes a streaming mode that buffers these datasets on the fly. this mode supports the core map primitive, which works on each data batch as it is streamed. datasets streaming helped enable recent research into distributed training of a very large open nlp model @xcite . indexing datasets includes tools for easily building and utilizing a search index over an arbitrary dataset. to construct the index the library can interface either with faiss or elasticsearch @xcite . this interface makes it easy to efficiently find nearest neighbors either with textual or vector queries. indexing was used to host the open-source version of retrieval-augmented generation @xcite , a generation model backed by the ability to query knowledge from large-scale knowledge sources. metrics datasets includes an interface for standardizing metrics which can be documented, versioned and matched with datasets. this functionality is particularly useful for benchmark datasets 9 https://robustnessgym.com/ figure 3 : datasets viewer is an application that shows all rows for all datasets in the library. the interface allows users to change datasets, subsets, and splits, while seeing the dataset schema and metadata. data viewer a benefit of the standardized interface of the library is that it makes it trivial to build a cross-task dataset viewer. as an example, hugging face hosts a generic viewer for the entirety of datasets (figure 3 ) 10 . in this viewer, anyone on the web can open all almost 650 different datasets and view any example. because the tables are typed, the viewer can easily show all component features, structured data, and multi-modal features. ## conclusion hugging face datasets is an open-source, community-driven library that standardizes the processing, distribution, and documentation of nlp datasets. the core library is designed to be easy to use, fast, and to use the same interface for datasets of varying size. at 650 datasets from over 250 contributors, it makes it easy to use standard datasets, has facilitated new use cases of cross-dataset nlp, and has advanced features for tasks like indexing and streaming large datasets.