Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the H ugging F ace and GEM Data and Model Cards (2021)

Abstract: Developing documentation guidelines and easy-to-use templates for datasets and models is a challenging task, especially given the variety of backgrounds, skills, and incentives of the people involved in the building of natural language processing (NLP) tools. Nevertheless, the adoption of standard documentation practices across the field of NLP promotes more accessible and detailed descriptions of NLP datasets and models, while supporting researchers and developers in reflecting on their work. To help with the standardization of documentation, we present two case studies of efforts that aim to develop reusable documentation templates – the HuggingFace data card, a general purpose card for datasets in NLP, and the GEM benchmark data and model cards with a focus on natural language generation. We describe our process for developing these templates, including the identification of relevant stakeholder groups, the definition of a set of guiding principles, the use of existing templates as our foundation, and iterative revisions based on feedback.

## introduction dataset and model documentation is a necessary step in identifying potential issues with machine learning (ml) systems and addressing their broader impacts @xcite @xcite . in their overview of data collection and use in ml, @xcite identify issues that have frequently arisen such as considerations for how subjects are represented in datasets, spurious cues that may be exploited by ml model, and concerns about the content in datasets collected through crawling methodologies. they advocate for careful documentation of datasets and their collection processes in order to surface these problems. however, best practices for documentation have seen no widespread adoption even for the most popular datasets and models. indeed, writing such detailed documentation requires additional effort from researchers who may lack the required resources or familiarity with the process. providing dataset and model creators with guidelines and several examples in a single place to inspire and inform prospective writers could thus drive widespread adoption of documentation. research efforts that involve a large number of models or datasets are particularly well positioned to develop and maintain specific guidelines and best practices by making documentation a required component for submitting contributions. by bringing together the domain expertise of participants and the experience of researchers who have a greater familiarity with documenting data and models, these efforts provide an opportunity to develop and refine templates that balance generality and informativeness. in addition, by requiring appropriate documentation for any involved model or dataset, these efforts can set a precedent that informs future endeavours. we encourage organizations to consider their role in the successful uptake of documentation practices, such as providing their members with adequate resources to understand the goals and motivations of documentation and measured steps towards integrating documentation into current research norms. in this paper, we present two case studies of creating documentation templates and guides in natural language processing (nlp): the hugging face (hf) dataset hub 1 and the benchmark for gen-eration and its evaluation and metrics (gem). 2 we use the term data card to refer to documentation for datasets in both cases and the term model card to refer to documentation for models in the gem workshop, following @xcite . focusing on these settings allows us to ground what constitutes 'good' documentation in these contexts, namely technical user-oriented information, scientific reproducibility, and social contextualization of data and data-driven systems. ## related work in the u.s., research involving direct interventions on human subjects is subject to the federal policy for the protection of human subjects (or common rule). 3 this policy tasks institutional review boards (irbs) with certifying that such research follows established ethical standards and regulations. while this review is sometimes decried as cumbersome @xcite , the process ensures that researchers both reflect and communicate ahead of time how the data will be collected and used, why it is necessary to answer the question at hand, and how protected information will be handled to prevent harm to the human subjects. whereas much of the data that supports current methods in ml and especially nlp is created by, gives information about, and is used to train models that will likely affect these same human subjects, this relationship does not constitute a direct intervention as defined in the common rule. however, metcalf and crawford argue that the definition is too narrow when one considers the similarity in potential harms and advise that data-driven methods should be subject to a similar from of ethical review, which includes clear communication about the goals and mechanisms for collecting and safeguarding the data. despite existing literature on database documentation in hci and related fields @xcite , documentation in ml has only recently gained traction. in a survey of ml projects across india, east and west african countries, and the usa, @xcite analyze compounding events causing negative, downstream effects from data issues, resulting in technical debt foot_3 over time, and identify insuffi-cient documentation to be a trigger of these events in 20% of the 53 cases. they report impacts such as time and effort lost in using incorrect data, barriers to completing models, and data that had to be abandoned due to it no longer being usable. in response to issues like the ones found by sambasivan et al., many research groups have proposed documentation schemata for different parts of the ml pipeline. arnold et al. (2019) to address the distinct challenges of working with language data, such as those summarized by @xcite , other researchers have proposed specialized documentation for work in nlp. the first such example is by @xcite who propose a version of data statements for documenting aspects of the data from a linguistic perspective. in addition to documenting the data that a model sees during training, there is also the need to document experiments, especially those involving humans. thus, @xcite develop a human evaluation datasheet, with the goal of describing human evaluation experiments rather than naturally occurring language data. starting from the templates by @xcite and @xcite , we iteratively extend and adapt our templates for the hf and gem contexts. ## methods to guide the development of our templates, we draw from the methods of value sensitive design (eg., friedman and hendry, 2019) by identifying stakeholders and assessing their values, which friedman and hendry define as "what is important to people in their lives, with a focus on ethics and morality" (pg. 24). the values of the various stakeholders, including developers themselves, may influence the development of a technology in a number of ways @xcite . in this section, we conduct a stakeholder analysis and describe the principles we follow in the development of our templates. in section 7, we explore the potential social impact of the templates and our positionality in the design process. ## case study i: huggingface data cards as a first case study of data card development, we present the template developed for the hugging-face open source nlp libraries. the full template is available in ## case study ii: gem data and model cards our second case study of data card development is the template we developed for the datasets and model submissions of the generation, its evaluation, and metrics (gem) workshop. we present the full template in ## conclusion we detail the processes and principles we followed to produce the documentation templates used in the hf library and the gem benchmark. we ground this work in the current discussion of documentation as a way to communicate the impacts of ml systems. as touched on in section 3.2, extensive documentation is only a tool to support the communication of decisions that led to the creation of datasets and models and the positionality of their creators; it is not a direct solution to the harms caused by ml systems. we present our templates to encourage others to consider important questions that may be asked of their own work. the templates from both case studies are open source and we welcome contributions and feedback from authors and users to continually revise and improve them. moreover, while the templates described in this work are designed for specific contexts and may not be fully applicable to others, they can be used as starting points for adaption to other settings.