Documenting Geographically and Contextually Diverse Language Data Sources (2024)

Abstract: Contemporary large-scale data collection efforts have prioritized the amount of data collected to improve large language models (LLM). This quantitative approach has resulted in concerns for the rights of data subjects represented in data collections. This concern is exacerbated by a lack of documentation and analysis tools, making it difficult to interrogate these collections. Mindful of these pitfalls, we present a methodology for documentation-first, human-centered data collection. We apply this approach in an effort to train a multilingual LLM. We identify a geographically diverse set of target language groups (Arabic varieties, Basque, Chinese varieties, Catalan, English, French, Indic languages, Indonesian, Niger-Congo languages, Portuguese, Spanish, and Vietnamese, as well as programming languages) for which to collect metadata on potential data sources. We structure this effort by developing an online catalogue in English as a tool for gathering metadata through public hackathons. We present our tool and analyses of the resulting resource metadata, including distributions over languages, regions, and resource types, and discuss our lessons learned.

## introduction current trends in developing large language models (llm) require the use of vast amounts of data (brown et al., 2020; gao et al., 2020; rae et al., 2021). typically, this data is collected from online sources, ranging from highly edited and structured text such as wikipedia to the myriad text and audiovisual components of web pages, e.g., collected by the common crawl foundation. @xcite however, recent research has raised concerns about the creation and use of such data resources. for instance, wikipedia is highly biased in terms of the topics covered and the demographics of its contributors, particularly along gender, race, and geographic lines (barera, 2020), resulting in concerns of representation in the technologies developed on wikipedia-derived data. data from common crawl has similarly been shown to correlate with country-level population density, relative access to the internet, and per capita gdp (dunn, 2020) and to contain significant amounts of hate speech and sexually explicit content (luccioni and viviano, 2021). irrespective of the data source, typical web-crawling collection practices have no structures for supporting informed consent beyond websites' own policies that users rarely read (cakebread, 2017; obar and oeldorf-hirsch, 2020). several documentation schemas for natural language processing (nlp) datasets (bender and friedman, 2018; gebru et al., 2021; holland et al., 2018; stoyanovich and howe, 2019; mcmillan-major et al., 2023) have been proposed to aid nlp researchers in documenting their datasets (gao et al., 2020; biderman et al., 2022; gehrmann et al., 2021; wang et al., 2021) and to retrospectively document and analyze datasets that were developed and released by others without thorough documentation (bandy and vincent, 2021; kreutzer et al., 2022; birhane et al., 2021; dodge et al., 2021). data docu-mentation to support transparency has gained traction, following calls for a reevaluation of the acquisition and use of data in machine learning (ml) at large (birhane and prabhu, 2021; jo and gebru, 2020; paullada et al., 2021; gebru et al., 2021; bender et al., 2021). building on this work, we propose a documentation-first and humancentered method for data collection for nlp that emphasizes consent, representation, self-determination, and privacy. using this method, we create a data catalogue for training multilingual llms that promotes responsible data collection and data subjects' rights to control over their own data. we conclude that starting documentation processes during the data collection phase can contribute to building a more representative dataset and allows for early identification of ethical concerns. our contributions consist of the data catalogue tool, foot_0 which remains openly available for use in collecting metadata and for searching existing entries, as well as the humancentered methodology of data collection in collaboration with language communities for representative language modeling and other nlp tasks. ## related work since the early 90s, nlp data organizations have maintained catalogs for datasets and tools in order to support language research. 5 while the metadata for these catalogs are openly available, accessing the language resources (e.g., annotated corpora and lexicons) and supporting tools may require paying for a license to the resource or for membership to the catalog. the fees support the creation, licensing, storage, and maintenance of new datasets and language research initiatives. the ldc, for example, currently provides access to 1016 datasets. @xcite open source dataset catalogs have also been constructed as supporting technical infrastructure in the context of nlp and ml libraries. the natural language toolkit (nltk), developed since 2001, is a python package with utilities for nlp tasks that includes access to widely used corpora such as the brown corpus (kučera and francis, 1967) as well as features for adding datasets and using datasets locally (bird et al., 2009). the hugging face datasets library (lhoest et al., 2021) and tensorflow library (tensor flow authors, 2021) both provide tools for loading datasets from remote and local repositories and include catalogs of directly accessible datasets. sadilar provides its own catalog of annotated language datasets and processing tools, with links for downloading resources that are licensed for distribution. other catalogs of nlp datasets do not provide access to the datasets themselves, but provide information about uses and categories. for example, papers with code links academic publications that use the same dataset with information about the dataset. @xcite masader similarly provides metadata about arabic-language nlp datasets without hosting the data (alyafeai et al., 2022). our work is an effort to merge the careful and wellestablished data collection and documentation practices from organizations such as the ldc with the collaborative, open source tools for dataset construction. while large-scale nlp research requires vasts amounts of data, the work that goes into curating, documenting, and maintaining the data is often undervalued (sambasivan et al., 2021), resulting in data collections that are often too large to document post-hoc (bender et al., 2021) and contain significant quantities of unwanted media (luccioni and viviano, 2021). we provide an alternate approach to data collection and management in nlp; this approach prioritises documentation in the data creation process, engages communities to inform data curation, and contributes to a more representative dataset. ## the catalogue the primary goal of the catalogue (see appendix a for screenshots of the form) was to support the creation of a training dataset for language modeling that integrated the european language resource association (elra), the chinese ldc, the ldc for indian languages (ldcil), and clarin. we created an openly accessible form in english for submitting metadata for potential sources for the identified language groups. @xcite we used an iterative approach to collectively develop questions that elicit the metadata, descriptions of the information being requested, and answer prompts to support efficient documenting. wherever possible, we formatted the questions as multiple choice questions with an optional free-form field, should the pre-existing options be insufficient. after building the online form, we tested the form with actual examples, i.e., the le monde newspaper and its publishing company group le monde to ensure its validity. ## additional features there are two other modes for interacting with the catalogue: a validation mode, for validating submitted entries, and a visualization mode, for filtering and mapping specific submitted entries. because we intended to make the catalogue openly available on the web past the end date of the workshop, we included the validation functionality to allow users to confirm that metadata for submitted entries was correct and could be updated if ever the information was no longer correct (e.g., if a license for an entry changed). the purpose of the visualization mode was to support later users of the catalogue in seeing the general distribution of submitted resources of the catalogue across languages and geographic regions and searching for specific resources within those categories. to validate an entry, the validator can confirm the previously submitted metadata or edit and resubmit the entry. the catalogue then saves both the original and the validated submission. the visualizations include a pie chart detailing the proportion of entries by language and an interactive map which shows the number of submitted entries for a region or country as defined by the location of the data creators or data custodians. in figure 1 , the color gradient indicates the number of entries by country and location markers indicate regions that can be examined for more details. both the map and a pie chart can be filtered using one of the many properties produced by the form, e.g., the resource, license, or media type. entries returned by the filter can be selected to display their descriptions. ## community hackathons with the catalogue submission form developed, we could begin to collect and document potential data sources for review prior to developing the full dataset towards the workshop's llm goal. whereas prior data collection processes utilized automatic methods for collecting as much data as possible, we wanted our collection process to prioritize sources that were created by language communities and that were determined by language communities to be representative of their language use. in order to center the metadata collection for as many languages as possible around communities who speak those languages, we decided to crowdsource our metadata collection by organizing community hackathons. @xcite to do so, we reached out to regional community organizations focused on ml and nlp to collaborate in leading local hackathons and put out a similar call within the workshop for individuals who spoke one or more of the listed languages. the task for each hackathon was for participants to use the catalogue submission form to submit as much metadata as they could find on potential data sources for their language or languages. we developed a guide foot_7 with instructions and suggestions for the hackathon participants for each section of the catalogue submission form. a coalition member and/or a collaborating organizer from a partner organization was available to interact with participants and answer questions arising while filling the form and to discuss details about potential resources or institutions. in total, we organized 6 hackathons for specific communities and regions of the world based on the availability of organizers and their familiarity with the communities, namely african languages in collaboration with masakhane, @xcite asian languages with machine learning tokyo, @xcite basque, english in the americas and europe, english in the indo-pacific region, and spanish in latin america with latinx in ai. @xcite the hackathons took place online in october-december, 2021, lasting one to six hours. we announced hackathons using social media, in coordination with the relevant partner organizations. because we advertised primarily to members of the workshop, social media followers of the workshop, and members of the partner organizations, the hackathons attracted participants who were generally interested in language modeling and specifically wanted to support the workshop goals of having greater language representation in the to-be-trained workshop language model. no further incentives were used to encourage participation. during the hackathons we only collected a name and e-mail. after the hackathons, we sent a 10-question survey to all participants to collect further information. ## discussion the result of our efforts is an openly available catalogue of 192 data sources, with each of our target natural language groups constituting at least 10% of the total submitted entries. @xcite the majority of these resources are primary and processed resources, with data custodian primarily located in the americas, europe and australia. the sources recorded in the catalogue were used as a core component of the training dataset for training the llm developed by the coalition. in addition, the development of the catalogue serves as an opportunity for methodological reflection on documentation-first and human-centered data collection in nlp. in this section we discuss lessons learned from creating and crowdsourcing the catalogue and present recommendations for future data collection efforts. ## ethical considerations beyond the limitations outlined in §7, future users of the catalogue and the data it references should be aware of a number of ethical considerations relevant to it. whilst the catalogue is open, the data it registers have their own licenses and usage restrictions that users must abide by (e.g. licenses that preclude commercial uses of data). for instance, appropriately handling personally identifiable information (pii) must be included in plans for the catalogue, with attention to the detection and implications of different types of pii. in the following sections we reflect on these topics, based on lessons learned during the catalogue development. ## conclusion we have presented our design processes, our humancentered metadata collection efforts, and our resulting successes and challenges in creating a data catalogue targeting 13 language groups. next steps for the catalogue include translating the form into more languages, filling in missing information for existing entries, and adding more entries to continue efforts toward greater representation across languages and regions. we also plan to update the interactive aspects of the catalogue with more advanced features, i.e., the survey respondents' recommendations and automated screening of new submissions to avoid duplication of entries. the resources within the catalogue (both collected during the hackathons and submitted later) have contributed to the development of the bigscience roots corpus and the subsequent training of the bloom open-access multilingual language model (laurençon et al., 2022; scao et al., 2023). this work produced the data catalogue form, the submission website, and the human-centered methodology of data collection in collaboration with language communities for representative language modeling and other nlp tasks. the catalogue tool remains openly available for use in collecting metadata towards new dataset development projects and for searching existing entries for specific languages and regions. the catalogue form is available for adaption and translation for future documentation and metadata collection efforts to build on. we also discuss a number of challenges, ethical considerations, and recommendations for representative data collection efforts to continue to engage with, particularly in relation to licensing and personally identifiable information. we expect that the form may need to be updated as documentation requirements for nlp and ml systems become regulated and official documentation standards are developed. scaling the hackathon collection methodology to support larger data collection efforts as well as smaller language communities will require further research and collaboration efforts. despite these challenges, we hope to encourage others to follow conscientious documentation practices prior to releasing data collections, especially for large-scale nlp applications.