Beyond Counting Datasets: A Survey of Multilingual Dataset Construction and Necessary Resources (2022)

Abstract: While the NLP community is generally aware of resource disparities among languages, we lack research that quantifies the extent and types of such disparity. Prior surveys estimating the availability of resources based on the number of datasets can be misleading as dataset quality varies: many datasets are automatically induced or translated from English data. To provide a more comprehensive picture of language resources, we examine the characteristics of 156 publicly available NLP datasets. We manually annotate how they are created, including input text and label sources and tools used to build them, and what they study, tasks they address and motivations for their creation. After quantifying the qualitative NLP resource gap across languages, we discuss how to improve data collection in low-resource languages. We survey language-proficient NLP researchers and crowd workers per language, finding that their estimated availability correlates with dataset availability. Through crowdsourcing experiments, we identify strategies for collecting high-quality multilingual data on the Mechanical Turk platform. We conclude by making macro and micro-level suggestions to the NLP community and individual researchers for future multilingual data development.

## introduction datasets play fundamental roles in advancing language technologies @xcite . however, large disparities exist among languages in terms of the scale of existing datasets @xcite and the resulting task performance @xcite . multilingual resources also harbor unique annotation artifacts, such as translationese @xcite . understanding dataset construction processes can help explain the true landscape of multilingual nlp datasets. despite low-level awareness among multilingual researchers about disparities in linguistic resources, scant research to date has examined the quality of labeled data in non-english languages. existing surveys such as @xcite have studied the scale of available resources per language, without exploring how such datasets are developed, and often simply count number of datasets. we provide the first large-scale survey of multilingual dataset characteristics, examining in depth how and why these datasets are constructed and quantifying their availability. we study 156 datasets, each covering at least one non-english language, from the acl anthology and hugging-face @xcite . we propose a new annotation scheme with 13 attributes, including the task it addresses, its source of input text, its label collection process, its stated motivation, and tools used to create it, such as translation services (section 3). our annotation yields rich information about the status of multilingual datasets (section 4). as shown in figure 1 , language coverage varies significantly across tasks, and label collection methods differ across languages and tasks. 222 languages are covered by the 156 datasets, and, on average, 5.6 datasets per language are available. however, we posit that this number is misleading when interpreting the landscape of multilingual nlp datasets: we observe the prevalence of automatically induced labels particularly in low resource languages; onethird of the datasets we surveyed use automatically induced labels, and 68% of the languages have no manually annotated data. while wikipedia and news texts are available for a wide range of languages, texts specifically written for the nlp task are not available for most languages. furthermore, about 20% of surveyed datasets involved translation in their creation, and these were used at a much higher rate in highly cited resources to study crosslingual transfer. these types of quantitative and qualitative disparities persist in multilingual supervised datasets, motivating us to identify underlying bottlenecks to create multilingual datasets. we investigate whether the resources required for nlp data collection affect the prevalence of multilingual datasets. these include the availability of crowdworkers and nlp researchers with sufficient language proficiency as well as raw input text (section 5). these data collection resources all correlate with the number of datasets in each language, suggesting paths for a more equitable data landscape. to assess the paths for crowdsourcing in non-english languages on a popular crowdsourcing platform, we design controlled data collection experiments in six languages, quantifying the challenges even in relatively high-resource languages. we conclude our survey by relating researchderived suggestions to the nlp community and to individual researchers for multilingual data construction. we also provide concrete suggestions for crowdsourcing platforms and crowdsourcing quality control tips, recommended translation services, and publication venues. we host our survey at https: //multilingual-dataset-survey.github.io , permitting readers to review multilingual resources and submit their datasets following our schema. ## survey scope, scheme and process in the following section, we first identify the 156 open-sourced nlp datasets published in nlp venues that contain at least one non-english language. we then describe our proposed annotation scheme with 13 attributes, seven of which are analyzed for the first time ( scope. the scope of our work includes labeled datasets, where a system is expected to generate a label (output) y given input text x. the output label is not limited to a single categorical label and includes generated text. we filter out (1) unlabeled data, including parallel corpora (studied in @xcite ; (2) language identification datasets; we compiled dataset lists in december 2021 sourced from: (1) the acl paper anthology identified through a keyword search, and (2) the opensource hugging face datasets after filtering ex- aspect descriptions categories language ♦ target languages {iso 639-1 language code, not mentioned} task type ♦ ten coarse-grained nlp task type that the dataset addresses {classification (sentiment analyis), classification (sentence pair), classification (other), qa (w/ retrieval), qa (machine reading), structured prediction, sequence tagging, generation (summarization), generation (other), other} dataset size avg. # of data in each language { < 100, 100 ∼ 1000, 1000 ∼ 10k, > 10k } creator ♦ who led dataset creation {industry, individual researchers, university} pub. venue venue of paper publication {*cl, lrec, *acl workshop, findings, neurips datasets and benchmarks track, arxiv, n/a} pub. year year of paper publication year of publication between 2008 -2021 motivation motivation for dataset creation {cross-lingual transfer, single task (multilingual) w/ ml training, single task (single lang), multi-task (single lang)} input text (x) source ♦ where input text is from {annotated (authors, linguists), commercial sources, crowdsourced, curated linguistic resources (wordnet, etc.), curated source (exams, scientific papers, etc.), media, template-based, web, wikipedia} x language languages where x is collected {english, its own language, both, other language, not mentioned} label (y) collection ♦ how label is collected {annotated (authors, linguists), automatically induced, crowdsourced, curated linguistic resources, not mentioned} y language languages where y is collected {english, its own language, not mentioned} reuse reusing released datasets? {yes-eng, yes-other-lang, yes-eng&other-lang, no} translation ♦ type of translation used during data collection {automatic translation, human (author), human (non-author), no translation, unclear} we selected these keywords to focus our study on labeled multilingual nlp text-only datasets. the preceding steps helped us select 151 papers of the 73,384 papers in the acl anthology, and we subsequently filtered out papers due to unavailability of datasets. for multi-task datasets for a specific language (such as klue; @xcite , we decomposed the unified benchmark into each subtask and included each sub-task as a unique dataset. for example, in the klue benchmark, there are eight tasks in total, and we annotate each sub-task dataset independently. standardized multi-task test suites that directly reuse existing resources such as xtreme @xcite , or dataset / sub-task dataset that directly reuses a previously published dataset with no significant modification to any of the schemes were not included, and we only include the original dataset. our final survey includes 156 datasets from 112 papers. ## survey results we summarize our findings, focusing on novel aspects. for each category of each aspect, we report the number of datasets belonging to that category and the unique languages they cover. we also highlight noteworthy correlations between multiple aspects, e.g., how label collection methods correlated with the type of tasks addressed. ## pilot study on crowdsourcing multilingual data on mturk how easy is it to collect multilingual dataset on popular annotation platform (mturk) at the moment? crowdsourcing enables large-scale, cost efficient data collection; however, for many languages, the number of language-proficient crowdworkers is limited @xcite . we quantify the availability of mturk workers with proficiency in non-english languages. we formulated a four-way sentiment analysis task using the multilingual amazon review corpus @xcite ) and analyzed the annotation quality, cost, and time to finish tasks in english, spanish, german, french, japanese and chinese of crowdworkers. in all settings, we asked annotators to translate the same english sentence to assess their actual (rather than professed) language proficiency. we found that many production-level mt systems fail to translate his sentence due to its compositionality. further, we investigated the newly introduced "language qualification" in mturk, which available for only four aforementioned languages and brazilian * https://prolific.co * https://www.mturk.com/ portuguese as of 2022. for our sentiment analysis task, without the language qualification, the accuracy of human binary classification performance in all non-english languages (55.2%) was significantly worse than that of english (77%). past recommendations, such as constraining location and hit acceptance rate, are insufficient (as of 2022) to collect high quality data even for languages considered "easier" to crowdsource in prior work @xcite . language qualification improved performance by up to 40% and reduced the prevalence of cheating across all languages. however, with the language qualification, the data collection process usually took more time and cost ($1 per assignment). more pilot study details are in appendix c. ## discussion and suggestions this work provides the first large-scale meta survey on public multilingual nlp datasets, focusing on the novel aspects under-explored in prior work. we found that many languages lack a diverse set of manually annotated datasets and coverage of tasks and input&label sources. particularly, except for summarization, generation tasks for non-english languages show limited language coverage. we conclude this work by presenting concrete suggestions to both the nlp community (section 7.1) and to individual researchers aspiring to create new multilingual nlp datasets (section 7.2). ## conclusion we present the first large-scale comprehensive survey on characteristics of multilingual datasets, exposing that the disparity among languages is not only quantitative (i.e., the number of the datasets) but also qualitative (e.g., how and why those datasets are created). we also discuss building blocks for constructing data resources, from language proficient researchers to crowdworkers. our mturk experiments show the challenges of quality and costs of annotating multilingual datasets on mturk and suggest several approaches to tackle those challenges. we conclude our survey with a list of concrete suggestions for researchers interested in constructing language resources. limitations on dataset documentation. throughout our survey process, we found inadequate dataset documentation, limiting the coverage of our survey. we suggest that individual researchers provide the input data source and the labeling methodology; if people were involved in dataset creation, their demographic information should be provided, as well. such information can help researchers analyze potential bias embedded in the dataset @xcite . surveyed dataset collection process. despite our best efforts, we do not claim to cover all relevant datasets. our collection process overlooks datasets that are published at non-acl venues and not in hugging face as well as papers that do not match our search keywords. for instance, we missed multilingual machine reading comprehension datasets @xcite and morphology datasets @xcite . we also found a very low presence of indigenous language datasets. none of 10 indigenous american languages from a recent study @xcite was represented in our survey. that said, we host http: //multilingual-dataset-survey.github.io where researchers can submit their dataset information and periodically update our analysis. furthermore, we constantly encountered poorly written documentation or unavailable datasets during our annotation processes. during annotation, whenever this paper's dataset annotators encountered unclear documentation, they made their best guess to put datasets into predefined categories. if no evidence could be found for the inference, they put "not mentioned" as a result. all unclear decision were adjudicated by at least three annotators. using country names as a proxy for languages spoken. in section 5, we attempted to approximate the number of nlp researchers with language proficiency in different languages. to do this, we mapped the names of acl submission country to the most commonly spoken languages in those countries. we acknowledge that (1) the country of origin of researchers might be different from the country of submission, (2) researchers native language might not be listed in the commonly spoken languages and (3) the mapping might be incomprehensive. classification accuracy by merging 1, 2-star ratings as negative and 4, 5-star ratings as positive. translation. for non-english tasks, as an additional check (for worker's understanding ability and cheating detection), we require crowdworkers to translate a simple yet compositional english sentence from an wikipedia article, * "washington, d.c., formally the district of columbia, also known as just washington or just d.c., is the capital city of the united states, but it is not located in the washington state." some widely-used online translation tools give sub-par translations on this sentence because of its compositional structure. we forbid workers to use online translation platforms in the task prompt. we collect the gold translation from native speakers as references. in addition, we ask them to rate the collected translations as correct (3), partially correct (2), or incorrect (1). * we also compute the bleu score against the reference answer. we present some sample translation results as well as the gold translation provided by human native speaker annotators in