Toward More Meaningful Resources for Lower-resourced Languages (2022)

Abstract: In this position paper, we describe our perspective on how meaningful resources for lower-resourced languages should be developed in connection with the speakers of those languages. Before advancing that position, we first examine two massively multilingual resources used in language technology development, identifying shortcomings that limit their usefulness. We explore the contents of the names stored in Wikidata for a few lower-resourced languages and find that many of them are not in fact in the languages they claim to be, requiring non-trivial effort to correct. We discuss quality issues present in WikiAnn and evaluate whether it is a useful supplement to hand-annotated data. We then discuss the importance of creating annotations for lower-resourced languages in a thoughtful and ethical way that includes the language speakers as part of the development process. We conclude with recommended guidelines for resource development.

## introduction recent years have seen increased interest from the natural language processing community in developing both models and datasets for what may be termed "lower-resourced" languages. advances in transfer learning and the increased availability of data and benchmarks in these languages have made it straightforward to create what appear to be high-performing models for these languages with little or no annotated data. yet despite the popularity and apparent effectiveness of these systems, the unique challenges and best practices for developing datasets and models for lower-resourced languages are rarely discussed alongside the system themselves. in this position paper, through commentary on our experiences working with existing datasets and a discussion of current trends, we explore what resources * denotes equal contribution. before we can elaborate further, we must address a terminological issue which creates a stumbling block when attempting to discuss resources and language technology for the languages of interest to us. throughout this paper, we will use the term lowerresourced language to refer to languages that have received fewer resources-as measured in any numbers of dimensions such as models, datasets, papers, funding, etc.-than the most popularly-studied languages in the field of natural language processing. we explicitly use the comparative lower rather than low to emphasize the continuum that exists across languages regarding the resources available for developing language technology. whether a language is lower-resourced in a specific context may depend on what is available for the task at hand. due to singular efforts, a language that may have otherwise been underserved by the research community may have a rich set of resources for a single task like machine translation (mt), but might not have annotation for other tasks. for example, consider the inuktitut language, which has a substantial amount of governmentdomain parallel data @xcite that enabled an mt shared task @xcite , but has few labeled datasets for other tasks and is not included in any large multilingual language models we are aware of. we take an open and intersectional perspective to what might be called a lower-resourced language, acknowledging that this designation is both imperfect and often the result of many contributing factors. for example, many languages referred to in this way may be less-widely spoken, underserved by the research community and funding agencies, or used by marginalized or minoritized populations. in summary, our use of the term lower-resourced is intended to reflect a continuous, not categorical, status and one that is multi-dimensional, depending on task and context. the goal of this paper is to argue for our perspective regarding what are the most effective ways to construct and use resources for building language technology for lower-resourced languages. this paper's contributions come at two levels. at the more concrete level, we discuss particular issues related to using wikidata and wikiann as sources of information about names, and highlight how automatic processes to take advantage of this information can go wrong when no human expertise is involved. at a higher level, we discuss the problematic nature of developing language technology datasets and models with no or limited interaction with the population of speakers of the languages involved. the structure of the paper is as follows. we first review two popular resources used in nlp for lower-resourced languages and the impact that they have had on the field. while we will discuss shortcomings of these resources, sometimes demonstrating them with experiments, the goal of this section is not to publish a critique of these resources, but rather to make other researchers aware of their shortcomings and limitations. by acknowledging and understanding their limitations, we can better understand how to use them most appropriately and develop future resources that do not share the same limitations. we then turn to the importance of annotation and dataset creation processes that meaningfully involve speakers of the languages under study. we discuss open challenges for nlp for lowerresourced languages, and conclude with suggested guidelines for researchers performing research in this area. ## wikidata: a source for name labels wikidata foot_0 is an open and collaboratively edited knowledge graph, hosted by the wikimedia foundation. the wikidata graph consists of entity nodes connected by labeled edges that represent binary relations. each entity and relation is identified by a unique wikidata identifier, e.g. q4346375 (association for computational linguistics) and p361 (part-of relation). while english labels are typically used for the page titles on the wikidata website, most entities have labels available in several languages, with the most well-edited entries having labels in hundreds of languages. this makes wikidata an appealing source for constructing multilingual nlp resources related to entity names, as parallel names can in theory be trivially extracted from each entity. for example, wikidata could be used to harvest name lists for a named entity recognition (ner) system, or as a source of parallel names for translation or transliteration models. in this section, we show wikidata's promise for extracting names in lowerresourced languages as well as the data issues that arise in attempting to use it for this purpose. ## wikiann we now turn our attention to a different wikipediarelated resource, one derived from it. wikiann @xcite is a dataset originally created for named entity tagging and linking of 282 languages present in wikipedia. pan et al. generate "silverstandard" named entity annotations "by transferring annotations from english to other languages through cross-lingual links and kb properties, refining annotations through self-training and topic selection, deriving language-specific morphology features from anchor links, and mining word translation pairs from cross-lingual links." we question the appropriateness of treating wikiann as a multilingual benchmark for ner. even in a higher resourced language, the practice of evaluating a task on automatically derived data is sub-standard, hence the original authors referring to it as a "silver standard." this kind of evaluation can only show how close one model comes to replicating the automated data collection process and does not reflect human performance. just examining the english data of wikiann reveals a number of entity names that would otherwise never be marked as names by a human. strings of text such as independently released, if i were a boy, and list of books written by teenagers are annotated as organizations. i was glad, the latter's studio, and were promoted are annotated as locations, and range has expanded, a twelve-year-old passenger was found alive, and artavasdes ii, who served as are tagged as person. while these examples are a small sample, we identified hundreds like these with either span issues, wikipedia-specific entities like lists, incorrect entity types or entities that were simply not names. one might argue that this represents less than 1% of the english wikiann data and is therefore noise. however, human annotation does not typically have mistakes of thes types and also has the benefit of being able to report inter-annotator agreement so that researchers can better understand the difficulty of the task. ## human annotation: still essential we have spent much of our paper describing the complexities of working with large-scale, wikiderived datasets, demonstrating that while they have some utility, they must be used with caution. this caution stems from the fact that though the data contained in them originated from human contributions, in its final form in a resource, the data has been removed from its original quality checks. for example, in wikidata, names may be copied from one language to another en masse, and in wikiann, the ner "annotation" is system output trained on relatively distant human supervision. we have spent so much of our paper describing these shortcomings to demonstrate that there is no "free lunch" when it comes to avoiding human annotation or quality checking of datasets. we believe that human annotation processes that are ultimately participatory-involving speakers of the languages as stakeholders and collaborators, not mere annotators for hire-like that of the masakhaner project which we have featured through the previous section and related projects @xcite are the most important direction for developing language technology for lower-resourced languages. a discussion of efforts to annotate lowerresourced languages would not be complete without a discussion of the resources developed as part of the darpa foot_11 lorelei (low resource languages for emergent incidents) program. the lorelei program began in fall 2015, and a major thrust of the program was producing annotation for many lower-resourced languages @xcite . the linguistic data consortium (ldc) is in the process of releasing the 31 language packs developed as a part of the program, which have been available to the primarily u.s.-based groups funded by the program for years. tracey and strassel (2020) stated that they planned to release 1-2 packs per month in 2020. however, as of november 2021, even though the research efforts of the program have largely concluded, only 7 language packs have been released to the general public: akan, amharic, oromo, somali, tigrinya, ukranian, and vietnamese. each of these language packs is available for $200 usd to non-members of the ldc. while that is less expensive than typical ldc datasets, that cost can be prohibitively expensive for the speakers of the languages in the packs, who may live in countries with substantially lower wages and may not have the backing of a well-funded research lab. while it is unfortunate that so little data has been released to date and that the data is not freely available, the main contrast we would like to draw between lorelei and efforts such as masakhaner is the involvement of the speakers of lower-resourced languages. speakers of the languages included in the lorelei datasets did not have any significant involvement in the construction of the datasets beyond their role as annotators. this is in no way unique to the lorelei program; it is the status quo for annotation projects. returning to the issues we raise in our introduction, we want to highlight that a confluence of factors come together to make a language lowerresourced, among them often a marginalization and/or minoritization of its speakers. we should consider whether it is ethical to have a paradigm in which the marginalized have no say in research that involves them, and in this case may not even be able to access the result of their work years after it is performed. developing resources for languages that have had fewer resources created for them to date poses a unique set of ethical challenges that differs from higher-resourced language work, and engaging language speaker in a participatory fashion can help mitigate the risk of harm. ## challenges before we conclude, we wish to highlight challenges that we believe should be addressed as a part of continuing to develop resources and models for lower-resourced languages. ## conclusion in closing, we want to refer to the state of affairs highlighted by @xcite ; some researchers are preoccupied with a data-centric view to the point of completely removing the need to involve speakers of the language in any part of the process. through our discussion of the shortcomings of wikidata and wikiann for the specific purposes that we have evaluated them, we demonstrated the gaps that are created when the dataset creation process is divorced from the speakers of the language. our perspective on the process of dataset and model creation can be summarized through these guidelines we propose for future work on lower-resourced languages: