Participatory Research for Low-resourced Machine Translation: A Case Study in A frican Languages (2020)

Abstract: Research in NLP lacks geographic diversity, and the question of how NLP can be scaled to low-resourced languages has not yet been adequately solved. ‘Low-resourced’-ness is a complex problem going beyond data availability and reflects systemic problems in society. In this paper, we focus on the task of Machine Translation (MT), that plays a crucial role for information accessibility and communication worldwide. Despite immense improvements in MT over the past decade, MT is centered around a few high-resourced languages. As MT researchers cannot solve the problem of low-resourcedness alone, we propose participatory research as a means to involve all necessary agents required in the MT development process. We demonstrate the feasibility and scalability of participatory research with a case study on MT for African languages. Its implementation leads to a collection of novel translation datasets, MT benchmarks for over 30 languages, with human evaluations for a third of them, and enables participants without formal training to make a unique scientific contribution. Benchmarks, models, data, code, and evaluation results are released at https://github.com/masakhane-io/masakhane-mt .

## introduction language prevalence in societies is directly bound to the people and places that speak this language. consequently, resource-scarce languages in an nlp context reflect the resource scarcity in the society from which the speakers originate @xcite . through the lens of a machine learning researcher, "lowresourced" identifies languages for which few digital or computational data resources exist, often classified in comparison to another language @xcite . however, to the sociolinguist, "low-resourced" can be broken down into many categories: low density, less commonly taught, or endangered, each carrying slightly different meanings @xcite . in this complex definition, the "low-resourced"-ness of a language is a symptom of a range of societal problems, e.g. authors oppressed by colonial governments have been imprisoned for writing novels in their languages impacting the publications in those languages @xcite , or that fewer phd candidates come from oppressed societies due to low access to tertiary education @xcite . this results in fewer linguistic resources and researchers from those regions to work on nlp for their language. therefore, the problem of "lowresourced"-ness relates not only to the available resources for a language, but also to the lack of geographic and language diversity of nlp researchers themselves. the nlp community has awakened to the fact that it has a diversity crisis in terms of limited geographies and languages @xcite : research groups are extending nlp research to low-resourced languages @xcite @xcite , and workshops have been established @xcite @xcite . we scope the rest of this study to machine translation (mt) using parallel corpora only, and refer the reader to @xcite for an assessment of low-resourced nlp in general. ## the machine translation process we reflect on the process enabling a sustainable process for mt research on parallel corpora in terms of the required agents and interactions, visualized in figure 1 . content creators, translators, and curators form the dataset creation process, while the language technologists and evaluators are part of the model creation process. stakeholders (not displayed) create demand for both processes. stakeholders are people impacted by the artifacts generated by each agent in the mt process, and can typically speak and read the source or the target languages. to benefit from mt systems, the stakeholders need access to technology and electricity. content creators produce content in a language, where content is any digital or nondigital representation of language. for digi-tal content, content creators require keyboards, and access to technology. translators translate the original content, including crowd-workers, researchers, or translation professionals. they must understand the language of the content creator and the target language. a translator needs content to translate, provided by content creators. for digital content, the translator requires keyboards and technology access. curators are defined as individuals involved in the content selection for a dataset (bender and friedman, 2018), requiring access to content and translations. they should understand the languages in question for quality control and encoding information. language technologists are defined as individuals using datasets and computational linguistic techniques to produce mt models between language pairs. language technologists require language preprocessors, mt toolkits, and access to compute resources. evaluators are individuals who measure and analyse the performance of a mt model, and therefore need knowledge of both source and target languages. to report on the performance on models, evaluators require quality metrics, as well as evaluation datasets. evaluators provide feedback to the language technologists for improvement. ## case study: masakhane africa currently has 2144 living languages @xcite . despite this, african languages account for a small fraction of available language resources, and nlp research rarely considers african languages. in the taxonomy of @xcite , african languages are assigned categories ranging from "the left behinds" to "the rising stars", with most languages not having any annotated data. even monolingual resources are sparse, as shown in in addition to a lack of nlp datasets, the african continent lacks nlp researchers. in 2018, only five out of the 2695 affiliations of the five major nlp conferences were from african institutions @xcite . ∀ et al. (2020) attribute this to a culmination of circumstances, in particular their societal embedding @xcite and socio-economic factors, hindering participation in research activities and events, leaving researchers disconnected and distributed across the continent. consequently, existing data resources are harder to discover, especially since these are often published in closed journals or are not digitized @xcite . for african languages, the implementation of a standard crowd-sourcing pipeline as for example used for collecting task annotations for english, is at the current stage infeasible, due to the challenges outlined in section 3 and above. additionally, no standard mt evaluation set for all of the languages in focus exists, nor are there prior published systems that we could compare all models against for a more insightful human evaluation. we therefore resort to intrinsic evaluation, and rely on this work becoming the first benchmark for future evaluations. we invite the reader to adopt a metaperspective of this case study as an empirical experiment: where the hypothesis is that participatory research can facilitate low-resourced mt development; the experimental methodology is the strategies and tools employed to bring together distributed participants, enabling each language speaker to train, contribute, and evaluate their models. the experiment is evaluated in terms of the quantity and diversity of participants and languages, and the variety of research artifacts, in terms of benchmarks, human evaluations, publications, and the overall health of the community. while a set of novel human evaluation results are presented, they serve as demonstration of the value of a participatory approach, rather than the empirical focus of the paper. ## conclusion we proposed a participatory approach as a solution to sustainably scaling nlp research to low-resourced languages. having identified key agents and interactions in the mt development process, we implement a participatory approach to build a community for african mt. in the process, we discovered successful strategies for distributed growth and communication, knowledge sharing and model building. in addition to publishing benchmarks and datasets for previously understudied languages, we show how the participatory design of the community enables us to conduct a human evaluation study of model outputs, which has been one of the limitations of previous approaches to low-resourced nlp. the sheer volume and diversity of participants, languages and outcomes, and that for many for languages featured, this paper constitutes the first time that human evaluation of an mt system has been performed, is evidence of the value of participatory approaches for low-resourced mt. for future work, we will (1) continue to iterate, analyze and widen our benchmarks and evaluations, (2) build richer and more meaningful datasets that reflect priorities of the stakeholders, (3) expand the focus of the existing community for african languages to other nlp tasks, and (4) help implement similar communities for other geographic regions with lowresourced languages. secondary school undergraduate masters phd online courses (a) highest level of education. student researcher data scientist academic other software engineer linguist (b) occupation.