Machine Translation Between High-resource Languages in a Language Documentation Setting (2022)

Abstract: Language documentation encompasses translation, typically into the dominant high-resource language in the region where the target language is spoken. To make data accessible to a broader audience, additional translation into other high-resource languages might be needed. Working within a project documenting Kotiria, we explore the extent to which state-of-the-art machine translation (MT) systems can support this second translation – in our case from Portuguese to English. This translation task is challenging for multiple reasons: (1) the data is out-of-domain with respect to the MT system’s training data, (2) much of the data is conversational, (3) existing translations include non-standard and uncommon expressions, often reflecting properties of the documented language, and (4) the data includes borrowings from other regional languages. Despite these challenges, existing MT systems perform at a usable level, though there is still room for improvement. We then conduct a qualitative analysis and suggest ways to improve MT between high-resource languages in a language documentation setting.

## introduction we report on our investigations of whether and how existing machine translation (mt) systems can support the work of documenting and describing endangered languages. rather than targeting lowresource mt, we look at translating between highresource languages, aiming to save time for the language experts and language community members working on the language documentation project. specifically, we are working with a linguist documenting kotiria (also known as wanano), an east tukano language spoken in the brazil-colombia borderlands in northwestern amazonia. documentation and description of kotiria on the brazilian side of the border has been ongoing since 2000, resulting in numerous publications, including a reference grammar @xcite , and a documentary archive of primarily monologic language data (approx. 10 hours of mythical, historical, and personal narratives, public addresses, and instructional speech). a second documentation project focusing on language use and interaction in daily life resulted in a much larger corpus -approximately 60 hours -of primarily conversational data. both projects were carried out within the participatory research paradigm @xcite , with indigenous speakers involved in both recording and annotation of data in elan, 1 including translation of the indigenous language data into portuguese. further grammatical analysis and annotation of these documentary materials, including translation from portuguese into english, is ongoing but proceeds slowly. researchers of endangered languages worldwide generally work alone or at best in small teams to deal with enormous amounts of data, further underscoring the gap between technological advances that facilitate production of large, high quality documentary corpora and researchers' ability to single-handedly process the resulting materials. the kotiria case is no different, and even basic tasks, such as adding english translations to the two existing corpora, extend over years. the corpus from the more recent kotiria language documentation project presents additional challenges. first, language use in conversation is by its very nature more complex to annotate and analyze than monologic speech because it is rife with features such as reductions, cut-offs, overlaps, intonational contours, and other details of production, as well as grammatical structures whose meaning can only be understood in sequential context @xcite . additionally, due to the multilingual nature of social life in the region where kotiria is spoken @xcite , recordings contain numerous instances of speech in other indigenous languages, such as tukano. though extremely rich, such data constitutes a lifetime (or perhaps several lifetimes) of processing work for a lone-wolf researcher. automatic translation -or machine translation (mt) -has made tremendous progress over the last few years @xcite @xcite , and mt systems are used more and more in everyday life, e.g., in browser extensions, smartphone apps, or as a first translation pass in software for professional (human) translators. initial translations in a language documentation project are often made into the dominant high-resource language in the documented language's region (portuguese for kotiria). as mt between high-resource languages is typically of high quality @xcite , we investigate if mt systems can assist with producing additional translations between the region's dominant high-resource language (portuguese) and english, which can help make the created resources accessible to a broader community. our goal is to produce first-pass translations automatically, such that the language experts in the language documentation project need not devote years to the process, but rather can do post-correction of the first-pass translations. this should yield significant time savings @xcite , freeing up the experts to work on other aspects of the project. importantly, such a translation in a documentation context constitutes multiple challenges not present in general mt: (1) the sentences that need to be translated are out-of-domain with respect to the system's training data, (2) the data is conversational, (3) the source-side data contains nonstandard and uncommon expressions, often reflecting properties of the documented language, and (4) the text includes borrowings from other regional languages. while those challenges could be minimized by training on in-domain data from the concrete translation task, such data is generally either not available or too small for effective finetuning. first, we employ 3 state-of-the-art mt systems to translate portuguese sentences for which we have gold-standard translations into english. we evaluate the results both manually and with automatic metrics and find that google translate performs best. second, we analyse the outputs of google translate, exploring what types of examples it fails and succeeds on. we observe that the conversational nature of the kotiria data and particular properties of kotiria-to-portuguese translations cause many errors. we end by discussing how to improve mt for language documentation data. mt of out-of-domain data our setting requires mt models to generalize to out-of-domain data: available translations are too few for training or finetuning, and, in other language documentation settings, no translations into additional highresource languages might be available at all. however, mt systems often struggle to perform well on data they have not been trained on -e.g., systems trained on 2019 news do not perform well on 2020 news, due to a topic shift towards the coronavirus @xcite . domain adaptation (da), which has been studied extensively @xcite @xcite , though not in the context of a language documentation workflow, can yield improvements. techniques include finetuning @xcite or backtranslation @xcite . for surveys on da for mt, we refer readers to @xcite ; @xcite . we investigate how well general state-of-the-art mt systems translate between highresource languages in a language documentation setting. in future work, we will take inspiration from research on da and investigate how to build better systems for our use case. ## conclusion using data from the documentation of kotiria, we investigated how general state-of-the-art mt systems perform when translating from portuguese to english in a language documentation setting. we found that, among 3 systems, google translate performs best and at a level that makes it a promising option for documentary linguists. we then performed a qualitative analysis of google translate and observed a number of systematic error patterns directly linked to properties of our language documentation project. finally, we suggested multiple ways to improve systems for this setting, including model adaptation, targeted multilinguality, and the incorporation of linguistic features.