Train Global, Tailor Local: Minimalist Multilingual Translation into Endangered Languages (2023)

Abstract: In many humanitarian scenarios, translation into severely low resource languages often does not require a universal translation engine, but a dedicated text-specific translation engine. For example, healthcare records, hygienic procedures, government communication, emergency procedures and religious texts are all limited texts. While generic translation engines for all languages do not exist, translation of multilingually known limited texts into new, endangered languages may be possible and reduce human translation effort. We attempt to leverage translation resources from rich resource languages to efficiently produce best possible translation quality for well known texts, which is available in multiple languages, in a new, severely low resource language. We examine two approaches: 1.) best selection of seed sentences to jump start translations in a new language in view of best generalization to the remainder of a larger targeted text(s), and 2.) we adapt large general multilingual translation engines from many other languages to focus on a specific text in a new, unknown language. We find that adapting large pretrained multilingual models to the domain/text first and then to the severely low resource language works best. If we also select a best set of seed sentences, we can improve average chrF performance on new test languages from a baseline of 21.9 to 50.7, while reducing the number of seed sentences to only ∼1,000 in the new, unknown language.

## introduction a language dies when no one speaks it. an endangered language is a language that is spoken by enough people that it could survive under favorable conditions but few or no children are learning it @xcite @xcite . more than half of the 7,139 languages will die in the next 80 years @xcite . endangered languages may survive and thrive if they gain prestige, power and visibility @xcite . frisian, for example, struggles to gain prestige in germany, and is endangered even though it has a large number of speakers. hebrew, conversely, has been revived as a spoken language because it is critical to the development and identity of the jewish community. we empower endangered language communities by exercising a language. this can be achieved by translating important texts to their language so that these communities can gain information, knowledge, power and visibility in their own language. one life-saving example of this knowledge-transfer is translating water, sanitation and hygiene (wash) text into their languages, a process that has long started before the covid-19 pandemic but has gained much attention since then @xcite . the problem in these scenarios, therefore, is not to build a high accuracy translation engine for any texts using huge data corpora, but rather to build a good translation for a known text (for which translations in many other languages exist), but in a new language with only extremely little seed data (a few hundred sentences). we assume there is little to no endangered language data and few human translators. to produce high quality translation, existing methods rely on a seed corpus produced by human translators. previous work has shown progress in using extremely small seed corpora with as small as ∼1,000 lines of data and has found that random sampling performs better than choosing a fixed por-1 tion of the text to build a seed corpus @xcite @xcite . but researchers have yet to 1.) examine various active learning (al) methods to improve accuracy and effectiveness in building better optimized seed corpora so as to minimize the initial human effort and 2.) completely solve the problem of using large multilingual models for representational learning so that we can train (or adapt) them to a new language using an extremely small seed corpus. to solve these two problems, we propose explainable and robust active learning methods that perform as well as or better than random sampling; we transfer methods learned on data of known languages to the new, endangered language. we also examine different training schedules and we find a strategic way of growing large multilingual models in a multilingual and multi-stage fashion with extremely small endangered seed corpora. in our translation workflow, human translators are informed by machine sentence ranking to produce a seed corpus. machine systems then use this seed corpus to produce a full translation draft. human translators post-edit the draft, and feed new data to machines each time they finish post-editing a portion of the text. in each iteration, machines produce better and better drafts with new data, and human translators find it easier and faster to postedit. together they complete the translation of the whole text into an endangered language (figure 1 ). to produce sentence ranking, traditional active learning approaches assume abundant data, but we have little to no data in the target endangered language. we question this assumption and build seed corpora by ranking all sentences in existing translations from other languages to generalize to a new, endangered language. this ranking is targetindependent as we do not require any endangered language data. to produce such a ranking, we explore active learning methods ( to curate a seed corpus in the new, endangered language where we have no data initially, we pass the sentence ranking learned from known languages to human translators. human translators take this ranking, and translate the top few (∼1,000 or less) sentences, curating the seed corpus. to train on such small seed corpus, we find pretraining to be key. for the pretrained model, we either create our own pretrained model by training on known languages, or use an existing pretrained model. we explore both paths in our work, with and without activating the knowledge in existing large pretrained models. we observe an average increase of 28.8 in chrf score over the baselines. our contribution is three-fold: 1. we develop 14 active learning methods on known languages and transfer ranking to the new, endangered language; 2. we activate the knowledge of large multilingual models by proposing multilingual and multi-stage adaptations through 24 different training schedules; we find that adapting pretrained models to the domain and then to the endangered language works best; 3. we aggregate scores from 115 languages to provide a universal ranking and increase robustness by relaxed memoization method. ## methods we translate a fixed text that is available in many languages to a new, endangered language. in our translation workflow, we first develop active learning methods to transfer sentence ranking from known languages to a new, endangered language. we then pass this ranking to human translators for them to translate the top few (∼1,000 or less) sentences into the endangered language, curating the seed corpus. we finally train on the seed corpus, either from scratch or from a pretrained model. we build training schedules on an extremely small seed corpus, we also build active learning strategies of creating and transferring the sentence ranking to the new, endangered language. we propose and compare 24 training schedules and 14 active learning methods for machine translation into a new, endangered language. to compare all active learning algorithms fairly, we use the same translation system unit as a control for all experiments, varying only the seed corpora built by different methods. we select the same number of words in all seed corpora as most translators are paid by the number of words @xcite @xcite . ## data existing research classifies world languages into resource 0 to 5, with 0 having the lowest resource and 5 having the highest @xcite . we choose 10 target languages ranging from resource 0 to 5 ( to translate into these languages, our text is the bible in 125 languages @xcite . each endangered seed corpus contains ∼3% of the text, while all other languages have full text. our goal is to translate the rest of the text into the endangered language. in pretraining, we use a 80/10/10 split for training, validation and testing, respectively. in training, we use approximately a 3.0/0.2/96.8 split for training, validation and testing, respectively. our training data for each experiment is ∼1,000 lines. we use bpe with size of ∼3,000 for the endangered language and ∼9,000 for the combined @xcite . training on ∼100 million parameters with geforce rtx 2080 ti and rtx 3090, we use a 6-layer encoder and a 6-layer decoder with 512 hidden states, 8 attention heads, 512 word vector size, 2,048 hidden units, 6,000 batch size, 0.1 label smoothing, 2.5 learning learning rate and 1.0 finetuning learning rate, 0.1 dropout and attention dropout, a patience of 5 after 190,000 steps in [n] 2 with an update interval of 1000, a patience of 5 for [n+1] 2 with an update interval of 200, and a patience of 25 for [n+1] and [1] 2 with an update interval of 50, "adam" optimizer and "noam" decay method @xcite . ## results for simplicity, we use the centeredness method to combine translations from all source languages and have one score per metric. to compare across different methods, all experiments have the same test set (3,461 lines), the intersection of all test sets. our models improve over the baselines: with schedule i, we observe an average improvement of 24.7 in chrf score over the m2m100 baseline ( our strategic training schedule improves the translation further by activating the knowledge of m2m100 : with schedule b and the 4-gram model, we observe an average improvement of 18.6 in chrf score over the multilingual baseline ( ## future work we propose 24 training schedules for translation into endangered languages. we also propose and compare 14 active learning methods to build seed corpus without any endangered language data. our model is robust with large multilingual models. while the industry trend is to move towards bigger models with bigger data, our minimalist approach not only uses fewer languages, but we also aggregate over fewer languages. this saves computation power and resources, and therefore time and money, while improving translation performance. however, we still face challenges with the lack of local coherence and context. the excerpt-based approach enjoys advantage with formality, cohesion and contextual relevance. active learning methods, on the contrary, do not have consecutive sentences and therefore lose local coherence and pose challenges to human translators @xcite @xcite @xcite @xcite . this is an active research area. evaluation is still a challenge. it is difficult to find native speakers and establish long-term collaborations. there is also much variety among endangered languages. some are more accessible than others and these might provide earlier, realistic evaluation of our method. empowering endangered languages is not just a technology problem. it requires much efforts in communication with local communities. through our technologies, we would like to work with local communities to revive endangered languages and bring them to flourish.