Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Text-based Translation (2021)

Abstract: We translate a closed text that is known in advance into a severely low resource language by leveraging massive source parallelism. In other words, given a text in 124 source languages, we translate it into a severely low resource language using only ∼1,000 lines of low resource data without any external help. Firstly, we propose a systematic method to rank and choose source languages that are close to the low resource language. We call the linguistic definition of language family Family of Origin (FAMO), and we call the empirical definition of higher-ranked languages using our metrics Family of Choice (FAMC). Secondly, we build an Iteratively Pretrained Multilingual Order-preserving Lexiconized Transformer (IPML) to train on ∼1,000 lines (∼3.5%) of low resource data. In order to translate named entities well, we build a massive lexicon table for 2,939 Bible named entities in 124 source languages, and include many that occur once and covers more than 66 severely low resource languages. Moreover, we also build a novel method of combining translations from different source languages into one. Using English as a hypothetical low resource language, we get a +23.9 BLEU increase over a multilingual baseline, and a +10.3 BLEU increase over our asymmetric baseline in the Bible dataset. We get a 42.8 BLEU score for Portuguese-English translation on the medical EMEA dataset. We also have good results for a real severely low resource Mayan language, Eastern Pokomchi.

## introduction we translate a closed text that is known in advance into a severely low resource language by leveraging massive source parallelism. in other words, we aim to translate well under three constraints: having severely small training data in the new target low resource language, having massive source language parallelism, having the same closed text across all languages. generalization to other texts is prefer- eastern pokomchi english famd famp famd famp chuj* dadibi danish* dutch* cakchiquel* thai norwegian* afrikaans* guajajara* gumatj italian norwegian* toba navajo afrikaans* german* myanmar cakchiquel* dutch* danish* slovenský kanjobal portuguese spanish latin guajajara* french frisian* ilokano mam* german* italian norwegian kim marshallese french russian chuj* frisian* portuguese we focus on five challenges that are not addressed previously. most multilingual transformer works that translate into low resource language limit their training data to available data in the same or close-by language families or the researchers' intuitive discretion; and are mostly limited to less than 30 languages @xcite @xcite . instead, we examine ways to pick useful source languages from 124 source languages in a principled fashion. secondly, most works require at least 4,000 lines of low resource data @xcite @xcite ; we use only ∼1,000 lines of low resource data to simulate real-life situation of having ex-tremely small seed target translation. thirdly, many works use rich resource languages as hypothetical low resource languages. moreover, most works do not treat named entities separately; we add an orderpreserving lexiconized component for more accurate translation of named entities. finally, many multilingual works present final results as sets of translations from all source languages; we build a novel method to combine all translations into one. we have five contributions. firstly, we rank the 124 source languages to determine their closeness to the low resource language and choose the top few. we call the linguistic definition of language family family of origin (famo), and we call the empirical definition of higher-ranked languages using our metrics family of choice (famc). they often overlap, but may not coincide. secondly, we build an iteratively pretrained multilingual order-preserving lexiconized transformer (ipml) training on ∼1,000 lines of low resource data. using iterative pretraining, we get a +23.9 bleu increase over a multilingual orderpreserving lexiconized transformer baseline (mlc) using english as a hypothetical low resource language, and a +10.3 bleu increase over our asymmetric baseline. training with the low resource language on both the source and target sides boosts translation into the target side. training on randomly sampled 1,093 lines of low resource data, we reach a 31.3 bleu score testing on 30,022 lines of bible. we have a 42.8 bleu score for portuguese-english translation on the medical emea dataset. thirdly, we use a real-life severely low resource mayan language, eastern pokomchi, a class 0 language @xcite as one of our experiment setups. in addition, we also use english as a hypothetical low resource language for easy evaluation. we also add an order-preserving lexiconized component to translate named entities well. to solve the variable-binding problem to distinguish "ian calls yi" from "yi calls ian" @xcite @xcite , we build a lexicon table for 2,939 bible named entities in 124 source languages including more than 66 severely low resource languages. finally, we combine translations from all source languages by using a novel method. for every sentence, we find the translation that is closest to the translation cluster center. the expectated bleu score of our combined translation is higher than translation from any of the individual sources. ## data we use the bible dataset and the medical emea dataset @xcite . emea dataset is from the european medicines agency and contains a lot of medical information that may be beneficial to the low resource communities. our method can be applied to other datasets like wash guidelines. for the bible dataset, we use 124 source languages with 31,103 lines of data and a target low resource language with ∼1,000 lines (∼3.5%) of data. we have two setups for the target low resource language. one uses eastern pokomchi, a mayan language; the other uses english as a hypothetical low resource language. we train on only ∼1,000 lines of low resource data from the book of luke and test on the 678 lines from the book of mark. mark is topically similar to luke, but is written by a different author. for the first stage of pretraining, we use 80%, 10%, 10% split for training, validation and testing. for the second stage onwards, we use 95%, 5% split of luke for training and validation, and 100% of mark for testing. eastern pokomchi is mayan, and english is germanic. since our database does not have ten members of each family, we use famo + , the expanded version of famo. for english, we include five germanic languages and five romance languages in famo + ; for eastern pokomchi, we include five mayan languages and five amerindian languages in famo + . the amerindian family is broadly believed to be close to the mayan family by the linguistic community. we construct famcs by comparing different ways of ranking linguistic distances empirically based on ∼1,000 lines of training data. in to imitate the real-life situation of having small seed target translation data, we choose to use ∼1,000 lines (∼3.5%) of low resource data. we also include eastern pokomchi in addition to using english as a hypothetical low resource language. though data size can be constrained to mimic severely low resource scenarios, much implicit information is still used for the hypothetical low resource language that is actually rich resource. for example, implicit information like english is germanic is often used. for real low resource scenarios, the family information may have yet to be determined; the neighboring languages may be unknown, and if they are known, they are highly likely to be low resource too. we thus use eastern pokomchi as our real-life severely low resouce language. in addition to the bible dataset, we work with the medical emea dataset @xcite . using english as a hypothetical language, we train on randomly sampled 1,093 lines of english data, and test on 678 lines of data. since there are only 9 languages in germanic and romance families in emea dataset, we include a slavic language polish in our famo + for experiments. the emea dataset is less than ideal comparing with the bible dataset. the bible dataset contains the same text for all source languages; however, the emea dataset does not contain the same text. it is built from similar documents but has different parallel data for each language pair. therefore, during test time, we do not combine the translations from various source languages in the emea dataset. ## results we compare our iteratively pretrained multilingual order-preserving lexiconized transformer (ipml) with five baselines in though the number of low resource training lines is the same as others, information is highly asym- ## conclusion we use ∼1,000 lines of low resource data to translate a closed text that is known in advance to a severely low resource language by leveraging massive source parallelism. we present two metrics to rank the 124 source languages and construct famcs. we build an iteratively pretrained multilingual order-preserving lexiconized transformer and combine translations from all source languages into one by using our centric measure. moreover, we add a multilingual order-preserving lexiconized component to translate the named entities accurately. we build a massively parallel lexicon table for 2,939 bible named entities in 124 source languages, covering more than 66 severely low resource languages. our good result for the medical emea dataset shows that our method is useful for other datasets and applications. our final result can also serve as a ranking measure for linguistic distances though it is much more expensive in terms of time and resources. in the future, we would like to explore more metrics that are fast and efficient in ranking linguistic distances to the severely low resource language. source sentence ipml translation reference ket idi limmabas iti dinna ti baybay ti galilea, nakitana ni simon ken ni andres a cabsatna, nga iwaywayatda ti iket iti baybay; ta dumadaclisda idi. eh noq ojik i rub'an i jesús juntar i k'isa palaw i galilea, xrilow reje i simón ruch'ihil i andres, re' i rutuut i k'isa palaw, ruum jinaj i k'isa palaw barco. noq k'ahchi' rik'iik i jesús chi chii' i k'isa palaw ar galilea, xrilow reje wach i simón ruch'ihil i ruchaaq', andres rub'ihnaal. re' keh aj karineel taqe, k'ahchi' kikutum qohoq i kiya'l pan palaw. ket idi nagna pay bassit nakitana ni santiago nga anac ni zebedeo ken ni juan a cabsatna, nga addada idi iti barangayda, a tartarimaanenda dagiti iketda.