Active Learning for Massively Parallel Translation of Constrained Text into Low Resource Languages (2021)

Abstract: We translate a closed text that is known in advance and available in many languages into a new and severely low resource language. Most human translation efforts adopt a portionbased approach to translate consecutive pages/chapters in order, which may not suit machine translation. We compare the portion-based approach that optimizes coherence of the text locally with the random sampling approach that increases coverage of the text globally. Our results show that the random sampling approach performs better. When training on a seed corpus of ∼1,000 lines from the Bible and testing on the rest of the Bible (∼30,000 lines), random sampling gives a performance gain of +11.0 BLEU using English as a simulated low resource language, and +4.9 BLEU using Eastern Pokomchi, a Mayan language. Furthermore, we compare three ways of updating machine translation models with increasing amount of human post-edited data through iterations. We find that adding newly post-edited data to training after vocabulary update without self-supervision performs the best. We propose an algorithm for human and machine to work together seamlessly to translate a closed text into a severely low resource language.

## introduction machine translation has flourished ever since the first computer was made @xcite . over the years, human translation is assisted by machine translation to remove human bias and translation capacity limitations @xcite @xcite @xcite . by learning human translation taxonomy and post-editing styles, machine translation borrows many ideas from human translation to improve performance through active learning @xcite @xcite . we propose a workflow to bring human translation and machine translation to work together seamlessly in translation of a closed text into a severely low resource language as shown in figure 1 and algorithm 1. given a closed text that has many existing translations in different languages, we are interested in translating it into a severely low resource language well. researchers recently have shown achievements in translation using very small seed parallel corpora in low resource languages @xcite @xcite . construction methods of such seed corpora are therefore pivotal in translation performance. historically, this is mostly determined by field linguists' experiential and intuitive discretion. many human translators employ a portion-based strategy when translating large texts. for example, translation of the book "the little prince" may be divided into smaller tasks of translating 27 chapters, or even smaller translation units like a few consecutive pages. each translation unit contains consecutive sentences. consequently, machine translation often uses seed corpora that are chosen based on human translators' preferences, but may not be optimal for machine translation. we propose to use a random sampling approach to build seed corpora when resources are extremely limited. in other words, when field linguists have limited time and resources, which lines would be given priority? given a closed text, we propose that it would beneficial if field linguists translate randomly sampled ∼1,000 lines first, getting the first machine translated draft of the whole text, and then post-edit to obtain final translation of each portion iteratively as shown in algorithm 1. we recognize that the portion-based translation is very helpful in producing quality translation with formality, cohesion and contextual relevance. thus, our proposed way is not to replace the portion-based approach, but instead, to get the best of both worlds and to expedite the translation process as shown in figure 1 . the main difference of the two approaches is that the portion-based approach focuses on preserving coherence of the text locally, while the random-sampling approach focuses on increasing coverage of the text globally. our results show that the random sampling approach performs better. when training on a seed corpus of ∼1,000 lines from the bible and testing on the rest of the bible (∼30,000 lines), random sampling beats the portion-based approach by +11.0 bleu using english as a simulated low resource language on a family of languages ranked by distortion, and by +4.9 using a mayan language, eastern pokomchi, training on a family of languages based on linguistic definition. using random sampling, machine translation is able to produce an apt first draft of the whole text that expedites the subsequent translation iterations. moreover, we compare three different ways of incorporating incremental post-edited data during the translation process. we find that self-supervision using the whole translation draft affects performance adversely, and is best to be avoided. we also show that adding the newly post-edited text to training with vocabulary update performs the best. algorithm 1: proposed joint human machine translation sequence for a given closed text. input: a text of n lines consisting multiple books/portions, parallel in l source languages output: a full translation in the target low resource language, l 0. initialize translation size, @xmath0, vocabulary size, @xmath1, vocabulary update size, @xmath2; 1. randomly sample s (∼1,000) sentences with vocabulary size vs for human translators to produce the seed corpus, update @xmath3, @xmath4; 2. rank and pick a family of close-by languages by linguistic, distortion or performance metric ; while @xmath5. pretrain on the full texts of neighboring languages ; 4. train on the n sentences of all languages in multi-source multi-target configuration ; 5. train on the n sentences of all languages in multi-source single-target configuration ; 6. combine translations from all source languages using the centeredness measure ; 7. review all books/portions of the translation draft ; 8. pick a book/portion with n lines and v more vocabulary ; 9. complete human post-editing of the portion chosen, 2 related works ## methodology we train our models using a state-of-the-art multilingual transformer by adding language labels to each source sentence @xcite @xcite . we borrow the order-preserving named entity translation method by replacing each named entity with __nes @xcite using a multilingual lexicon table that covers 124 source languages and 2,939 named entities @xcite . for example, the sentence "somchai calls juan" is transformed to "__opt_src_en __opt_tgt_ca __ne0 calls __ne1" to translate to chuj. we use families of close-by languages constructed by ranking 124 source languages by distortion measure (famd), performance measure (famp) and linguistic family (famo + ); the distortion measure ranks languages by decreasing probability of zero distortion, while the performance measure incorporates an additional probability of fertility equalling one @xcite . using families constructed, we pretrain our model first on the whole text of nearby languages, then we train on the ∼1,000 lines of low resource data and the corresponding lines in other languages in a multi-source multi-target fashion. we finally train on the ∼1,000 lines in a multi-source single-target fashion @xcite . we combine translations of all source languages into one. let all n translations be t i , @xmath6, . . . , n and let similarity between translations t i and t j be s ij . we rank all translations according to how centered it is with respect to other sentences by summing all its similarities to the rest through j s ij for @xmath7, . . . , n . we take the most centered translation for every sentence, max i j s ij , to build the combined translation output. the expectation of the combined score is higher than that of any of the source languages @xcite . our work differs from the past research in that we put low resource translation into the broad collaborative scheme of human machine translation. we compare the portion-based approach with the random sampling approach in building seed corpora. we also compare three methods of updating models with increasing amount of human post-edited data. we add the newly post-edited data to training in three ways: with vocabulary update, without vocabulary update, or incorporating the whole translation draft in a self-supervised fashion additionally. for best performance, we build the seed corpus by random sampling, update vocabulary iteratively, and add newly post-edited data to training without self-supervision. we also have a larger test set, we test on ∼30,000 lines rather than ∼678 lines from existing research. we propose a joint human machine translation workflow in algorithm 1. after pretraining @xcite . our model produces a translation draft of the whole text. since the portion-based approach has the advantage with formality, cohesion and contextual relevance, human translators may pick and post-edit portion-by-portion iteratively. the newly post-edited data with updated vocabulary is feed back to the machine translation models without self-supervision. in this way, machine translation systems rely on quality parallel corpora that are incrementally produced by human translators. human translators lean on machine translation for quality translation draft to expedite translation. this creates a synergistic collaboration between human and machine. ## data we work on the bible in 124 source languages @xcite , and have experiments for english, a simulated language, and eastern pokomchi, a mayan language. we train on ∼1,000 lines of low resource data and on full texts for all the other languages. we aim to translate the rest of the text (∼30,000 lines) into the low resource language. in pretraining, we use 80%, 10%, 10% split for training, validation and testing. in training, we use 3.3%, 0.2%, 96.5% split for training, validation and testing. our test size @xmath8. we use the book "luke" for the portion-based approach as suggested by many human translators. training on ∼100 million parameters with geforce rtx 2080 ti, we employ a 6-layer encoder and a 6-layer decoder with 512 hidden states, 8 attention heads, 512 word vector size, 2,048 hidden units, 6,000 batch size, 0.1 label smoothing, 2.5 learning rate, 0.1 dropout and attention dropout, an early stopping patience of 5 after 190,000 steps, "bleu" validation metric, "adam" optimizer and "noam" decay method @xcite . we increase patience to 25 for larger data in the second stage of training in figure 2a and 2b . ## results we observe that random sampling performs better than the portion-based approach. random sampling gives a performance gain of +11.0 for english on famd and +4.9 for eastern pokomchi on famo + in to simulate human translation efforts in step 7 and 8 in algorithm 1, we rank 66 books of the bible by bleu score on english's famd and eastern pokomchi's famo + . we assume that bleu ranking is available to us to simulate human judgment. in reality, this step is realized by human translators skimming through the translation draft and comparing performances of different books by intuition and experience. in section 6, we will discuss the limitation of this assumption. performance ranking of the simulated low resource language may differ from that of the actual low resource language. but the top few may coincide because of the nature of the text, independent of the language. in our results, we observe that the narrative books performs better than the philosophical or poetic books. the book "1 chronicles" performs best for both english and eastern pokomchi, and the book "philemon" performs worst for both languages. a possible explanation is that "1 chronicles" is mainly narrative, and contains many named entities that are translated well by the order-preserving lexiconized model. if we compare bleu scores of the best-performing book, random sampling outperforms the portion-based approach by +11.2 on english's famd, and by +1.3 on eastern pokomchi's famo + . in on the other hand, we see that both updated-vocab and old-vocab performs better than seed and self-supervised. updated-vocab's performance is better than old-vocab. an explanation could be that updated-vocab has more expressive power with updated vocabulary. therefore, in our proposed algorithm, we prefers vocabulary update in each iteration. if the vocabulary has not increased, we may skip pretraining to expedite the process. we show how the algorithm is put into practice for english and eastern pokomchi in figure 2a and 2b . we take the worst-performing 11 books as the held-out test set, and divide the other 55 books of the bible into 5 portions. each portion contains 11 books. we translate the text by using the randomly sampled ∼1,000 lines of seed corpus first, and then proceed with human machine translation in algorithm 1 in 5 iterations with increasing number of post-edited portions. the red dotted line is the overall performance of the whole text excluding the seed corpus. we observe that the red dotted curve is steadily increasing for both languages. however, since we are interested in the test results of the held-out set, we evaluate only on the solid lines plotted. for english, we observe that philosophical books like "ecclesiastes" and poetry books like "song of solomon" perform very badly in the beginning, but begin to achieve above 90 bleu scores after adding 33 books of training data. the high performance is due to the multilingual cross-lingual transfer and this is the main reason why we set up our problem as translation of a closed text that are available in many languages to the low resource language. however, some books like "philemon", "hebrews", "james", "titus" remains difficult to translate even after adding 55 books of training data. this shows that adding data may benefit some books more than the others. a possible explanation is that there are multiple authors of the bible, and books differ from each other in style and content. some books are closely related to each other, and may benefit from translations of other books. but some may be very different and benefit much less. for eastern pokomchi, even though the performance of the most difficult 11 books never reach the near perfect bleu score of 90s like that of english experiments, all books has bleu scores that are steadily increasing. surprisingly, we observe good performance with the books that remain difficult with large training data in the english experiments. "philemon", for example, increases to a bleu score of 81.9 with 55 books of training data in eastern pokomchi while it has a bleu score of 28.4 with 55 books of training data in english. this surprising result shows that what is difficult for simulated low resource languages may not be as difficult for real low resource languages. even though eastern pokomchi gives a lower overall bleu score than english, it has a better generalization to the most difficult book. ## conclusion we propose to use random sampling to build seed parallel corpora instead of using the portionbased approach in severely low resource settings. training on ∼1,000 lines, the random sampling approach outperforms the portion-based approach by +11.0 for english's famd, and by +4.9 for eastern pokomchi's famo + . we also compare three different ways of updating the machine translation models by adding newly post-edited data iteratively. we find that vocabulary update is necessary, but self-supervision by pretraining with whole translation draft is best to be avoided. one limitation of our work is that in real life scenarios, we do not have the reference text in low resource languages to produce the bleu scores to decide the post-editing order. consequently, field linguists need to skim through and decide the post-editing order based on intuition. however, computational models can still help. one potential way to tackle it is that we can train on ∼1,000 lines from another language with available text and test on the 66 books. since our results show that the literary genre plays important role in the performance ranking, it would be reasonable to determine the order using a "held-out language" and then using that to determine order in the target low resource language. in the future, we would like to work with human translators who understand and speak low resource languages. another concern human translators may have is the creation of randomly sampled seed corpora. to gauge the amount of interest or inertia, we have interviewed some human translators and many are interested. however, it is unclear whether human translation quality of randomly sampled data differs from that of the traditional portion-based approach. we hope to work with human translators closely to determine whether the translation quality difference is manageable. we are also curious how our model will perform with large literary works like "lord of the rings" and "les misérables". we would like to see whether it will translate well with philosophical depth and literary complexity. however, these books often have copyright issues and are not as easily available as the bible data. we are interested in collaboration with teams who have multilingual data for large texts, especially multilingual covid-19 data.