Improving J ejueo- K orean Translation With Cross-Lingual Pretraining Using J apanese and K orean (2022)

Abstract: Jejueo is a critically endangered language spoken on Jeju Island and is closely related to but mutually unintelligible with Korean. Parallel data between Jejueo and Korean is scarce, and translation between the two languages requires more attention, as current neural machine translation systems typically rely on large amounts of parallel training data. While low-resource machine translation has been shown to benefit from using additional monolingual data during the pretraining process, not as much research has been done on how to select languages other than the source and target languages for use during pretraining. We show that using large amounts of Korean and Japanese data during the pretraining process improves translation by 2.16 BLEU points for translation in the Jejueo → Korean direction and 1.34 BLEU points for translation in the Korean → Jejueo direction compared to the baseline.

## introduction low-resource machine translation has recently attracted more attention in the field of natural language processing as neural machine translation (nmt) systems typically do not perform well for low-resource languages, where parallel data are lacking @xcite . current machine translation systems typically use tens or even hundreds of millions of parallel sentences as training data, but this type of data is only available for a small number of language pairs @xcite . however, there are many examples of lowresource languages that have many speakers @xcite , so more attention is needed in the field of machine translation to serve speakers of these languages. additionally, for the purpose of helping to preserve language and culture and providing equitable access to technology, it is important to improve machine translation for speakers of all languages, even those that have a small number of speakers. jejueo (jeju language, iso 639-3 language code: jje) is a language spoken on jeju island, located just south of the korean peninsula. it is closely related to but mutually unintelligible with korean (iso 639-3 language code: kor) @xcite . it was also classified as a critically endangered language by unesco in 2010, meaning that its youngest fluent speakers are grandparents or greatgrandparents @xcite . despite academic efforts to preserve jejueo @xcite @xcite , datadriven approaches have not been explored deeply @xcite . there are only 5,000 -10,000 fluent speakers of jejueo, and most of these speakers are more than 70 years old @xcite , so it is hard to acquire jejueo data themselves, let alone parallel data between jejueo and korean. despite this scarcity of data, translation between jejueo and korean is an important task due to their lack of mutual intelligibility. we propose a method that uses an mbart @xcite implementation of fairseq foot_0 @xcite and leverages the use of large amounts of linguistically similar languages during pretraining to improve the accuracy of translation between korean and jejueo. we show that using large amounts of japanese and korean monolingual data during pretraining improves translation by 2.16 bleu points in the @xmath0.34 bleu points in the @xmath1. we build on this work by taking a closer look at how the selection of language for these monolingual data used during the pretraining process affects translation quality in the case of translation between jejueo and korean. our methods are described in the following section. ## conclusions and future work we have shown how pretraining on a large amount of carefully selected monolingual data can improve the quality of translation between korean and jejueo, a low-resource language pair. by using japanese and korean data during the pretraining process, our model was exposed to some linguistic diversity beyond korean and jejueo from a language of relatively high linguistic similarity, allowing our model to improve translation by 2.16 bleu points for translation in the @xmath2.34 bleu points for translation in the @xmath3. if enough is known linguistically about the source and target languages, it is important to carefully select additional but similar languages to use during the pretraining process. pretraining with korean alone and pretraining with other languages of low linguistic similarity generated models that performed worse than the baseline. syntactic similarity may be of particular importance as korean, jejueo, and japanese all share a similar syntax while differing mostly in vocabulary. korean, japanese, and jejueo are all considered synthetic sov (subject-object-verb) languages, while chinese is an analytic svo language. this drastic difference in syntax may explain how using chinese during the pretraining process resulted in a marked drop in translation quality. japanese, jejueo, and korean, however, do share many words that come from chinese origins. for future work, we are interested in better leveraging these cognates found amongst korean, jejueo, and japanese as shared representations that can be used as additional linguistic information for improving translation quality. francis zheng, machel reid, edison marrese-taylor, and yutaka matsuo. 2021. low-resource machine translation using cross-lingual language model pretraining. in proceedings of the first workshop on natural language processing for indigenous languages of the americas, pages 234-240, online. association for computational linguistics.