Benefiting from Language Similarity in the Multilingual MT Training: Case Study of I ndonesian and M alaysian (2022)

Abstract: The development of machine translation (MT) has been successful in breaking the language barrier of the world’s top 10-20 languages. However, for the rest of it, delivering an acceptable translation quality is still a challenge due to the limited resource. To tackle this problem, most studies focus on augmenting data while overlooking the fact that we can borrow high-quality natural data from the closely-related language. In this work, we propose an MT model training strategy by increasing the language directions as a means of augmentation in a multilingual setting. Our experiment result using Indonesian and Malaysian on the state-of-the-art MT model showcases the effectiveness and robustness of our method.

## introduction in machine translation (mt), the definition of "lowresource" is not always tied to the language itself, but also to the pair of which languages we want to translate. for example, although japanese cannot be classified as a low-resource language, training a japanese-to-indonesian (@xmath0. accordingly, researches on multilingual mt @xcite focus on improving translation results in low-resource language with the help of other high-resource languages in a unified singular model. however, such improvements are just significant in the translation directions involving english (@xmath1, while on non-english translation pair (@xmath2. in this work, we propose an mt model training strategy that benefits from the similar language, even when the similar language does not include initially in the model. we showcased the effectiveness of our proposed method using a commonly known similar language family of @xmath3. we use indonesian and malaysian as examples of closely-related languages. both languages are commonly known as such due to their similar geographical and historical contexts, where it was used as the lingua franca throughout the malay archipelago for over a thousand years @xcite . popular strategies to improve the models that use low-resourced languages are based on data augmentation or transfer learning @xcite ) from a richer-resourced language. however, in this work, we focus on investigating exclusively the impact of including additional language-direction in the training process of a multilingual mt. we present several alternatives in a multilingual context where translation directions of similar languages can be used in combination, and how they can benefit each other. as malaysian and indonesian are similar languages, we hypothesize that performing multilingual training from japanese in these two directions could be mutually beneficial. we detail our proposal in section 3. in section 4 we describe the settings of the data and the models built. the performance of the different models are displayed in section 5 and an analysis of the outputs in section 6. finally, we conclude this paper in section 7 and propose different experiments that could be carried out in the future. ## related work in a resource-rich condition, improving an mt model can be done by simply adding more parallel data. this is possible for some european language pairs such as en-de, fr-en, en-it and others. however, for most of the languages in the world, such data is more limited, which also yields to a adding ja-ms as additional multilingual direction, (d) @xcite add additional multilingual direction with partially-related pairs, and (e) @xcite add additional multilingual direction with fully-related pairs. more limited performance. several method such as mixture-ofexperts @xcite , data augmentation by paraphrasing @xcite @xcite or by backtranslation @xcite . unfortunately, augmentation methods looking at the closeness of the language are sometimes overlooked. we argue that before applying an augmentation method that generates synthetic data, we should first focus on using the already available natural data from neighboring similar languages. previous studies, such as the work of aw et al. @xcite and @xcite , focus on developing translation between indonesian and malaysian from the perspective of low-resource language. however, we observed that in practice, the translation demand actually comes from high-resource to low-resource language and vice-versa. given that, the monolingual data will be imbalanced in either source and target of the translation, in addition to the difficulties of looking for parallel data. similarly, @xcite studied transfer learning for nmt, in which a model built on highresourced language data is used as an initialization for training on a similar low-resource language. while this was developed with data scarcity in mind, this is different from our work as we focus on the benefits of training simultaneously two low-resourced languages rather than transferring the knowledge from one language to another. furthermore, @xcite proposed a method to paraphrase between indonesian and malaysian through a confusion network. the para-phrase between both languages were then used to enrich the phrase table probability in the statistical machine translation (smt) settings. unfortunately, such methods are not compatible with the current state-of-the-art mt model, where probabilities are implicit and updated by backpropagation. ## extending the bilingual mt with more training directions in this study, we develop a training strategy that leverages the already available natural data as a means of augmentation, with the closeness of the language in mind, in a multilingual training context. in particular, we use a multilingual pretrained model such as the mbart-50 @xcite (more information in section 4). this configuration allows us to explore different approaches to build models. we take advantage of this multilingual setting to propose alternatives to improve the japanese-to-indonesian and japanese-to-malaysian translations. the mbart-50 is a sequence-to-sequence model trained on 50 languages (many-to-many language directions, pivoting via english). 1 the model is built following the work of @xcite . first, a denoising autoencoder on different languages is trained. then, they performed multilingual training using parallel data where each sentence (both source and target) has a language identifier tag attached in the beginning. in this work, we refer as "train" to the process of training the mbart-50 model, although in practice it corresponds to a fine-tuning task. the term "finetune" is used exclusively to describe the process of further tuning a model that has already been trained with our data. a summary of our experiments is presented in figure 1 where we display different configurations of inclusion of language directions. a straightforward approach to address the problem would be to build one mt model in each language direction (onedirect, figure 1a ). we use this setting as the baseline. one common alternative to benefit from both languages is to concatenate the datasets (oneconcat, figure 1b ). in our case, we append the training sentence pairs from ja-ms into that of ja-id. then we execute the training assuming @xmath4, mbart-50, has not been trained on malaysian sentences). this study explores mostly how the translation quality can improve when the mt models of similar languages such as indonesian and malaysian are trained together instead as a separate translation direction. therefore, we build a model where the train consist on a multilingual training using both ja-id and ja-ms data (multdirect, figure 1c ). additionally, we are also interested in exploring the impact of introducing additional similar language pairs. particularly, we explore increasing the language direction in two cases: multpart, which involves adding @xmath5, including therefore an additional language in the source side (figure 1d ); and multfull, which imply adding exclusively sentences on fully-related languages of the target sides, i.e. indonesian and malaysian (figure 1e ). note that, except for oneconcat, in all the experiments we only change the number of language directions in the training process. the training data of each language pair remains always the same. ## experimental settings to conduct the experiments, we build models based on the pretrained mbart-50 @xcite model built in fairseq @xcite . it consists of a transformer @xcite model with 12 layers both in the encoder and the decoder. all the sentences, regardless of the language, are tokenized using the same sentencepiece @xcite model. the vocabulary of encoder and decoder is shared. we use the data from "ccmatrix" (fan in those experiments were more language direction are added (i.e. multpart and multfull), we also use the datasets from "ccmatrix". the sizes of these are displayed in we use l1-l2 notation to refer to a dataset of pairs of sentences and @xmath6. in the case of multiple translation directions from the same source language, we use @xmath7,l3} notation. finally, l1↔l2 implies that the translation directions are both @xmath8. ## experimental results the performances of the models are evaluated using both bleu @xcite metric, which is based on the overlap of n-grams, and chrf2 @xcite which is a character-based dataset size en-id 15.7m en-ms 5.4m ms-id 7.8m in the first subtable, we show the performance of onedirect, i.e. bilingual mt trained on a single direction, @xmath9. we use these models as baselines. ## conclusion and future work in this work, we explored different techniques to improve the training of mt models to translate from japanese into indonesian and malaysian. as finding resources in these target languages is not always easy, we focused on how to benefit from their similarities in multilingual conditions. the results showed how training in both directions jointly boosts the translation quality of each translation. however, an interesting outcome is that simply including additional language pairs alone does not necessarily lead to improvements. in some cases, an additional step of fine-tuning was required so the models achieve higher performances than those built in a single direction. we believe that the outcomes of these experiments are also applicable to other languages. in the future, we want to explore this approach for other language families, such as romance or slavic, or even dialects or variations of the same language. some of them may also be low-resourced and it may be difficult to find enough data to build competitive mt models. on top of that, the techniques investigated in this study are complementary to other strategies (e.g. data augmentation, zero-shot learning) that are commonly used to increase the performance of low-resourced mt models. accordingly, we suppose that a combination of them with our proposed training strategy could further improve the translation performance.