Improving Multilingual Neural Machine Translation For Low-Resource Languages: F rench, E nglish - V ietnamese (2020)

Abstract: Prior works have demonstrated that a low-resource language pair can benefit from multilingual machine translation (MT) systems, which rely on many language pairs’ joint training. This paper proposes two simple strategies to address the rare word issue in multilingual MT systems for two low-resource language pairs: French-Vietnamese and English-Vietnamese. The first strategy is about dynamical learning word similarity of tokens in the shared space among source languages while another one attempts to augment the translation ability of rare words through updating their embeddings during the training. Besides, we leverage monolingual data for multilingual MT systems to increase the amount of synthetic parallel corpora while dealing with the data sparsity problem. We have shown significant improvements of up to +1.62 and +2.54 BLEU points over the bilingual baseline systems for both language pairs and released our datasets for the research community.

## introduction neural machine translation (nmt) @xcite has achieved state of the art in various mt systems, including rich and low resource language pairs @xcite @xcite . however, the quality of lowresource mt is quite unpretentious due to the lack of parallel data while it has achieved better results on systems of the available resource. therefore, low-resource mt is one of the essential tasks investigated by many previous works @xcite @xcite . recently, some works present mt systems that have achieved remarkable results for low-resource language @xcite . inspired by these works, we collect data from the ted talks domain, then attempt to build multilingual mt systems from french, english-vietnamese. experiments demonstrate that both language pairs: french-vietnamese and english-vietnamese have achieved significant performance when joining the training. although multilingual mt can reduce the sparse data in the shared space by using word segmentation, however, rare words still exist, evenly they are increased more if languages have a significant disparity in term vocabulary. previous works suggested some strategies to reduce rare words such as using translation units at sub-word and character levels or generating a universal representation at the word and sentence levels @xcite . these help to downgrade the dissimilarity of tokens shared from various languages. however, these works require learning additional parameters in training, thus increasing the size of models. our paper presents two methods to augment the translation of rare words in the source space without modifying the architecture and model size of mt systems: (1) exploiting word similarity. this technique has been mentioned by previous works @xcite @xcite . they employ monolingual data or require supervised resources like a bilingual dictionary or wordnet, while we leverage relation from the multilingual space of mt systems. (2) adding a scalar value to the rare word embedding in order to facilitate its translation in the training process. due to the fact that nmt tends to have bias in translating frequent words, so rare words (which have low frequency) often have less opportunity to be considered. our ideal is inspired by the works of @xcite @xcite . @xcite and @xcite proposed various solutions to urge for translation of rare words, including modification embedding in training. they only experimented with recurrent neural networks (rnns) while our work uses the state-of-the-art transformer architecture. @xcite transforms the word embedding of a token into the universal space, and they learn plus parameters while our method does not. we apply our strategies in our fine-tuning processes, and we show substantial improvements of the systems after some epochs only. monolingual data are widely used in nmt to augment data for low-resource nmt systems @xcite @xcite @xcite . back-translation @xcite is known as the most popular technique in exploiting target-side monolingual data to enhance the translation systems while the self-learning method @xcite focuses on utilizing source-side monolingual data. otherwise, the dual-learning strategy @xcite also suggests using both source-and target-side monolingual data to tackle this problem. our work investigates the selflearning method @xcite on the low-resource multilingual nmt systems specifically related to vietnamese. besides, monolingual data are also leveraged in unsupervised @xcite or zero-shot translation @xcite . the main contributions of our work are: in section 2, we review the transformer architecture used for our experiments. the brief of multilingual translation is shown in section 3. section 4 presents our methods to deal with rare words in multilingual translation scenarios. the exploitation of monolingual data for low-resource multilingual mt is discussed in section 5. our results are described in section 6, and related work is shown in section 7. finally, the paper ends with conclusions and future work. self-attention is an important mechanism in the transformer architecture. it enables the ability to specify the relevance of a word with the remaining words in the sentence through the equation: the mt system is trained to minimize the maximum likelihood of k parallel pairs: the vocabulary of the source side is mixed from all source languages: @xmath0. @xcite has shown that if the languages shared the same alphabet and had many similar words, such system will get many advantages from multilingual mt. in fact, different words from many languages can differ in form, but they may share the same subwords. this significantly reduces the number of rare words in the mt systems. nevertheless, the rare word issue is still a challenge in nmt. we choose english and french are source languages in our experiment with the hope that they can share many tokens even though we do not have much data of those translation directions. ## exploiting monolingual data for low-resource multilingual nmt similar to the idea suggested in @xcite , we leverage monolingual data from the source-side to generate synthetic bilingual data. instead of using monolingual data from all source languages, we only attempt to exploit monolingual data of english. firstly, we train the multilingual nmt system from english, @xmath1. the best system is then used to translate english to vietnamese. lastly, the synthetic parallel data are mixed with original bilingual data in the normal training scheme. ## experiments 6.1 datasets we extracted data from ted talks domain foot_0 for two language pairs english-vietnamese and french-vietnamese. the details of those datasets are described in ## related work due to the unavailability of the parallel data for lowresource language pairs or zero-shot translation, previous works focus on the task to have more data such as leveraging multilingual translation @xcite @xcite @xcite or using monolingual data with back-translation, self-learning @xcite @xcite or mix-source @xcite technique. for leveraging multilingual translation, @xcite added language code and target forcing in order to learn the shared representations of the source words and specify the target words. @xcite demonstrated a one-to-many multilingual mt with three different strategies which modify their architecture. @xcite built many-to-one multilingual mt systems by adding a layer to transform the source embeddings and representation into a universal space to augment the translation of low resource language, which is similar to ours. @xcite implemented a massive many-to-many multilingual system, employing many low-resource language pairs. all of the mentioned works have shown substantial improvements in low-resource translation, however, they are less correlative to our translation tasks. although multilingual mt equips a shared space with many advantages, rare word translation is still the issue that needs to be considered. the task of dealing with rare words has been mentioned in previous works. @xcite copied words from source sentences by words from target sentences after the translation using a bilingual dictionary. @xcite and @xcite learned word similarity from monolingual data to improve their systems. our approach is similar to these works, but we only learn similarity from the shared multilingual space of mt systems. @xcite addressed the rare word problem by using the synonyms from wordnet. monolingual data is used to generate synthetic bilingual data in sparsity data issues. @xcite proposed back-translation method that uses a backward model to get the source data from the monolingual target data. in contrast, @xcite shown the self-learning technique by employing a forward model to translate monolingual source data into the target data. @xcite incorporated both mentioned techniques into their nmt systems. monolingual data is also demonstrated its efficiency in unsupervised machine translation @xcite or in zeroshot multilingual nmt @xcite . in our work, we use the self-learning method to produce pseudo bilingual data, and it is then used to train our low-resource multilingual nmt systems. ## conclusion and future work we have built multilingual mt systems for two lowresource language pairs: english-vietnamese and french-vietnamese, and proposed two approaches to tackle rare word translation. we show that our approaches bring significant improvements to our mt systems. we find that the pseudo bilingual can furthermore enhance a multilingual nmt system in case of @xmath2. in the future, we would like to use more language pairs in our systems and to combine proposed methods in order to evaluate the effectiveness of our mt systems.