EAG : Extract and Generate Multi-way Aligned Corpus for Complete Multi-lingual Neural Machine Translation (2022)

Abstract: Complete Multi-lingual Neural Machine Translation (C-MNMT) achieves superior performance against the conventional MNMT by constructing multi-way aligned corpus, i.e., aligning bilingual training examples from different language pairs when either their source or target sides are identical. However, since exactly identical sentences from different language pairs are scarce, the power of the multi-way aligned corpus is limited by its scale. To handle this problem, this paper proposes “Extract and Generate” (EAG), a two-step approach to construct large-scale and high-quality multi-way aligned corpus from bilingual data. Specifically, we first extract candidate aligned examples by pairing the bilingual examples from different language pairs with highly similar source or target sentences; and then generate the final aligned examples from the candidates with a well-trained generation model. With this two-step pipeline, EAG can construct a large-scale and multi-way aligned corpus whose diversity is almost identical to the original bilingual corpus. Experiments on two publicly available datasets i.e., WMT-5 and OPUS-100, show that the proposed method achieves significant improvements over strong baselines, with +1.1 and +1.4 BLEU points improvements on the two datasets respectively.

## introduction multilingual neural machine translation (mmmt) @xcite @xcite has achieved promising results on serving translations between multiple language pairs with one model. with sharing parameters of the model, mnmt can facilitate information sharing between similar languages and make it possible to translate between low-resource and zero-shot language pairs. since the majority of available mt training data are english-centric, i.e., english either as the source or target language, most non-english language pairs do not see a single training example when training mnmt models @xcite . therefore, the performance of mnmt models on non-english translation directions still left much to be desired: 1) lack of training data leads to lower performance for non-english language pairs @xcite ; 2) mnmt models cannot beat the pivot-based baseline systems which translate non-english language pairs by bridging through english @xcite . recently, @xcite re-kindle the flame by proposing c-mnmt, which trains the model on the constructed multi-way aligned corpus. specifically, they extract the multi-way aligned examples by aligning training examples from different language pairs when either their source or target sides are identical (i.e., pivoting through english, for @xmath0. since they directly extract the multi-way aligned examples from the bilingual corpus, we refer to their approach as the extraction-based approach. despite improving the performance, the scale of multi-way aligned corpus extracted by @xcite is always limited compared to english-centric bilingual corpus, e.g., only 0.3m german-russian-english multi-way aligned corpus extracted from 4.5m german-english and 33.5m english-russian bilingual corpus. a simple idea for remedying this problem is to add the roughly-aligned corpus by extracting the training examples when either their source or target sides are highly similar. however, our preliminary experiments show that the performance of the model decreases dramatically when we train the model with appending the roughlyaligned corpus. 1 one possible solution, referred to as the generation-based approach, is to generate the multi-way aligned examples by distilling the knowledge of the existing nmt model, e.g., extracting german-english-french synthetic threeway aligned data by feeding the english-side sentences of german-english bilingual corpus into the english-french translation model. although the generation-based approach can theoretically generate non-english corpus with the same size as original bilingual corpus, its generated corpus has very low diversity as the search space of the beam search used by nmt is too narrow to extract diverse translations @xcite @xcite , which severely limits the power of the generation-based approach. in order to combine advantages of the two branches of approaches mentioned above, we propose a novel two-step approach, named eag (extract and generate), to construct large-scale and high-quality multi-way aligned corpus for c-mnmt. specifically, we first extract candidate aligned training examples from different language pairs when either their source or target sides are highly similar; and then we generate the final aligned examples from the pre-extracted candidates with a well-trained generation model. the motivation behind eag is two-fold: 1) although identical source or target sentences between bilingual examples from different language pairs are scarce, highly similar sentences in source or target side are more wide-spread; 2) based on the pre-extracted candidate aligned examples which have highly similar source or target sentences, eag can generate the final aligned examples by only refining the sentences partly with a few modifications. therefore, the non-english corpus constructed by eag has almost identical diversity to the original bilingual corpus. experiments on the publicly available data sets, i.e., wmt-5 and opus-100, show that the proposed method achieves substantial improvements over strong baselines. mnmt considering training a separate model for each language pair is resource consuming, mnmt @xcite @xcite is introduced to translate between multiple language pairs using a single model @xcite @xcite . we mainly focus on the mainstream mnmt model proposed by @xcite , which only introduces an artificial token to the input sequence to indicate which target language to translate. c-mnmt c-mnmt is proposed to build a complete translation graph for mnmt, which contains training examples for each language pair @xcite . a challenging task remaining is how to get direct training data for non-english language pairs. in @xcite , non-english training examples are constructed by pairing the non-english sides of two training examples with identical english sides. however, this method can't get large-scale training examples since the quantity of exactly identical english sentences from different language pairs is small. another feasible solution is to generate training examples with pivot-based translation where the source sentence cascades through the pre-trained @xmath1. despite a large quantity of corpus it can generate, its generated corpus has very low diversity @xcite @xcite . ## methods the proposed eag has a two-step pipeline. the first step is to extract the candidate aligned examples from the english-centric bilingual corpus. the second step is to generate the final aligned examples from the candidates extracted in the first step. ## experiments and results for fair comparison, we evaluate our methods on the publicly available dataset wmt-5, which is used by @xcite . additionally, we test the scalability of our method by further conducting experiments on opus-100, which contains english-centric bilingual data from 100 language pairs @xcite . in the extraction process, we run our extraction code on the cpu with 24 cores and 200g memory. 2 in the generation process, we take transformer-big @xcite as the configuration for m, and m is trained with the self-constructed examples mentioned in section 3.2 on eight v100 gpu cards. 3 we choose transformer as the basic structure for our model and conduct experiments on two standard configurations, i.e, transformer-base and transformer-big. all models are implemented based on the open-source toolkit fairseq @xcite and trained on the machine with eight v100 gpu cards. 4 all bilingual models are trained for 300,000 steps and multi-lingual models are trained for 500,000 steps. we add a language token at the beginning of the input sentence to specify the required target language for all of the multi-lingual models. for the hyper-parameters β and γ, we set them as 0.5 and 0.3 by default and also investigate how their values produce effects on the translation performance. ## analysis we analyze the proposed method on opus-100 and take the transformer-base as the model structure. ## conclusion and future work in this paper, we propose a two-step approach, i.e., eag, to construct large-scale and high-quality multi-way aligned corpus from english-centric bilingual data. to verify the effectiveness of the proposed method, we conduct extensive experiments on two publicly available corpora, wmt-5 and opus-100. experimental results show that the proposed method achieves substantial improvements over strong baselines consistently. there are three promising directions for the future work. firstly, we plan to test whether eag is applicable to the domain adaptation problem in nmt. sec-ondly, we are interested in applying eag to other related tasks which need to align different example pairs. finally, we want to investigate other model structures for the generation process. algorithm 1 generating final aligned corpus: given the aligned candidates {x 1 , y 1 , x 2 , y 2 }; an nmt model m, noisy probability β, word list w b ; return the final aligned corpus {y 1 , ỹ2 }