PEACH : Pre-Training Sequence-to-Sequence Multilingual Models for Translation with Semi-Supervised Pseudo-Parallel Document Generation (2023)

Abstract: Multilingual pre-training significantly improves many multilingual NLP tasks, including machine translation. Most existing methods are based on some variants of masked language modeling and text-denoising objectives on monolingual data. Multilingual pre-training on monolingual data ignores the availability of parallel data in many language pairs. Also, some other works integrate the available human-generated parallel translation data in their pre-training. This kind of parallel data is definitely helpful, but it is limited even in high-resource language pairs. This paper introduces a novel semi-supervised method, SPDG, that generates high-quality pseudo-parallel data for multilingual pre-training. First, a denoising model is pre-trained on monolingual data to reorder, add, remove, and substitute words, enhancing the pre-training documents’ quality. Then, we generate different pseudo-translations for each pre-training document using dictionaries for word-by-word translation and applying the pre-trained denoising model. The resulting pseudo-parallel data is then used to pre-train our multilingual sequence-to-sequence model, PEACH. Our experiments show that PEACH outperforms existing approaches used in training mT5 and mBART on various translation tasks, including supervised, zero- and few-shot scenarios. Moreover, PEACH’s ability to transfer knowledge between similar languages makes it particularly useful for low-resource languages. Our results demonstrate that with high-quality dictionaries for generating accurate pseudo-parallel, PEACH can be valuable for low-resource languages.

## introduction machine translation (mt) involves transferring a text from one language to another. recent investigations have revealed that multilingual pretraining on a large corpus is profitable for nlp * equal contribution systems' performance on multilingual downstream tasks @xcite @xcite @xcite and knowledge transferability between languages @xcite @xcite . furthermore, using parallel data in pre-training encoder and encoder-decoder models effectively increases the models' performance in downstream tasks @xcite . the existing pre-training approaches are mainly based on masked language modeling (mlm) and its variations @xcite @xcite . although using parallel data in pre-training multilingual models improves their performance on downstream tasks, the amount of available parallel data is limited @xcite . moreover, mlm-based objectives for sequence-to-sequence (seq2seq) models usually ask the model to generate an output in the same language as input, which is not in the interests of translation tasks. additionally, mlm-based objectives use shared subwords or alphabets between different languages to learn shared embedding spaces across them (lample and @xcite @xcite ; this would not be possible for languages without shared alphabets. using dictionaries to define anchor points between different languages in cross-lingual pretraining of the encoder of seq2seq models has been investigated and shown to be effective for unsupervised translation @xcite . still, it never has been used as a method for pre-training multilingual seq2seq models. our proposed method, semi-supervised pseudo-parallel document generation (spdg), addresses the challenge of limited parallel data for lowresource languages by leveraging dictionaries to generate pseudo-parallel documents. spdg adopts unsupervised translation techniques @xcite to generate a high-quality translation for each pre-training document. we use a pre-trained denoising seq2seq model with word reordering, adding, removing, and substituting to enhance the quality of the word-by-word translated document. the improved unsupervised translated text is used as the target text for training our multilingual seq2seq model, peach, using spdg as a new pre-training method. spdg enables transfer of knowledge between similar languages, making it particularly useful for lowresource languages. our experiments show that peach outperforms the pre-trained models with mt5's mlm and mbart's mlm with reordering objectives in english, french, and german. additionally, peach demonstrates strong performance in zero-and fewshot scenarios. moreover, we test our model for other multilingual tasks, such as natural language inference, to investigate the model's ability in this task. our results show that our model achieves a higher score in this task than other objectives, which shows peach's ability to transfer knowledge between languages. the main contribution of this paper is twofold: ## related work among the first endeavor for mt, dictionary and rule-based methods were popular @xcite @xcite , followed by knowledge-based machine translation (kbmt) and statistical methods @xcite @xcite . the popularity of neural machine translation has only grown in the recent decade with the introduction of the first deep neural model for translation @xcite . while the rnn-based seq2seq models seemed to be promising in neural machine translation @xcite @xcite , the advent of the transformer architecture @xcite plays an integral role in modern mt. with the introduction of the transformer architecture, pre-training generalpurpose language models seemed to be an effective way to improve different nlp tasks @xcite . in most cases, transformer models were asked to denoise a noisy input to learn a language @xcite @xcite . one of the most popular pre-training objectives for both encoderonly and encoder-decoder models is called masked language modeling (mlm), in which the model should predict the masked part of a document and generate it in its output @xcite . however, many other objectives were also developed for encoder-decoder and encoder-only models @xcite . meanwhile, unsupervised methods for neural machine translation (nmt) using monolingual corpora based on adversarial learning @xcite and transformer-based text denoising @xcite was tested and demonstrated promising outcomes. using bilingual dictionaries for defining anchors in pre-training unsupervised translation models was successful @xcite but never has been used for generating data for supervised translation on a large scale. our work differs from using dictionaries as anchor points for learning a better representation for tokens in encoder @xcite . we use dictionaries to generate a pseudo translation of the source language in the target language instead of just defining some anchor points. thus, the model in pre-training steps learns to generate a text in the target language based on input in the source language using only monolingual data and dictionaries on a large scale. pre-training task-specific models by generating pseudo-summaries was successful in some cases for summarization @xcite , but it has not been performed for pre-training encoder-decoder seq2seq models for supervised translation according to the best of our knowledge. on the other hand, the endeavors for pre-training specific models for translation ended up in training multilingual language models @xcite . mt5 @xcite is trained with the mlm objective of t5 @xcite . in its pre-training objective, some spans of the input document are masked by specific tokens, and the model has to predict those spans by generating them in its output. mbart @xcite ) is another multilingual model based on the bart (lewis et al., 2020) model, pre-trained with mlm with reordering objective. in mbart's pre-training objective, the order of sentences in the input document is corrupted while a specific token masks some spans of the document. the model has to generate the original document in its output. peach is different from both mentioned models because we use a semi-supervised method to generate several pseudo-translations (one for each selected language) of each pre-training document. these translations are then fed to pre-train peach. furthermore, in the mentioned models, the inputs and outputs are from the same language while we ask the model to translate texts from one language to another in our pre-training phase. ## peach peach is a new sequence-to-sequence multilingual transformer model trained with spdg, a semisupervised pseudo-parallel document generation method. this section explains the pre-training objective and the model architecture. ## experiments in this section, we compare the results of peach, trained with spdg, with other common objectives utilized for pre-training multilingual models. to investigate the effectiveness of spdg in comparison with common objectives, we pre-trained two other models based on mt5's mlm objective @xcite ) and mbart's mlm with reordering objective @xcite in the same setup. the codes for pre-training and fine-tuning of all models are publicly available on github 3 . ## conclusion we introduced spdg, a semi-supervised method for pre-training multilingual seq2seq models, to address the lack of parallel data between different languages. in this new method, we use bilingual dictionaries and denoising models trained with reordering, adding, substituting, and removing words to generate a pseudo-translation for each pre-training document. we use this generated data to train our multilingual model, peach, for english, french, and german languages. our results show that peach outperforms the common pre-training objectives for training multilingual models. furthermore, peach shows a remarkable ability in zeroand few-shot translation and knowledge transfer between languages.