PARADISE : Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining (2022)

Abstract: Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora and do not make use of the strong cross-lingual signal contained in parallel data. In this paper, we present PARADISE (PARAllel &Denoising Integration in SEquence-to-sequence models), which extends the conventional denoising objective used to train these models by (i) replacing words in the noised sequence according to a multilingual dictionary, and (ii) predicting the reference translation according to a parallel corpus instead of recovering the original sequence. Our experiments on machine translation and cross-lingual natural language inference show an average improvement of 2.0 BLEU points and 6.7 accuracy points from integrating parallel data into pretraining, respectively, obtaining results that are competitive with several popular models at a fraction of their computational cost.

## introduction multilingual sequence-to-sequence pretraining has achieved strong results both in cross-lingual classification @xcite and machine translation @xcite . these models are usually pretrained on combined monolingual corpora in multiple languages using some form of denoising objective. more concretely, they noise each sequence x with a noising function g φ , and maximize the probability of recovering x given g φ (x): common noising functions include sentencepermutation and span masking @xcite . while these methods obtain strong cross-lingual performance without parallel data, they are usually trained at a scale that is prohibitive for most nlp practitioners. at the same time, it has been argued that the strict unsupervised scenario is not realistic @xcite , and parallel data could provide a stronger signal and make training more efficient. motivated by this, we propose paradise, a pretraining method for sequence-to-sequence models that exploits both word-level and sentence-level parallel data. the core idea of our approach is to augment the conventional denoising objective introduced above by (i) replacing words in the noised sequence according to a bilingual dictionary, and (ii) predicting the reference translation rather than the input sequence. despite their simplicity, we find that both techniques bring substantial gains over conventional pretraining on monolingual data, as evaluated both in machine translation and zero-shot cross-lingual transfer. our results are competitive with several popular models despite using only a fraction of the compute, providing strong support for the importance of the inclusion of parallel information in smaller-scale multilingual pretraining methods. ## proposed method as illustrated in figure 1 , we propose two methods for introducing parallel data into pretraining: dictionary denoising and bitext denoising. dictionary denoising. our first method encourages learning similar representations at the wordlevel by introducing anchor words through multilingual dictionaries @xcite . let d l (w) denote the translation of word w into language @xmath0. given the source sentence @xmath1, x 2 , . . . , x n ), we define its noised version g ψ (x) = (x 1 , x2 , . . . , xn ), where @xmath2.e. we replace each word with its translation into a random language with probability their 仕事 encoder decoder @xmath3. we set p @xmath4.4. given the dictionary-noised sentence, we train our model using the denoising auto-encoding objective in eq. 1: bitext denoising. our second approach encourages learning from both monolingual and parallel data sources, by including translation data in the pretraining process. given a source-target bitext pair (x, y) in the parallel corpus, assumed to be semantically equivalent, we model the following: in which we optimize the likelihood of generating the target sentence y conditioned on the noised version of the source sentence, g φ (x). 2 combined objective. our final objective combines ℓ mono , ℓ dict and ℓ bitext . 3 given that our corpus contains languages with varying data sizes, we sample sentences using the exponential sampling technique from @xcite . we use α @xmath5.5 to sample from the monolingual corpus, and α @xmath6.3 to sample from the parallel corpus. to prevent over-exposure to english on the decoder side when sampling from the parallel corpus, we halve the probability of to-english directions and renormalize the probabilities. in addition, given that we have fewer amounts of parallel data (used for ℓ bitext ) than monolingual data (used for ℓ mono and ℓ dict ), we sample between each task using α @xmath7.3. ## experimental settings we pretrain our models on 20 languages (english, french, spanish, german, greek, bulgarian, russian, turkish, arabic, vietnamese, thai, chinese, 2 to make our pretraining sequence length consistent with ℓmono and ℓdict, we concatentate randomly sampled sentence pairs from the same language pair to fit the maximum length. 3 we use the same noising function g φ used by @xcite and @xcite . hindi, swahili, urdu, japanese, basque, romanian, sinhala and nepalese), and evaluate them on machine translation and cross-lingual classification. ## related work most prior work on multilingual pretraining uses monolingual data only @xcite @xcite @xcite . there have been several proposals to incorporate parallel data into encoder-only models @xcite @xcite , with some approaches replacing words according to a bilingual dictionary, similar to our dictionary denoising objective @xcite @xcite . in contrast, we focus on sequence-to-sequence models, which we believe are more flexible and provide a more natural way of integrating parallel data. in that spirit, @xcite showed that vanilla machine translation models are already competitive in cross-lingual classification. closer to our work, @xcite incorporated parallel corpora into sequence-to-sequence pretraining by feeding concatenated parallel sentences to the encoder and using different masking strategies. in contrast, our approach feeds a noised sentence into the encoder, and tries to recover its translation in the decoder side, obtaining better results with a similar computational budget. concurrent to our work, @xcite extended t5 to incorporate parallel corpora using a similar approach to our bitext denoising. ## conclusions in this work, we proposed paradise, which introduces two new objectives to integrate parallel data into sequence-to-sequence pretraining. experimental results on machine translation and cross-lingual classification show that paradise provides significant improvements over mbart-style pretraining on monolingual corpora, obtaining results that are competitive with several popular models at a much smaller scale. given these findings, we encourage use of parallel data in smaller-scale multilingual pretraining work. in the future, we look to see if our improvements also hold at a larger scale.