DENTRA : Denoising and Translation Pre-training for Multilingual Machine Translation (2022)

Abstract: In this paper, we describe our submission to the WMT-2022: Large-Scale Machine Translation Evaluation for African Languages under the Constrained Translation track. We introduce DENTRA, a novel pre-training strategy for a multilingual sequence-to-sequence transformer model. DENTRA pre-training combines denoising and translation objectives to incorporate both monolingual and bitext corpora in 24 African, English, and French languages. To evaluate the quality of DENTRA, we fine-tuned it with two multilingual machine translation configurations, one-to-many and many-to-one. In both pre-training and fine-tuning, we employ only the datasets provided by the organizers. We compare DENTRA against a strong baseline, M2M-100, in different African multilingual machine translation scenarios and show gains in 3 out of 4 subtasks.

## introduction despite the compelling performance of machine translation (mt) in many european and asian languages, their quality in african languages is relatively low. this is primarily because there are approximately 2000 known languages in the african continent, out of which very few languages have any significant presence on the web @xcite @xcite . as a result, many african languages are not included in publicly available bitext resources, which are typically created by employing heuristics on large amounts of data crawled from the web @xcite @xcite . to take a step towards addressing the underrepresentation of african languages in mt, wmt-2022 presented the constrained translation track under large-scale multilingual african translation @xcite , which releases bitext and monolingual corpora for 24 african languages, and participants are only allowed to use the provided data. our submission is to the aforementioned track. roughly 34% of the provided data is monolingual, spread across 24 african languages pertaining to this task. since the volume of bitext data provided is limited, our submission aims to leverage the monolingual data to improve the performance of a multilingual machine translation model. to leverage monolingual data in translation, pretraining the model is an obvious choice. there are several existing multilingual pretrained models such as mbart @xcite , mt5 @xcite , byt5 @xcite , mrasp @xcite , mrasp2 @xcite , and m2m-100 foot_0 @xcite that are trained on monolingual data, bitext data, or both, and have been demonstrated to improve translation performance for specific language pairs. but, these models do not include many of the african languages of interest. for example, the 50 languages covered by mbart50 include only two out of the 24 african languages in the shared task while m2m-100 includes 14. moreover, all of these multilingual models rely on specially designated language id tokens to translate between each language pair. as a result, adding unseen languages requires pre-training again. @xcite investigated a way to leverage pre-trained models including m2m-100, mt5, byt5, and mbart for the translation of unseen languages. but the scarcity of african language texts in the pre-training corpora results in a marginal improvement in the translation quality of the fine-tuned model @xcite . among the pre-trained models, the authors have noted that fine-tuning m2m-100 results in the best translation performance for african languages. encoder @xmath0. <@xmath1. <@xmath2. <@xmath3. <@xmath4. <@xmath5. alifikiria kugombea urais mnamo 2016.<@xmath6.<@xmath7.<@xmath8. <@xmath9. <@xmath10. <@xmath11. <@xmath12. <@xmath13. it is martelly's fifth cep in four years. <@xmath14. <@xmath15. <@xmath16, mbart uses a denoising objective @xcite on combined monolingual corpora of several languages to train a transformer sequenceto-sequence model @xcite . this strategy reduces the dependency on bitext by learning meaningful representations for multiple languages. the pre-trained model is fine-tuned for mt using bitext data. while this methodology does result in improved mt performance, the pre-training objective used does not induce the representation of similar sentences across languages to align, since it uses only monolingual data @xcite . in contrast, mrasp2 combines the use of monolingual and bitext corpora. their pre-training methodology is geared towards closing the representation gap across languages, bringing words and phrases with similar meanings across languages closer in the feature space. this results in better multilingual translation performance @xcite . to induce cross-language representations, mrasp2 uses word or phrase level dictionaries to augment both monolingual and bitext data, by replacing randomly chosen tokens in the source sentence by its corresponding words in another language. since in this work we address primarily low-resource or under-represented languages, separate dictionaries are non-trivial to construct. m2m-100 is a massively multilingual model trained on heuristically mined massive bitext corpora spanning 100 languages and 9,900 language pairs @xcite . fine-tuning m2m-100 with bitext data from the shared task results in minimal improvement over a multilingual model trained from scratch (section 6). we hypothesize that this is due to m2m-100's subword tokenizer. since a majority of african languages are written in the latin script, several subword units are common to many languages. for example, using the m2m-100 tokenizer in the wmt dataset, about 96% of the distinct subwords in african languages also appear in english. since english corpora dominate the m2m-100 training dataset, the learnt representation of these common tokens are influenced majorly by english, limiting the contribution of african languages. to address all of the above, we propose dentra, which uses a novel pre-training strategy and is trained exclusively on languages from the shared task using both monolingual and bitext data. inspired by mrasp2 @xcite , our pretraining objective is also designed to explicitly reduce the representation gap between different languages. figure 1 shows an overview of our pretraining technique. to measure the effect of pre-training, we finetune the pre-trained model in one-to-many and many-to-one configurations of multilingual mt . in three out of four setups, average bleu score of fine-tuned dentra exceeds that of fine-tuned m2m-100 by up to 1.56 points. ## definitions and model architecture task description: the constrained translation track under wmt-2022 @xcite al., 2022b) consists of the following subtasks: english to 22 african languages (@xmath17, 22 african languages to english ({afs}→eng), french to 4 monolingual data s i denoising n(s i ) s i translate using mmt model m j (s i ) t j ' q 1 -q translation t j ' s i p p 1 -p denoising + translation n(t j ' ) s i (a) monolingual data bitext data s i , t j src: s i tgt: t j src: t j tgt: s i p p 1 -p denoising + translation n(s i ) t j p p 1 -p denoising + translation n(t j ) s i translation s i t j translation t j s i (b) bitext data figure 2: pre-training data preparation: pre-training objective for each example in the corpora are determined using the above trees. solid lines indicate that both paths are executed. dotted lines from a node indicate that one of its child nodes are selected at random, with the probability distribution along the edges. african languages (@xmath18, 4 african languages to french ({afs}→fra) and 48 african to african languages within geographical and cultural clusters ({afs}→{afs}). in all tasks, training datasets are from the shared task while the validation and the test sets are from flores 200 @xcite . ## methodology our overall methodology employs the pre-training followed by fine-tuning pipeline used in prior work in nmt. @xcite . we present the pre-training strategy used in this work in section 3.1 and discuss the fine-tuning configurations used in our submission in section 3.2. ## datasets and pre-processing for all experiments, we employ the datasets provided by the organizers, which mainly consist of datasets from opus (tiedemann, 2012), mafand @xcite and web crawled aligned through laser @xcite . it is worth mentioning that, in all pre-training and finetuning we only used a monolingual corpus of 26 languages and english and french-centric bitext. we have not used any african to african bitext in our experiments. prior to using for pre-training or fine-tuning, datasets were filtered, cleaned, and preprocessed. ## experimental setup we use tokenized bleu from moses foot_4 to measure the performance of all translations. prior to computing bleu we word tokenize all translations, also using moses. ## analysis in this section, we conduct a set of analytical experiments to better understand the datasets and what contributes to performance gains. figure 4 shows the bleu scores of the dentra_ft model on {eng,fra}→{afs} translation directions, on the both flores 200 test set and the in-domain test set sampled from the bitext data prior to training. the language pairs on the horizontal axis are ordered by the dataset size (left to right in increasing order) independently for english and french centric directions. for the english centric translation (@xmath19, the bleu scores on both test sets have little correlation with the dataset size, indicating noisy data. some languages, such as umb, fuv, kam, and yor have stark difference between flores and in-domain test sets, indicating that these datasets may have predictable patterns that have no relevance to the translation of these languages. this is further exemplified by the comparison to tso, which has a smaller dataset yet exhibits better generalization. further investigations reveal two primary problems with the bitext data. first, some of these languages have several duplicates in the african side of the data. for example, for kamba-english (kam-eng) dataset, the distinct number of kamba sentences is less than 5% of the total dataset size. however, this is not consistent across all languages exhibiting overfitting on the training data, as the number of distinct yoruba (yor) sentences in its bitext is about 95% of the total dataset. second, the african side of many datasets contain a large fraction of indic languages from the social media domain. strict heuristics designed based on manual inspection by the authors rejected about 637, 000 examples as being clearly in the hindi language. clearly, neither langid nor the laser encoder @xcite are able to reliably detect and align data for these languages. we postulate that low resource languages form a vicious cycle for mt systems trained on bitext data created using multilingual encoders. this opens avenues for future work to explore bitext creation for low resource languages. ## conclusion dentra has shown significant performance gains in multilingual machine translation for african languages as demonstrated in this paper. dentra integrates denoising, backtranslation, and translation into the same pre-training setup, and has helped to improve mt performance after finetuning for both english and french centric translation. we have shown that massively multilingual models like m2m-100 may not be a good choice for fine-tuning when the languages to be translated from/to are restricted to a small set. finally, we have studied the variation in performance and reported issues seen in heuristically created bitext data. while this is a known issue, we show this problem to be exacerbated for low-resource languages that share the alphabet with high-resource ones.