On the Complementarity between Pre-Training and Random-Initialization for Resource-Rich Machine Translation (2022)

Abstract: Pre-Training (PT) of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains (some- times, even worse) on resource-rich NMT on par with its Random-Initialization (RI) counterpart. We take the first step to investigate the complementarity between PT and RI in resource-rich scenarios via two probing analyses, and find that: 1) PT improves NOT the accuracy, but the generalization by achieving flatter loss landscapes than that of RI; 2) PT improves NOT the confidence of lexical choice, but the negative diversity by assigning smoother lexical probability distributions than that of RI. Based on these insights, we propose to combine their complementarities with a model fusion algorithm that utilizes optimal transport to align neurons between PT and RI. Experiments on two resource-rich translation benchmarks, WMT’17 English-Chinese (20M) and WMT’19 English-German (36M), show that PT and RI could be nicely complementary to each other, achieving substantial improvements considering both translation accuracy, generalization, and negative diversity. Probing tools and code are released at: https://github.com/zanchangtong/PTvsRI .

## introduction pre-training (pt; @xcite has achieved tremendous success in natural language processing fields. inspired by bert @xcite , recent works (song et al., 2019; * work was done when changtong was interning at jd explore academy. while recent studies have empirically shown their benefit for the low-resource translation task where the labeled (i.e. parallel) sentences are limited @xcite @xcite , we are generally confronted with resourcerich scenarios, e.g. millions of parallel sentence pairs, in wmt evaluations @xcite and industries. for these resource-rich tasks, pt becomes less effective (sometimes, even worse) than their random-initialization (ri) counterparts, for example, as @xcite reported, the pt underperforms ri if improperly utilized or significant amount of bi-text data is given. however, there is limited understanding of: 1 why does pt fail compared to ri in terms of translation accuracy in high-resource settings? 2 what is the difference between the optimized pt and ri models? 3 how can we harmonize pt with ri? can we just leverage their advantages? to this end, we introduce two probing analyses (i.e. loss landscape and lexical probability distribution) for the fully-optimized pt and ri models to investigate 1 and 2 , respectively. we find that: 1 supervised training with significant amount of parallel data, i.e. ri, is enough to optimize the model towards a better optimum point compared with pt, thus leading to better indomain (same domain between test and training set) performance; 2 pt mainly contributes to the generalization ability because of its flatter loss landscape and smoother lexical probability distribution, thus resulting in better out-domain performance and diversified generation. motivated by this finding, we propose a simple combination method to reach 3 by aligning the neurons and weights of pt and ri and then fusing them into a single model based on optimal transport (monge, 1781; singh and jaggi, 2020). experiments conducted on the wmt17 english-chinese and wmt19 english-german benchmarks show that pt can nicely complement ri, leading to better translation accuracy, generalization, and negative diversity. ## experimental setup data we conducted experiments on wmt17 english-chinese (en-zh) and wmt19 english-german (en-de) translation tasks, which are widely-used resource-rich benchmarks, and include 20m and 36m sentence pairs, respectively. setting to make a fair comparison, all the model backbone architectures are the same as the pretrained mbart.cc25 foot_0 . we tokenize with 250k sentencepiece model @xcite of mbart and remove tokens that are not present in the downstream task from the vocabulary. for pt, we also initialize the model with trimmed embedding layers. for evaluation, we use sacre-bleu @xcite to measure the translation quality with default tokenizer. in this section, we aim to better understand the similarities and differences between pt and ri by introducing two probing analyses. the analyzed pt and ri models are trained on wmt19 en-de benchmark. we first present the loss landscape visualization of both models in section 3.1, and then show their difference in lexical probability distribution in section 3.2. ## harmonizing pt and ri based on the above experiments, we prove that pt mainly contributes to the generalization ability and ri leads to better in-domain performance. to combine the advantage of both models, we fuse both models via optimal transport based on activations following singh and jaggi (2020). ## conclusion and future work this paper provides several insights for pre-training (pt) and random-initialization (ri) on resourcerich machine translation. we carefully investigate the impact of pt and ri on loss landscape and negative diversity and reveal that ri leads to a lower loss value of in-domain data, while pt is more beneficial to flatter loss and smoother lexical distribution. we also propose to combine them with model fusion via optimal transport. experimental results on wmt17 en-zh and wmt19 en-de benchmarks show that model fusion is a nice trade-off method to utilize the complementarity of pt and ri. for future work, we would explore other effective fusion algorithms @xcite to combine the merits of pt and ri. also, it will be interesting to investigate whether harmonizing pt and ri could complement existing data augmentation methods for nmt, e.g. back translation @xcite , data rejuvenation @xcite , bidirectional distillation @xcite and training @xcite .