Unifying the Convergences in Multilingual Neural Machine Translation (2022)

Abstract: Although all-in-one-model multilingual neural machine translation (MNMT) has achieved remarkable progress, the convergence inconsistency in the joint training is ignored, i.e., different language pairs reaching convergence in different epochs. This leads to the trained MNMT model over-fitting low-resource language translations while under-fitting high-resource ones. In this paper, we propose a novel training strategy named LSSD (LanguageSpecific Self-Distillation), which can alleviate the convergence inconsistency and help MNMT models achieve the best performance on each language pair simultaneously. Specifically, LSSD picks up language-specific best checkpoints for each language pair to teach the current model on the fly. Furthermore, we systematically explore three sample-level manipulations of knowledge transferring. Experimental results on three datasets show that LSSD obtains consistent improvements towards all language pairs and achieves the state-of-the-art.

## introduction neural machine translation (nmt) @xcite has witnessed enormous and significant progress, including network structures @xcite @xcite , attention mechanism @xcite @xcite and decoding strategies @xcite @xcite . while achieving promising performance @xcite , widely-used bilingual nmt actually causes huge computational cost especially when tackling numerous language pairs, thereby facilitating the recent emergence of multilingual nmt (ha et al., 2016; firat et al., 2016; johnson et al., 2017; lu et al., 1 our code is publicly available at https://github. com/orangeinsouth/lssd . 2018; @xcite . by directly translating multiple language pairs with one model, multilingual nmt @xcite @xcite quadratically accelerates deployment and effectively encourages transfer learning between similar languages, which greatly benefits low-resource directions @xcite and successfully enables zero-shot translation @xcite . despite the remarkable success, multilingual nmt @xcite clearly expresses a strong disagreement on the uniform convergence point across various translation corpora. the underlying reason is the imbalance and heterogeneity of available data in multilingual training @xcite , which also explains why the disagreement between highresource languages (hrls) and low-resource languages (lrls) is more pronounced, as illustrated in figure 1 . particularly, a fairly common and impor-tant observation about the learning of multilingual nmt is that hrls typically encounter underfitting, while extremely severe overfitting generally arises in lrls. in this paper, we cast this convergence inconsistency as the performance deficit between the multilingual nmt model and its own languagespecific best checkpoints, and aim to reduce this deficit. towards tacking this problem, we propose a novel training strategy dubbed language-specific self-distillation (lssd). at each training step, lssd appoints recent best checkpoints oriented to each language pair as teacher models and treats the current training model as the student learning from each teacher model in a knowledge distillation manner. differently depending on data selection and strength of performing knowledge distillation indeed, several practical strategies are employed to potentially provide a relatively finegrained manipulation of knowledge transferring: 1) lssd-whole, which performs distillation on all data samples equally; 2) lssd-selective, which selects samples for distillation conditioned on whether the teacher performs better than the student; 3) lssd-adaptive, which varies the distillation strength according to the sample-level performance ratio between the teacher and student. experimental results on ted talks foot_0 and wmt under many-to-one and one-to-many settings demonstrate that our method can obtain consistent and significant improvement through remedying the convergence inconsistency, ultimately achieving state-of-the-art translation performance. ## language-specific self-distillation in this section, we first introduce the overall process of lssd. then, we provide a detailed formalization of lssd, with special emphasis on the three finegrained manipulations on knowledge transferring. ## conclusion in this work, we propose a novel training strategy language-specific self-distillation (lssd) to remedy the convergence inconsistency in multilingual neural machine translation. moreover, we devise three sample-level manipulations for lssd. experimental results on three datasets demonstrate that lssd achieves sota performance. through analysis experiments, we also find that: 1) lssd significantly mitigates the convergence inconsistency (decreased 57% on average), which is quantified by performance deficit; 2) both low-resource and high-resource languages benefit from lssd.