Improving Multilingual Translation by Representation and Gradient Regularization (2021)

Abstract: Multilingual Neural Machine Translation (NMT) enables one model to serve all translation directions, including ones that are unseen during training, i.e. zero-shot translation. Despite being theoretically attractive, current models often produce low quality translations – commonly failing to even produce outputs in the right target language. In this work, we observe that off-target translation is dominant even in strong multilingual systems, trained on massive multilingual corpora. To address this issue, we propose a joint approach to regularize NMT models at both representation-level and gradient-level. At the representation level, we leverage an auxiliary target language prediction task to regularize decoder outputs to retain information about the target language. At the gradient level, we leverage a small amount of direct data (in thousands of sentence pairs) to regularize model gradients. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance by +5.59 and +10.38 BLEU on WMT and OPUS datasets respectively. Moreover, experiments show that our method also works well when the small amount of direct data is not available.

## introduction with neural machine translation becoming the state-of-the-art approach in bilingual machine translations @xcite , multilingual nmt systems have increasingly gained attention due to their deployment efficiency. one conceptually attractive advantage of multilingual nmt @xcite is its capability to translate between multiple source and target languages with only one model, where many wmt fr-de de-fr cs-de de-cs baseline 51.60% 39.80% 13.10% 20.50% ## approach in this section, we will illustrate the baseline multilingual models and our proposed joint representation and gradient regularizations. ## results in this section, we will demonstrate the effectiveness of our approach on both wmt-10 and opus-100 datasets. the full results are documented separately in tables 5 , 6 , and 7 for wmt-10, and tables 8 and 9 for opus-100. ## discussions on off-target translations in this section, we will discuss the off-target translation in the english-centric directions and its rela14 in our case, we obtain 1k for wmt and 2k for opus. ## related work multilingual nmt multilingual nmt aims to train one model to serve all language pairs @xcite @xcite . several subsequent works explored various parameter sharing strategies to mitigate the representation bottleneck @xcite @xcite . meanwhile, there are also notorious cases of off-target translation especially in english-free pairs. previous works either resort to back-translation techniques to generate synthetic english-free data @xcite , or to model-level changes to the encoderdecoder alignments @xcite . in contrast, we propose a joint representation and gradient regularization approach to reduce off-target translations and significantly improve performance across all language pairs. multi-task learning for nmt multi-task learning (mtl) is a widely used technique to share model parameters and improve generalization @xcite . for nmt, previous works have leveraged mtl to inject linguistic knowledge or leveraged monolingual data @xcite @xcite . our work leverages an auxiliary tlp loss to help learn more separable model states for different target languages. optimization learning previous works have studied the optimization challenges in multi-task training @xcite , where @xcite proposed to resolve gradient conflicts between different tasks. meanwhile for nmt, @xcite proposed to mask out or assign different weights to training samples based on the gradient alignments with validation set. @xcite proposed to resolve the pair-wise gradient conflicts between translation directions. in contrast, we propose tgp to guide the training process by projecting training gradients according to the oracle gradients. ## conclusions in this work, we aimed to reduce the off-target translations with our proposed joint representation (tlp) and gradient (tgp) regularization to guide the internal modeling of target languages. our results showed both approaches to be highly effective at improving translation quality and to a large extent reduced the off-target occurrences. as a future direction, we will investigate off-target translations during decoding time @xcite .