On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation (2023)

Abstract: While multilingual neural machine translation has achieved great success, it suffers from the off-target issue, where the translation is in the wrong language. This problem is more pronounced on zero-shot translation tasks. In this work, we find that failing in encoding discriminative target language signal will lead to off-target and a closer lexical distance (i.e., KL-divergence) between two languages’ vocabularies is related with a higher off-target rate. We also find that solely isolating the vocab of different languages in the decoder can alleviate the problem. Motivated by the findings, we propose Language Aware Vocabulary Sharing (LAVS), a simple and effective algorithm to construct the multilingual vocabulary, that greatly alleviates the off-target problem of the translation model by increasing the KL-divergence between languages. We conduct experiments on a multilingual machine translation benchmark in 11 languages. Experiments show that the off-target rate for 90 translation tasks is reduced from 29% to 8%, while the overall BLEU score is improved by an average of 1.9 points without extra training cost or sacrificing the supervised directions’ performance. We release the code at https://github.com/PKUnlp-icler/Off-Target-MNMT for reproduction.

## introduction multilingual nmt makes it possible to do the translation among multiple languages using only one model, even for zero-shot directions @xcite . it has been gaining increasing attention since it can greatly reduce the mt system's deployment cost and enable knowledge transfer among different translation tasks, which is especially beneficial for lowresource languages. despite its success, off-target is a harsh and widespread problem during zeroshot translation in existing multilingual models. for the zero-shot translation directions, the model translates the source sentence to a wrong language, which severely degrades the system's credibility. as shown in researchers have been noticing and working on solving the problem from different perspectives. for model trained on english-centric dataset, a straight forward method is to add pseudo training data on the zero-shot directions through backtranslation @xcite . adding pseudo data is effective since it directly turns zero-shot translation into a weakly supervised task. despite its effectiveness, it brings a lot more training cost during generating data and training on the augmented corpus and the supervised directions' performance is also reported to decrease due to the model capacity bottleneck @xcite . @xcite finds that instead of regarding all languages as one during the vocabulary building process, language-specific bpe can alleviate the off-target problem, yet it still costs the supervised directions' performance. in this work, we perform a comprehensive analysis of the off-target problem, finding that failure in encoding discriminative target language signal will lead to off-target and we also find a strong correlation between off-target rate of certain direction and the lexical similarity between the involved languages. a simple solution by separating the vocabulary of different languages in the decoder can decrease lexical similarity among languages and it proves to improve the zero-shot translation performance. however, it also greatly increases the model size (308m->515m) because a much larger embedding matrix is applied to the decoder. for a better performance-cost trade-off, we further propose language-aware vocabulary sharing (lavs), a novel algorithm to construct the multilingual vocabulary that increases the kl-divergence of token distributions among languages by splitting particular tokens into language-specific ones. lavs is simple and effective. it does not introduce any extra training cost and maintains the supervised performance. our empirical experiments prove that lavs reduces the off-target rate from 29% to 8% and improves the bleu score by 1.9 points on the average of 90 translation directions. together with back-translation, the performance can be further improved. lavs is also effective on larger dataset with more languages such as opus-100 @xcite and we also observe that it can greatly improve the english-to-many performance (+0.9 bleu) in the large-scale setting. ## related work off-target problem in zero-shot translation without parallel training data for zero-shot directions, the mnmt model is easily caught up in off-target problem @xcite @xcite @xcite @xcite where it ignores the target language signal and translates to a wrong language. several methods are proposed to eliminate the off-target problem. zhang et al. (2020); gu et al. (2019) resort different back-translation techniques to generate data for non-english directions. back-translation method is straight-forward and effective since it provides pseudo data on the zero-shot directions but it brings a lot more additional cost during generating data and training on the augmented corpus. @xcite introduced decoder pretraining to prevent the model from capturing spurious correlations, @xcite explored how language tag settings influence zero-shot translation. however, the cause for off-target still remains underexplored. ## conclusion in this paper, we delve into the hidden reason for the off-target problem in zero-shot multilingual nmt and propose language-aware vocabulary sharing (lavs) which could significantly alleviate the off-target problem without extra parameters. our experiments justify that lavs creates a better multilingual vocab than the original vocabulary sharing method for multiple languages.