LCS : A Language Converter Strategy for Zero-Shot Neural Machine Translation (2024)

Abstract: Multilingual neural machine translation models generally distinguish translation directions by the language tag (LT) in front of the source or target sentences. However, current LT strategies cannot indicate the desired target language as expected on zero-shot translation, i.e., the off-target issue. Our analysis reveals that the indication of the target language is sensitive to the placement of the target LT. For example, when placing the target LT on the decoder side, the indication would rapidly degrade along with decoding steps, while placing the target LT on the encoder side would lead to copying or paraphrasing the source input. To address the above issues, we propose a simple yet effective strategy named Language Converter Strategy (LCS). By introducing the target language embedding into the top encoder layers, LCS mitigates confusion in the encoder and ensures stable language indication for the decoder. Experimental results on MultiUN, TED, and OPUS-100 datasets demonstrate that LCS could significantly mitigate the off-target issue, with language accuracy up to 95.28%, 96.21%, and 85.35% meanwhile outperforming the vanilla LT strategy by 3.07, 3,3, and 7.93 BLEU scores on zero-shot translation, respectively.

## introduction multilingual neural machine translation (mnmt) aims to build a unified model to support the translation between any language pairs @xcite @xcite . one main challenge is the indication of the translation direction. pioneers @xcite @xcite utilize the language-specific encoder or decoder to distinguish source or target language. to simplify the architecture, @xcite and @xcite propose the language tag (lt) strategy, * work was done when zengkui sun was an intern at pattern recognition center, wechat ai, tencent inc, china. ## probing off-target issue in mnmt in this section, we probe the off-target issue in mnmt models with different lt strategies. firstly, we introduce the language rate/accuracy metric on this issue ( §3.1). secondly, we conduct experiments in terms of the distribution of this issue ( §3.2), the fine-grained language accuracy along decoding steps ( §3.3) and the language indication in the encoder ( §3.4). lastly, we expand our conclusion to propose lcs to mitigate this issue ( §3.5). ## analysis in this section, we explore the details of lcs to understand it better. we invest the generalizability of lcs to other approaches ( §5.1) and the effect of hyperparameter k ( §5.2), and the application of lcs to deeper encoders ( §5.3). ## conclusion in this paper, we have identified that the off-target issue is sensitive to the placement of lt and provide the analysis for the details of the issue with the prevalent lt strategies. we reveal that placing lt on the encoder side provides a more stable and sufficient language indication than the decoder side, and introducing the target information into the top layers of the encoder will mitigate the confusion between the source and the target language. based on our findings, we propose lcs to mitigate the offtarget issue, and further improve the performance of zero-shot translation. extensive experiments and analysis suggest that lcs boosts the performance of zero-shot translation and significantly mitigates the off-target issue without introducing extra parameters. besides, lcs could perform well in the noised data set.