Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization (2023)

Abstract: The multilingual neural machine translation (NMT) model has a promising capability of zero-shot translation, where it could directly translate between language pairs unseen during training. For good transfer performance from supervised directions to zero-shot directions, the multilingual NMT model is expected to learn universal representations across different languages. This paper introduces a cross-lingual consistency regularization, CrossConST, to bridge the representation gap among different languages and boost zero-shot translation performance. The theoretical analysis shows that CrossConST implicitly maximizes the probability distribution for zero-shot translation, and the experimental results on both low-resource and high-resource benchmarks show that CrossConST consistently improves the translation performance. The experimental analysis also proves that CrossConST could close the sentence representation gap and better align the representation space. Given the universality and simplicity of CrossConST, we believe it can serve as a strong baseline for future multilingual NMT research.

## introduction the objective of multilingual neural machine translation (nmt) is to construct a single, comprehensive model capable of translating between any pair of languages @xcite @xcite @xcite . this not only benefits low-resource translation @xcite , but also enables zeroshot translation @xcite . the success of zero-shot translation depends on the capability of the model to learn language-agnostic representations. the conventional multilingual nmt model @xcite , however, often struggles with learning the universal representations among different languages (figure 1 (a) ), which leads to poor zero-shot translation performance, particularly compared to the pivot-based methods @xcite . several methods have been proposed to improve the zero-shot translation performance by learning language-agnostic representations and maximizing cross-lingual transfer. some approaches modify the model architecture to achieve universal representations @xcite @xcite , while others utilize auxiliary training objectives to encourage similarity between the representations of different languages @xcite @xcite . specifically, @xcite introduce an agreementbased training approach to help the multilingual nmt model make consistent predictions based on the semantics-equivalent sentences. however, most existing methods are far from being widely used due to the degraded supervised translation performance, complicated algorithm implementation, and tedious hyperparameter search. in this paper, our primary goal is to provide a simple, easy-to-reproduce, yet effective strategy for learning multilingual nmt. inspired by @xcite , which boost the nmt performance by leveraging intra-lingual consistency regularization, we here propose a cross-lingual consistency regularization method, crossconst, to learn the universal representations across different languages (figure 1 (b) ) for boosting the zero-shot translation performance, where we introduce the explicit constraints to the semantic-equivalent sentence pairs by leveraging kullback-leibler (kl) regularization. the contributions of this paper can be summarized as follows: ## cross-lingual consistency for multilingual nmt in this section, we formally propose crossconst, a cross-lingual consistency regularization for learning multilingual nmt. we first review the multilingual neural machine translation (section 2.1), then introduce our method in detail (section 2.2). we theoretically analyze the regularization effect of crossconst (section 2.3) and propose a two-stage training strategy (section 2.4). ## low resource scenario we here investigate the performance of cross-const on the low-resource multilingual machine translation benchmark. for fair comparisons, we keep our experimental settings consistent with the previous work @xcite . ## high resource scenario we here investigate the performance of the cross-const on the high-resource multilingual machine translation benchmark. for fair comparisons, we keep our experimental settings consistent with the previous works @xcite . ## related work early works on multilingual nmt demonstrate its zero-shot translation capability @xcite . to further improve the zero-shot translation performance, one direction is to force the multilingual nmt encoder output to be language-agnostic via additional regularization constraints or training tasks @xcite @xcite @xcite @xcite . for example, @xcite introduce an agreement-based training approach to help the multilingual nmt model make consistent predictions based on the semantics-equivalent sentences. our method follows this line but outperforms these methods by introducing a simple yet effective cross-lingual regularization constraint, which effectively reduces discrepancies in representations across languages. another direction is to utilize extra data such as generated pseudo sentence pairs, monolingual datasets, and pretrained models @xcite @xcite @xcite . for example, @xcite encourages the multilingual nmt model to produce equivalent translations of parallel training sentence pairs into an auxiliary language. @xcite proposes random online back-translation to enforce the translation of unseen training language pairs. unlike these methods, crossconst does not require additional data and is orthogonal to these methods. we could further boost the zero-shot translation performance by combining our method with these data-driven approaches. ## conclusion in this paper, we propose crossconst: a simple but effective cross-lingual consistency regularization method for learning multilingual nmt models. we theoretically analyze the regularization effect of crossconst and verify its effectiveness for zero-shot translation. for the stable training of multilingual nmt, we propose a two-state training strategy that consists of multilingual nmt pre-training and crossconst finetuning. experiments on low and high resource multilingual translation benchmarks demonstrate crossconst's capabilities to improve translation performance in both supervised and zero-shot directions. further experimental analysis confirms that our method indeed leads to better cross-lingual representation alignment. given its universality and simplicity, we anticipate that researchers could leverage the simplicity of crossconst as a foundation to achieve new sota results in their own work. for future work, we will explore the effectiveness of cross-const on more multilingual tasks, such as multilingual sentence embedding, multilingual word alignment, etc.