Improving Zero-Shot Multilingual Translation with Universal Representations and Cross-Mapping (2022)

Abstract: The many-to-many multilingual neural machine translation can translate between language pairs unseen during training, i.e., zero-shot translation. Improving zero-shot translation requires the model to learn universal representations and cross-mapping relationships to transfer the knowledge learned on the supervised directions to the zero-shot directions. In this work, we propose the state mover’s distance based on the optimal theory to model the difference of the representations output by the encoder. Then, we bridge the gap between the semantic-equivalent representations of different languages at the token level by minimizing the proposed distance to learn universal representations. Besides, we propose an agreement-based training scheme, which can help the model make consistent predictions based on the semantic-equivalent sentences to learn universal cross-mapping relationships for all translation directions. The experimental results on diverse multilingual datasets show that our method can improve consistently compared with the baseline system and other contrast methods. The analysis proves that our method can better align the semantic space and improve the prediction consistency.

## introduction the many-to-many multilingual neural machine translation (nmt) @xcite @xcite @xcite ) model can support multiple translation directions in a single model. the shared encoder encodes the input sentence to the semantic space, and then the shared decoder decodes from the space to generate the translation of the target language. this paradigm allows the model to translate between language pairs unseen during training, i.e., zero-shot translation. zero-shot translation can improve the inference efficiency and make the model require less bilingual training data. performing zero-shot translation requires universal representations to encode the language-agnostic features and cross-mapping relationships that can map the semantic-equivalent sentences of different languages to the particular space of the target language. in this way, the model can transfer the knowledge learned in the supervised translation directions to the zero-shot translation directions. however, the existing model structure and training scheme cannot ensure the universal representations and cross-mappings because of lacking explicit constraints. specifically, the encoder may map different languages to different semantic subspaces, and the decoder may learn different mapping relationships for different source languages, especially when the model possesses high capacity. many researchers have made their attempts to solve this problem. @xcite propose to compress the output of the encoder into a consistent number of states to only encode the languageindependent features. @xcite add a regularizing loss to maximize the similarities between the sentence representations of the source and target sentences. @xcite propose contrastive learning schemes to minimize the sentence representation gap of similar sentences and maximize that of irrelevant sentences. all the above work tries to minimize the representation discrepancies of different languages at the sentence level, bringing two problems for nmt. firstly, these work usually get the sentence-level representation of the encoder output by max-pooling or averaging, which may potentially ignore the sentence length, word alignment relationship, and other token-level information. secondly, regularizing sentence representation mismatches to the working paradigm of the nmt model, because the decoder directly performs cross attention on the whole state sequences rather than the sentence representation. besides, all the above work focuses on the encoder side and cannot help learn the universal mapping relationship for the decoder. given the above, we propose a method to learn the universal representations and cross-mappings to improve the zero-shot translation performance. based on the optimal transport theory, we propose state mover's distance (smd) to model the differences of two state sequences at the token level. to map the semantic-equivalent sentences from different languages to the same place of the semantic space, we add an auxiliary loss to minimize the smd of the source and target sentences. besides, we propose an agreement-based training scheme to learn universal mapping relationships for the translation directions with the same target language. we mixup the source and target sentences to obtain a pseudo sentence. then, the decoder makes predictions separately conditioned on this pseudo sentence and the corresponding source or target sentences. we try to improve the prediction consistency by minimizing the kl divergence of the two output distributions. the experimental results on diverse multilingual datasets show that our method can bring 2~3 bleu improvements over the strong baseline system and consistently outperform other contrast methods. the analysis proves that our method can better align the semantic space and improve the prediction consistency. ## background in this section, we will give a brief introduction to the transformer @xcite model and the many-to-many multilingual translation. ## analysis in this section, we try to understand how our method improves the zero-shot translation. ## conclusion in this work, we focus on improving the zero-shot ability of multilingual neural machine translation. to reduce the discrepancy of the encoder output, we propose the state mover's distance based on the optimal transport theory and directly minimize the distance during training. we also propose an agreement-based training method to help the decoder make consistent predictions based on the semantic-equivalent sentences. the experimental results show that our method can get consistent im-provements on diverse multilingual datasets. further analysis shows that our method can better align the semantic space, improve the prediction consistency, and reduce the spurious correlations.