Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Features (2024)

Abstract: The many-to-many multilingual neural machine translation can be regarded as the process of integrating semantic features from the source sentences and linguistic features from the target sentences. To enhance zero-shot translation, models need to share knowledge across languages, which can be achieved through auxiliary tasks for learning a universal representation or cross-lingual mapping. To this end, we propose to exploit both semantic and linguistic features between multiple languages to enhance multilingual translation. On the encoder side, we introduce a disentangling learning task that aligns encoder representations by disentangling semantic and linguistic features, thus facilitating knowledge transfer while preserving complete information. On the decoder side, we leverage a linguistic encoder to integrate low-level linguistic features to assist in the target language generation. Experimental results on multilingual datasets demonstrate significant improvement in zero-shot translation compared to the baseline system, while maintaining performance in supervised translation. Further analysis validates the effectiveness of our method in leveraging both semantic and linguistic features.

## introduction the many-to-many multilingual neural machine translation (nmt) enables translation across multiple languages in a single model @xcite @xcite . this process involves integrating the semantic information from the source sentence and the linguistic features from the target sentence. specifically, the encoder captures the semantic information of the source sentence and maps it to the representation space, when the decoder integrates this representation with the inherent features of the target language to generate the target language sentences. by sharing parameters, multilingual nmt models enable knowledge transfer across languages, which significantly benefits low-resource machine translation @xcite , especially zero-shot translation. however, previous multilingual nmt models do not explicitly differentiate between semantic and linguistic features, resulting in the entanglement of knowledge and linguistics within the model. this entanglement negatively affects the performance of zero-shot translation in two main aspects. firstly, the linguistic features of the source language interfere with the encoder's ability to learn shared semantic information, causing the encoder to encode different languages into different representation subspaces @xcite , which hinders knowledge transfer across multiple languages. secondly, during the generation of the target language, the decoder is influenced by spurious correlations due to insufficient target language features, leading to off-target issues @xcite . numerous methods have been proposed to address these challenges. some methods focus on learning a universal cross-lingual representation space at the encoder @xcite @xcite , while others aim to facilitate target language generation at the decoder @xcite . for example, @xcite employ contrastive learning to minimize the distance between parallel sentence pairs and maximize the distance between irrelevant sentence pairs. @xcite leverage optimal transport theory to learn a universal representation and introduce an agreement-based training approach to make consistent predictions. @xcite promote consistent cross-lingual representation by cross-lingual consistency regularization. however, these methods primarily concentrate on learning 1 10410 language-agnostic semantic features while neglecting the language-specific linguistic features. such incompleteness of information can lead to less accurate semantic mapping and alignment degradation, which can harm the supervised and zero-shot translation performance. in this paper, we propose to utilize both semantic and linguistic features for multilingual nmt. our insight is to learn a semantic representation space non-destructively and utilize linguistic features to guide target language generation, thereby improving the performance of zero-shot translation and maintaining supervised translation performance lossless. specifically, at the encoder, we introduce a disentangler and disentangling learning task to facilitate harmless cross-lingual semantic alignment. at the decoder, we utilize a linguistic encoder to integrate low-level linguistic features from lower layers and high-level semantic features from higher layers. this fusion provides implicit guidance for the target language generation. experiments on multilingual datasets show that our method brings an average of 0.18+ bleu in the supervised translation direction and an average of 3.74+ bleu in the zero-shot translation direction compared with the baseline system. furthermore, the analysis demonstrates that our method can alleviate the problems caused by the entanglement of semantic and linguistic features, obtain a better semantic representation, and reduce the off-target rate in zero-shot translation. ## background in this section, we will give a brief introduction to the transformer @xcite model for nmt and many-to-many multilingual nmt. ## method the core idea of our approach is to improve multilingual nmt by exploiting semantic and linguistic features at the same time. to achieve this, we disentangle semantic and linguistic features for the encoder and utilize low-level linguistic features for the decoder. specifically, we propose a disentangler at the encoder to facilitate lossless semantic alignment. meanwhile, we introduce a linguistic encoder at the decoder to fuse low-level features to guide target language generation. figure 1 illustrates the framework of our approach. ## experiment we evaluate the performance of our model on the multilingual benchmarks, including supervised and zero-shot translation performance. ## analysis in this section, we will analyze what contributes to the performance. we first analyze the effects of each module via an ablation study. then, we evaluate the off-target rate and visualize the encoder and decoder sentence representations, demonstrating that our approach learns universal representations and generates target language more accurately. finally, we examine some translation cases to show the usefulness of our approach. ## conclusion in this paper, we propose to exploit both semantic and linguistic features to improve multilingual nmt. we design a disentangler and disentangling learning task at the encoder side to achieve lossless semantic alignment. meanwhile, we introduce a linguistic encoder at the decoder side to fuse the low-level representation containing more linguistic features and the high-level representation containing more semantic features to improve the target language generation. experiments on multilingual datasets indicate that our approach can significantly improve the performance of zero-shot translation while effectively maintaining the supervised translation performance. the analysis further demonstrates that utilizing both semantic and linguistic features can help the model learn semantic representation non-destructively and achieve more accurate target language generation. 9 10418