TJUNLP :System Description for the WMT 23 Literary Task in C hinese to E nglish Translation Direction (2023)

Abstract: This paper introduces the overall situation of the Natural Language Processing Laboratory of Tianjin University participating in the WMT23 machine translation evaluation task from Chinese to English. For this evaluation, the base model used is a Transformer based on a Mixture of Experts (MOE) model. During the model’s construction and training, a basic dense model based on Transformer is first trained on the training set. Then, this model is used to initialize the MOE-based translation model, which is further trained on the training corpus. Since the training dataset provided for this translation task is relatively small, to better utilize sparse models to enhance translation, we employed a data augmentation technique for alignment. Experimental results show that this method can effectively improve neural machine translation performance.

## introduction machine translation, as a core branch of natural language processing, has experienced significant development and received widespread attention in the past few years. propelled by deep learning and neural networks, architectures like the transformer @xcite and its derivative models, such as bert @xcite and gpt @xcite , have become mainstream methods for achieving efficient machine translation. these models, by learning underlying representations of language, are able to capture complex relationships and rich semantic information between texts. although neural machine translation with dense models has a promising future, it still faces many challenges. one of the main issues with the standard transformer-based dense multilingual neural machine translation model is the model's capacity bottleneck @xcite @xcite . while increasing the model's depth and breadth can effectively enhance its capacity, it severely reduces the model's execution efficiency and increases the hardware requirements for training the model. this often results in the need for large gpu devices, limiting the model's applications. therefore, in recent years, multilingual neural machine translation based on mixtureof-experts (moe) @xcite has been proposed. compared to dense models, moe-based multilingual machine translation activates only a portion of the network parameters during model training and inference @xcite , giving it excellent computational efficiency. under the same hardware conditions, it can achieve greater model capacity (compared to dense models, capacity can be increased by several tens of times) @xcite and shorter computation time. therefore, in this translation task evaluation, our basic model framework is based on the moe transformer. furthermore, when there is limited available data, overfitting can easily occur @xcite . combining the knowledge of multiple experts can often provide more accurate predictions than a single model. during model training, by allocating experts to focus on different input subsets, moe can help alleviate the overfitting issue @xcite . in this paper, we primarily focus on the wmt23 chinese to english machine translation task. to enhance the model's capacity while maintaining a high computational efficiency, we employ a neural machine translation model based on the moe transformer framework. this model can effectively expand the model parameters. moreover, since it's a domain-specific translation task with limited translation data corpus, we employed a strategy to initialize moe using dense models effectively. the rest of this paper is organized as follows. in section 2, we will present the models and methods we designed. section 3 primarily showcases the experimental results and discusses and analyzes the outcomes. section 4 concludes the paper and provides an outlook. ## method description to evaluate machine translation from chinese to english, we need to construct a machine translation model. therefore, in section 2.1, we first introduce the model's design and initialization strategy. in section 2.2, we primarily discuss the data alignment augmentation method, aiming to further utilize the data to enhance the model's performance. finally, we introduce the model's training strategy. ## experimental results and analysis we first introduce the parameter settings of our trained models, and then analyze the experimental results. ## conclusion this paper introduces the main techniques and methods used for the wmt23 chinese to english neural machine translation evaluation task. we employ a multilingual neural machine translation model based on the moe transformer framework. this model effectively achieves a vast and efficient parameterization. moreover, given that it's a domain-specific translation task with limited translation data corpus, we utilized an effective strategy of initializing the moe model using a dense model. this ensures that the moe model has already grasped the fundamental features of the data at the start of training, providing a stable foundation for the moe model and reducing the risks of instability and overfitting during training. experimental results demonstrate that these methods can significantly improve the translation quality of neural machine translation.