ISTIC ’s Neural Machine Translation System for IWSLT ’2020 (2020)

Abstract: This paper introduces technical details of machine translation system of Institute of Scientific and Technical Information of China (ISTIC) for the 17th International Conference on Spoken Language Translation (IWSLT 2020). ISTIC participated in both translation tasks of the Open Domain Translation track: Japanese-to-Chinese MT task and Chinese-to-Japanese MT task. The paper mainly elaborates on the model framework, data preprocessing methods and decoding strategies adopted in our system. In addition, the system performance on the development set are given under different settings.

## introduction this paper describes the neural machine translation (nmt) system of the institute of scientific and technical information of china (istic) for the 17th international conference on spoken language translation (iwslt 2020) @xcite . istic participated in the japanese-to-chinese and chinese-to-japanese mt tasks of the open domain translation track. in this evaluation, we adopted the nmt google transformer @xcite architecture as a part of our system. we use the data released by the organizer and adopted general and specific preprocessing methods to the training and development data. several filtering methods of corpus are explored to improve the quality of the training data. a corpus filtering method based on elasticsearch is used to select the development data similar to test data. we adopted a model averaging strategy in the decoding phase and different results are combined in post-processing stage to obtain the final translation. the performance of the system is compared under different settings in the two translation directions, and further analyzed. ## system architecture figure 1 shows the flow chart of istic's nmt system in this evaluation. our model architecture, data processing and decoding strategy are given below. our baseline system used in this evaluation is the transformer @xcite based on a full attention mechanism, which includes an encoder and a decoder, as shown in figure 2 . transformer does not use a recurrent neural network @xcite or a convolutional neural network @xcite , but is completely based on attention mechanism. it can achieve algorithm parallelism, speed up model training, further alleviate long-distance dependence and improve translation quality. figure 2 . transformer model @xcite the encoder and decoder are formed by stacking n layer blocks. each layer of encoder contains two sub-modules, namely a multi-head self-attention module and a feed-forward neural network module. the multi-head self-attention module divides the dimension of hidden state into multiple parts， and each part is separately calculated by using self-attention function, furthermore, these output vectors are concatenated together. multi-head mechanism enables the model to pay more attention to the feature information of different positions and different sub-spaces. the multi-head attention method includes two steps: 1) dot product attention calculation; 2) multi-head attention calculation. the calculation method of dot product attention can be expressed as: each layer of the decoder is composed of three sub-modules. in addition to the two modules similar to the encoder, an decoder-encoder attention module is added between them and can focus attention on source language information in decoding process. in order to avoid the problem that too many layers cause the model to be difficult to converge, both the encoder and the decoder use residual connection and hierarchical regularization techniques. to make the model obtain the position information of the input sentence, additional position encoding vectors are added to the input layer of the encoder and decoder. after the encoder obtains a hidden state, transformer model inputs the hidden state into the softmax layer and scores with candidate vocabulary to obtain the final translation result. ## conclusions this paper introduces the main techniques and methods used by the institute of scientific and technical information of china on the task of two-directions translation of japanese and chinese in iwslt 2020 open domain translation track. we use the architecture of transformer model based on a full attention mechanism. several filtering methods were explored in data preprocessing, and the model average strategy is adopted in decoding. similar development set is chosen based on es and the results are merged by post-processing. experimental results show that these methods can effectively improve the quality of translation. due to limited time, many methods are not able to execute during this evaluation. our adopted translation model still has a lot room to improve. we expect to learn more advanced