R ussian- E nglish Bidirectional Machine Translation System (2020)

Abstract: This review depicts our submission to the WMT20 shared news translation task. WMT is the conference to assess the level of machine translation capabilities of organizations in the word. We participated in one language pair and two language directions, from Russian to English and from English to Russian. We used official training data, 102 million parallel corpora and 10 million monolingual corpora. Our baseline systems are Transformer models trained with the Sockeye sequence modeling toolkit, supplemented by bi-text data filtering schemes, back-translations, reordering and other related processing methods. The BLEU value of our translation result from Russian to English is 35.7, ranking 5th, while from English to Russian is 39.8, ranking 2th.

## introduction we participated in wmt20 shared news translation task by building neural translation systems for one language pair and two language directions, from english to russian and from russian to english. our systems are based on the framework of the transformer neural machine translation model, using many techniques and approaches, including the use of bpe subword segmentation for open-vocabulary translation with a fixed vocabulary, large-scale back-translation,and model ensembling. neural machine translation @xcite has emerged as the most promising machine translation approach in recent years, showing superior performance on public benchmarks. the proposed attention mechanism brought a new revolution in the neural machine translation in most cases, making the overall effect of translation much better than before. then, the transformer @xcite that makes full use of the attention mechanism demonstrated outstanding performance and effectiveness. up to now, most of work uses the structure of transformer, and its superiority has been widely recognized. since the beginning of machine translation research, the translation between russian and english has been extensively developed.as early as 1954, georgetown university in the united states under the ibm company completed the english-russian machine translation experiment with the ibm-701 computer, which opened the prelude of machine translation research. during the period, there are three core technologies, rule-based machine translation, statistical machine translation and neural machine translation. however, as the application field of machine translation became more and more complex, the limitations of various technologies started to become obvious. due to more application scenarios and higher requirements for accuracy, model optimization problems appeared. the translation between russian and english is extremely difficult because their linguistic features are distinguished and the lexical composition and grammatical structure of russian are more complicated than those of english. in the early period,statistical machine translations were hoped to be implemented through phrase-based methods @xcite and related techniques for language models and translation models. these methods have solved the russian-english translation problems to a certain extent. yet, at the same time, there exists translation problems that are high time cost and poor translation effect. since then, the emergence of neural machine translation has brought new developments to russian-english machine translation. the basic modeling framework for neural machine translation is an end-to-end sequence generation model, a framework and method for transforming input sequences into output sequences. there are two points in the core part. one is to represent the input sequence through the encoder, and the other is to obtain the output sequence through the decoder. in addition, for machine translation, neural machine translation not only includes encoding and decoding, but also uses rnn @xcite or other methods to encode sentence pairs. it also introduces an additional mechanism, the attention mechanism @xcite , to help us to convert sequences. these innovations lead to an increase in translation performance in comparison to earlier models. later, transformer appeared, which greatly enhances the neural machine translation performance. this paper is based on transformer, a neural machine translation network structure, to develop a two-way evaluation task between russian and english. taking into account the language characteristics of russian and english, we have done appropriate operations in data preprocessing, including removing duplicates, deleting unreasonable sentence pairs, lowercase and latinization operations, and judging sentence alignment problems, removing the parallel corpus with problems. the filtered parallel corpus is then sent to the model for training and the training results are tested.after getting the trained model, we start to consider using the back-translation operation to augment the data, continuing to filter the generated artificial corpus, and put it into the model training together with the original parallel corpus. finally, ensemble @xcite , average and rerank @xcite operations are implemented on different models to improve the overall performance of the translation system. ## background neural network machine translation is based on a sequence-to-sequence overall structure consisting of an encoder and a decoder. the encoder converts the source language sentence into an intermediate sequence result, and the decoder converts the intermediate sequence result into a target language sentence. there is also the attention mechanism to help make the results perform better. in the construction of the overall translation system, we used a lot of excellent methods proposed earlier in the literature. the basic model used here is transformer, introduced by @xcite . the transformer is an attention-based structure proposed to deal with tasks that require sequence models, such as machine translation. traditional neural machine translation mostly uses rnn or cnn as the model base of encoder-decoder, and google's latest attentionbased transformer model abandons the inherent formula and does not use any cnn or rnn structure. the model works in high-level parallel process, so training speed is also relatively fast while improving translation performance. but it is still computationally expensive. the structure of transformer is shown in figure 1 . the model is divided into two parts: the encoder and the decoder. the encoder is stacked by six identical layers, each with two more sublayers. the first sub-layer is a long self-attention mechanism, and the second sub-layer is a simple fully connected feed forward network. a residual connection is added outside the two layers, and then layer normalization is performed. the output dimensions of all sub-layers and embedding layers of the model are d models ; the decoder also stacks six identical layers. however, in addition to the two layers in the encoder, the decoder also adds a third sub-layer, as shown in the figure which also uses the residual and layer normalization. ## data we use all available bitext data which provided by wmt for the russian-english language pair. for the monolingual data we use english and russian newscrawl as well as a filtered part of commoncrawl in russian. we choose to use russian commoncrawl to augment our monolingual data due to the relatively small size of russian newscrawl compared to english. ## experiment for this evaluation task, we first start from the data preprocessing, through data expansion operations to obtain the data that needs to be trained, and then input the transformer model for training. we test the training results and finally ensemble results according to the model generated by different strategies, average and rerank operations,for the best results. next, the specific experiment content will be presented separately. the overall project process is showed in figure 2 . ## results results and ablations from russian to english are shown in ## conclusions this paper describes our submission to the wmt20 news translation task. in the evaluation task, we es-tablished a russian-english bidirectional machine translation system based on transformer. for translations between russian and english, we use the same strategy of filtering bitext data, performing beam-search back-translation on monolingual data. then we train strong individual models on a combination of this data. each of these models is finetuned and ensembled into a final system that is used for decoding with model reranking. in the final list, we got 2th in ru-en, and 5th in en-ru. good results have been obtained in limited time and hardware resources, which is also in line with the industry's demands for service construction. in the whole experiment process, we also gained a lot of experience in data processing and experimental design, which will be of great help in later research and study. we will continue to improve the previous experiments, strive to get better results, and see what rankings can eventually be achieved, in preparation for the next year.