TSM ind: A libaba and Soochow University’s Submission to the WMT 22 Translation Suggestion Task (2022)

Abstract: This paper describes the joint submission of Alibaba and Soochow University to the WMT 2022 Shared Task on Translation Suggestion (TS). We participate in the English to/from German and English to/from Chinese tasks. Basically, we utilize the model paradigm fine-tuning on the downstream tasks based on large-scale pre-trained models, which has recently achieved great success. We choose FAIR’s WMT19 English to/from German news translation system and MBART50 for English to/from Chinese as our pre-trained models. Considering the task’s condition of limited use of training data, we follow the data augmentation strategies provided by Yang to boost our TS model performance. And we further involve the dual conditional cross-entropy model and GPT-2 language model to filter augmented data. The leader board finally shows that our submissions are ranked first in three of four language directions in the Naive TS task of the WMT22 Translation Suggestion task.

## introduction computer-aided translation (cat) @xcite @xcite has become more and more popular to help increase the quality of machine translation @xcite result. it also improves the efficiency of translators by combining the results of machine translation and the content edited by translators in the process of translation or post-editing @xcite @xcite @xcite . post-editing based on machine translation is typical in cat. recent works @xcite @xcite propose interactive protocols and algorithms so that humans and machines can collaborate during translation, and machines can automatically provide feedback on humans' edits. one interesting mode is translation suggestion (ts) @xcite , which offers alternatives for specific spans of words in the generated machine translation. it will be convenient if the model refines translation results in those specified locations with potential translation errors. @xcite released a benchmark dataset for ts, wets, which is one of the shared tasks in wmt22. at the same time, they proposed an end-to-end transformer-like model for ts as the benchmark system. however, the lack of many labeled ts data limits the training of a large transformer model to some extent. though @xcite have tried to utilize xlm-roberta @xcite to initialize the encoder of the transformer, the decoder has to be trained from scratch, which leads to relatively low bleu scores for some specific ts spans. we investigate the potential of other encoder-decoder pre-trained models by experiments to see if there is still room for improvement. finally, we have found that pre-trained transformer nmt models could be suitable choices to be fine-tuned with the limited size of ts data. in addition, we applied similar data augmentation strategies proposed in @xcite , but use the well-trained alignment models between source and target languages from @xcite to filter out high-quality augmented data. our submissions are ranked first in three of four language directions in the wmt22 translation suggestion task. ## the model we train a simple end-to-end transformer model for each language pair to generate the translation suggestion candidates. the source sentence and the masked translation, in which an incorrect span requiring an alternative has been replaced with a special mask tokens in advance, are concatenated with a special separation token @xcite . afterward, ## data augmentation we follow the data augmentation methods provided by @xcite to generate three types of pseudo data for ts model training: data sampled on the golden parallel corpus, data sampled on the pseudo parallel corpus, and data extracted with word alignment. however, the details of the pseudo data augmentation in this paper are slightly different from those of @xcite . full details are exhibited in the following subsections. algorithm 1 algorithm of phrase align input: y, r, a output: p 1 function generatealign(y, r, a): raw parallel corpus is firstly filtered by the sentence length. all sentence pairs that have less than 20 words or more than 80 words on any side are removed. considering that there might be noise data in the corpus, we apply the dual conditional cross-entropy model @xcite to obtain a quality score for each sample. sentence pairs with low quality are filtered. then we generate a pseudo corpus with the remained high-quality parallel corpus. (x, r) is marked as the sentence pair of the parallel corpus, where x is the source sentence and r is the golden reference. ∥r∥ represents the number of tokens in r. the first step is to randomly sample the length l to mask for the reference r from a uniform distribution: then a span with l tokens r i:j is randomly selected by: finally, we get the ts training data (x, r ¬i:j , r i:j ) from each parallel sentence pair (x, r), where r ¬i:j is denoted as the masked version of r, in which r i:j is replaced with a mask token, e.@xmath0. ## conclusion in this paper, we present our translation suggestion systems, tsmind, for the wmt 2022 translation suggestion task. different from previous work, we use well-trained nmt models as the pre-trained models and applied a two-phase training strategy. we explore three data augmentation strategies from previous work and utilize the dual conditional cross-entropy model to filter out low-quality augmented data. the leader board finally shows that our submissions are ranked first in three of four language directions in the naive ts task of wmt22 translation suggestion task.