Findings of the WMT 2022 Shared Task on Translation Suggestion (2022)

Abstract: We report the result of the first edition of the WMT shared task on Translation Suggestion (TS). The task aims to provide alternatives for specific words or phrases given the entire documents generated by machine translation (MT). It consists two sub-tasks, namely, the naive translation suggestion and translation suggestion with hints. The main difference is that some hints are provided in sub-task two, therefore, it is easier for the model to generate more accurate suggestions. For sub-task one, we provide the corpus for the language pairs English-German and English-Chinese. And only English-Chinese corpus is provided for the sub-task two. We received 92 submissions from 5 participating teams in sub-task one and 6 submissions for the sub-task 2, most of them covering all of the translation directions. We used the automatic metric BLEU for evaluating the performance of each submission.

## introduction computer-aided translation (cat) @xcite @xcite has attained more and more attention for its promising ability in combining the high efficiency of machine translation (mt) @xcite @xcite and high accuracy of human translation (ht). a typical way for cat tools to combine mt and ht is pe @xcite , where the human translators are asked to provide alternatives for the incorrect word spans in the results generated by mt. to further reduce the post-editing time, researchers propose to apply ts into pe, where ts provides the sub-segment suggestions for the annotated incorrect word spans in the results of mt, and their extensive experiments show that ts can substantially reduce translators' cognitive loads and the post-editing time @xcite . as there is no explicit and formal definition for ts, we observe that some previous works similar or related to ts have been proposed @xcite @xcite . however, there are two main pitfalls for these works in this line. first, most conventional works only focus on the overall performance of pe but ignore the exact performance of ts. this is mainly because the golden corpus for ts is relatively hard to collect. as ts is an important sub-module in pe, paying more attention to the exact performance of ts can boost the performance and interpretability of pe. second, almost all of the previous works conduct experiments on their in-house datasets or the noisy datasets built automatically, which makes their experiments hard to be followed and compared. additionally, experimental results on the noisy datasets may not truly reflect the model's ability on generating the right predictions, making the research deviate from the correct direction. therefore, the community is in dire need of a benchmark for ts to enhance the research in this area. to address the limitations mentioned above and spur the research in ts, we make our efforts to construct a high-quality benchmark dataset with human annotation, named wets, foot_0 which covers four different translation directions. the main motivation of this shared task is twofold. the first goal is to analyze the challenges in the area of ts, which can provide some new directions for the further researches and applications in this area. secondly, we want to make the researchers notice the gaps between the golden and automatically generated synthetic corpus. and we want to see the performance of different techniques on the golden corpus. as the source and translation sentence are both the inputs of ts, it is interesting to see how the interactions between the source and translation sentences can improve the final suggestions. in order to evaluate the quality of the participating systems, we use the automatic metric, bleu @xcite . specifically, we adopt the widely used toolkit, sacrebleu @xcite to calculate the bleu score for the top-1 suggestion against the reference sentences. 2 for chinese, the bleu score is calculated on teh character with the default tokenizer for chinese. as for english, the bleu score is calcualted on the case-sensitive words with the default tokenizer 13a. five teams participated in this first campaign of the translation suggestion shred task, most of them cover the four translation directions. we will describe each system which submits the technical paper in detail. ## task description this section describes the task definition in the first edition of ts shared task. we finely divide the task of ts into two sub-tasks, namely vanilla ts and ts with hints, according to whether the translators' hints are considered. ## data description this section introduces the proposed dataset wets used in the shred task, which is a golden corpus for four translation directions, including english-to-german, german-to-english, chinese-to-english and english-to-chinese. ## participants five participants submitted their systems to the subtask one of ts shared task. and two participants submitted their systems to the second sub-task. in sub-task one, 92 runs were submitted in total (each team is only allowed to submit less than 15 runs). ## discussion and analysis comparing the results of the bleu scores of all submissions with our baseline systems, there is a significant gap between the submitted and baseline systems. this shows that there is a large space for us to try different techniques to improve the performance of ts. by comparing the results of different submitted systems, we find that different pre-training models have a large difference on the final performance. this is a similar trend with other nlp tasks. therefore, we believe that this is an interesting and promising direction for us to pay much more efforts. all submitted systems have investigated different approaches for constructing the synthetic corpus and almost all of them have achieved much improvements with the synthetic corpus. the noise in the synthetic corpus is a major problem which negatively affects the final performance. therefore, how to filter or decrease the noise is an open question. the team of mind-ts applies the pre-trained lm to filter the synthetic corpus and obtain better performance on 3 out of 4 tracks based on the highquality synthetic corpus. we can investigate more effective approaches to detect and filter the noise in the synthetic corpus. however, another interesting direction which are not investigated by the submissions is modeling the interaction between the source and translation sentences efficiently. compared to mt, the main difference for ts is that the input for ts is dualsource, namely the source and translation sentence. we believe that efficiently modeling the interaction between the source and translation sentences can improve the final performance. ## conclusion we present the results of first edition of the translation suggestion shared task. for the goal of this task, we create and release the first golden benchmark dataset, called wets, which covers the language pairs for english-chinese and english-german. we wish the released corpus can spur the researches in this area. this year we received 92 submissions from 5 participating teaming in the sub-task one and 6 submissions for the sub-task 2, most of them covering the two translation directions. results of these submissions show that the pre-trained models and synthetic corpus are two important factors for the final performance.