W e TS : A Benchmark for Translation Suggestion (2022)

Abstract: Translation suggestion (TS), which provides alternatives for specific words or phrases given the entire documents generated by machine translation (MT), has been proven to play a significant role in post-editing (PE). There are two main pitfalls for existing researches in this line. First, most conventional works only focus on the overall performance of PE but ignore the exact performance of TS, which makes the progress of PE sluggish and less explainable; Second, as no publicly available golden dataset exists to support in-depth research for TS, almost all of the previous works conduct experiments on their in-house datasets or the noisy datasets built automatically, which makes their experiments hard to be reproduced and compared. To break these limitations mentioned above and spur the research in TS, we create a benchmark dataset, called WeTS , which is a golden corpus annotated by expert translators on four translation directions. Apart from the golden corpus, we also propose several methods to generate synthetic corpora which can be used to improve the performance substantially through pre-training. As for the model, we propose the segment-aware self-attention based Transformer for TS. Experimental results show that our approach achieves the best results on all four directions, including English-to-German, German-to-English, Chinese-to-English, and English-to-Chinese.

## introduction computer-aided translation (cat) @xcite @xcite has attained more and more attention for its promising ability in combining the high efficiency of machine translation (mt) @xcite @xcite and high accuracy of human translation (ht). a typical way for cat tools to combine mt and ht is pe @xcite , where the human translators are asked to provide alternatives for the incorrect word spans in the results generated by mt. to further reduce the post-editing time, researchers propose to apply ts into pe, where ts provides the sub-segment suggestions for the annotated incorrect word spans in the results of mt, and their extensive experiments show that ts can substantially reduce translators' cognitive loads and the post-editing time @xcite . as there is no explicit and formal definition for ts, we observe that some previous works similar or related to ts have been proposed @xcite @xcite . however, there are two main pitfalls for these works in this line. first, most conventional works only focus on the overall performance of pe but ignore the exact performance of ts. this is mainly because the golden corpus for ts is relatively hard to collect. as ts is an important sub-module in pe, paying more attention to the exact performance of ts can boost the performance and interpretability of pe. second, almost all of the previous works conduct experiments on their in-house datasets or the noisy datasets built automatically, which makes their experiments hard to be followed and compared. additionally, experimental results on the noisy datasets may not truly reflect the model's ability on generating the right predictions, making the research deviate from the correct direction. therefore, the community is in dire need of a benchmark for ts to enhance the research in this area. to address the limitations mentioned above and spur the research in ts, we make our efforts to construct a high-quality benchmark dataset with human annotation, named wets, 1 which covers four different translation directions. as collecting the golden dataset is expensive and labor-consuming, we further propose several methods to automatically construct synthetic corpora, which can be utilized to improve the ts performance through pre-training. as for the model, we for the first time propose the segment-aware self-attention based transformer for ts, named sa-transformer, which achieves superior performance to the naive transformer @xcite . the main contributions of this paper are summarized as follows: ## construct synthetic corpus since constructing the golden corpus is expensive and labor-consuming, automatically building the synthetic corpus is very promising. in this section, we describe several ways for constructing synthetic corpus for ts based on the parallel corpus of mt. ## experiments and results we first describe the experimental settings, including datasets, pre-processing, and hyper-parameters; then we introduce the baseline systems and report the main experimental results. ## analysis this section provides detailed analysis about the proposed approach, and performance on two translation directions are presented for most of the following experiments. ## conclusion and future work in this paper, we propose a benchmark for the task of translation suggestion. we construct and share a golden dataset, named wets, for the community, and propose several ways for automatically constructing the synthetic corpora which can be used to improve the performance substantially. additionally, we for the first time propose the segmentaware self-attention based transformer, named sa-transformer, which achieves the best performance on all four translation directions. we hope our work will provide a new perspective and spur future researches on ts. there are two promising directions for the future work. first, we decide to introduce new techniques from recommendation systems to generate more diverse and accurate suggestions. second, modeling the interactions between the source and translation sentences implicitly may be helpful to improve the performance.