Findings of the Word-Level A uto C ompletion Shared Task in WMT 2022 (2022)

Abstract: Recent years have witnessed rapid advancements in machine translation, but the state-of-the-art machine translation system still can not satisfy the high requirements in some rigorous translation scenarios. Computer-aided translation (CAT) provides a promising solution to yield a high-quality translation with a guarantee. Unfortunately, due to the lack of popular benchmarks, the research on CAT is not well developed compared with machine translation. In this year, we hold a new shared task called Word-level AutoCompletion (WLAC) for CAT in WMT. Specifically, we introduce some resources to train a WLAC model, and particularly we collect data from CAT systems as a part of test data for this shared task. In addition, we employ both automatic and human evaluations to measure the performance of the submitted systems, and our final evaluation results reveal some findings for the WLAC task.

## introduction in past decades, the machine translation community has witnessed a significant evolution from statistical machine translation @xcite @xcite to neural machine translation (nmt) @xcite @xcite @xcite . nmt has achieved a rapid and tremendous advancement in translation performance @xcite . despite its success in many real-world applications, its translation quality still can not satisfy the high requirements in some scenarios. in such rigorous scenarios, one promising approach is to leverage machines to assist human translation, such as computer-aided translation (cat) @xcite @xcite @xcite @xcite . * the authors are listed alphabetically. however, the development in cat is much slower than in machine translation. for example, there are hundreds of research papers on machine translation in natural language processing conferences each year, whereas only a few papers on cat are published. one of the main reasons is that there are few popular benchmarks or shared tasks for cat research, which enable researchers to make continuous progress in this area. consequently, in wmt this year, we hold a new shared task, wordlevel autocompletion (wlac), to facilitate the research in cat. generally, wlac aims to autocomplete a word when a human translator types a sequence of characters @xcite . as a basic functionality in many cat systems, wlac is used to accelerate the editing process for human translators and it plays an important role in cat. in this paper, we describe the overview for the shared task of wlac in wmt 2022, such as task description, datasets, participants and their evaluations. the shared task involves two language pairs, including chinese-english and german-english and it contains four subtasks corresponding to all four directional pairs. for data preparation, since it is too costly to collect realistic data with a considerable scale to train wlac models, we follow the standard practice to construct the training data from a bilingual corpus by simulation. moreover, to make the testing stage resemble the realistic scenario in cat, we collect some data from two cat systems as a part of test data. in this shared task, we receive 27 submissions in total for all subtasks from five participants which are quickly summarized in this paper. to evaluate the submissions, we particularly conduct human evaluation in addition to automatic evaluation. after evaluation, we finally obtain some findings from the submission results, which we hope may inspire future advancements for the wlac task. r t b l 0 i z q q p o s m g i v 5 p b t r m w z c j e 1 7 q l e u u f v 3 b r j k v a w p w a l 1 x v m r 2 4 9 z g q e 0 t f c t o 5 6 b g d + j y n a m c f r q p h o t y k z 0 g b 1 l j y 1 q + 9 k s 9 j s c w a v p w l j z j w 2 z q b 8 3 m h p p p y k c o 5 l n 1 m t e l v 7 n d v i t 3 v g z l 0 l q u l l 5 o t a v x m q k b 4 d 0 u u j m x m q s y h s 3 w q k b u k w z s t 2 v b a n e 8 p d x s f o i 6 l 1 w r x 4 u k 7 x b r r 1 f o i f t o a c p r q e g 9 1 c h b j b 4 g m d 4 h t d n 7 l w 4 7 8 7 h f l t g l h a o 4 q + c z x / g y 5 i @xmath0 w h m u s j r p b j a k z p c x p 0 m 6 p r m m l n l x 5 q e e l z h i 5 4 z 1 j f i 2 7 8 b b 5 5 r s 6 s m i r h r o 1 t s o b q 7 4 2 m r s z m o 8 b o 5 g n n s p e l / 3 m 9 f m m b p 2 task description and data preparation 2.1 task description suppose x is a source sequence, s is a sequence of human typed characters and @xmath1, c r ) is a translation context. the translation pieces c l and c r are on the left and right hand sides of s, respectively. formally, given the tuple (x, c, s), the wlac task aims to predict the target word w with s as its prefix, which is the most appropriate to be placed between c l and c r @xcite . to make the task more general in real-world scenarios, wlac task assumes that the left context c l and right context c r can be empty, which leads to the following four types of context: ## evaluation metric we use both automatic evaluation and human evaluation to measure all submitted systems. ## submitted systems and results in this year, there are five teams participating in this shared task and we receive 27 submissions from them. in this section, first, we quickly describe the participants and their submitted systems, then we present their evaluation results in terms of both automatic and human evaluations, and finally, we shed light on some findings according to evaluation results. ## conclusion word-level autocompletion is a basic functionality in computer-aided translation systems to facilitate the editing efficiency for translators. in wmt this year, the word-level autocompletion shared task is introduced and it covers two language pairs including four directional subtasks. we provide high-quality test datasets and human evaluation to evaluate different systems fairly. on all subtasks we receive 27 submissions from five participants which address the wlac task from different perspectives. automatic and human evaluations on these submissions reveal some key findings which may provide valuable insights for future research on this task. finally, we hope that wlac task will attract more researchers to participate in the exploration of computer-aided translation.