Findings of the WMT Shared Task on Machine Translation Using Terminologies (2021)

Abstract: Language domains that require very careful use of terminology are abundant and reflect a significant part of the translation industry. In this work we introduce a benchmark for evaluating the quality and consistency of terminology translation, focusing on the medical (and COVID-19 specifically) domain for five language pairs: English to French, Chinese, Russian, and Korean, as well as Czech to German. We report the descriptions and results of the participating systems, commenting on the need for further research efforts towards both more adequate handling of terminologies as well as towards a proper formulation and evaluation of the task.

## introduction language domains that require very careful use of terminology are abundant. the need to adequately translate within such domains is undeniable, as shown by e.g. the different wmt shared tasks on biomedical translation. more interestingly, as the abundance of research on domain adaptation shows, such language domains are (a) not adequately covered by existing data and models, while (b) new (or "surge") domains arise and models need to be adapted, often with significant downstream implications: consider the new covid-19 domain and the large efforts for translation of critical information regarding pandemic handling and infection prevention strategies. in the case of newly developed domains, while parallel data are hard to come by, it is fairly straightforward to create word-or phrase-level terminologies, which can be used to guide professional translators and ensure both accuracy and consistency. this shared task foot_0 replicated such a scenario, and invited participants to explore methods to incorporate terminologies into either the training or the inference process, in order to improve both the accuracy and consistency of mt systems on a new domain. the shared task focused on five language pairs, with systems evaluated on: the goal of this 3-stage approach (with both development and surprise language pairs) is to avoid approaches that overfit on language selection, and instead evaluate the more realistic scenario of needing to tackle the new domain in a new language in a limited amount of time. the surprise language pairs were announced 3 weeks before the start of the evaluation campaign. the organizers provided training/development data and terminologies for the above language pairs. test sets were released at the beginning of the evaluation period. the participating teams were invited to participate in any or all of the language pairs. ## participants and system descriptions we received a total of 43 submissions from 9 teams. below we provide a short description of each submission. ## results and discussion the results and rankings for english-french are listed in in the english-french translation task, there are two winning submissions. two promt submissions ranked first according to exact-match accuracy (along with a cuni submission), but the promt.soft submission is statistically significantly better than the other two with respect to comet, hence it is one of the winning submission. the second winning submission is the one by kep, which ranks first according to comet, but also according to 1-ter, which indicates that it might strike a good balance between general translation quality and term consistency. in the english-chinese translation task there is a single winning submission, the one by termmind (system 2), which ranks first according to both metrics. we note that another submission (hw-tsc) is statistically significantly better than all submissions in all metrics except for 1-term, but this submission is an unconstrained one, and hence it is excluded from the rankings. in english-russian the promt submission promt.soft is the clear winner, ranking as the single best system according to exact-match accuracy, as well as one of the two best systems according to comet. interestingly, the other system that ranks first according to comet (promt.smartnd.v2) ranks first according to 1-term score, but also last according to exact-match accuracy, denoting perhaps an orthogonality between the goals of terminological consistency and general translation quality, where prioritizing one over the other leads to performance drops along the other dimension. last, the submissions by kep are the winning ones for english-korean and czech-german. for the former language pair it was the only submitted system (see discussion on potential reasons), while for czech-german it ranked for best system according to exact match accuracy with the other submission (by tildemt), but was significantly better according to comet. although tildemt used a more sophisticated approach to the terminology translation, the kep team had a stronger baseline and used ensembling which significantly increased both general translation quality and the term accuracy. ## related work phrase-based statistical mt systems @xcite allowed for fine-grained control over the system's output by design, e.g. by incorporating domain-specific dictionaries into the phrase table, or by forcing translation choices for certain words or phrases. on the other hand, the currently stateof-the-art approach of neural machine translation (nmt) does not inherently allow for such control over the system's output. some approaches src mozkové ::::::: metastázy vykazovaly nekonsistentní nebo žádnou fluorescenci. tgt hirn :::::::: metastasen zeigten inkonsistente oder keine fluoreszenz. . . . muskel ::::: blutung oder hämatom. tildemt zerebrale :::::::: metastasen zeigten eine inkonsistente oder keine fluoreszenz. . . . :::::::: blutungen in den muskeln oder hämatom kep hirn :::::::: metastasen zeigten eine inkonsistente oder keine fluoreszenz. . . . muskel ::::: blutung oder hämatom. in any case, the best practice for incorporating terminological constraints in nmt is both underresearched and still not settled yet, especially in the case of morphologically rich languages, underlying the need for this shared task. ## conclusion we presented the results of the first edition of the wmt21 shared task on mt using terminologies. for the purposes of the task we created new evaluation datasets, annotated by professional translators for their terminology consistency, based on the tico-19 data for english to french, chinese, russian, and korean, as well as a dataset for czech-german based on the emea corpus. the shared task received 43 submissions from 9 teams, 8 from industry and 1 from academia, underscoring the general applicability of our focus problem ('how best can we use a terminology in mt?') on real-world settings. most submissions add soft or hard constraints on the source side that the mt learns to handle, as proposed in @xcite , but other novel approaches include terminology filtering for selecting between multiple options provided by the terminology, or replacing terms with placeholders to be inserted after the mt has produced the output. we devised multiple terminology-targeted metrics and evaluated systems along both these metrics as well as general translation quality. in most cases we find that, encouragingly, one does not necessarily have to sacrifice general translation quality for terminology compliance, as long as the terminology is of adequate standards. in future iterations of the shared task, we will take into account the distinction between terminologies created for humans (which are abundant) and terminologies created specifically for mt systems which need to be created, and have different requirements/specifications that the former. in addition, we will attempt to consider a new domain, rather than focusing again on the biomedical domain and specifically covid-19 (although this is a great example of a "surge" domain that immediately required that translation providers and mt engines adapt in order to handle translations of large volumes of text in this novel domain).