Automated Evaluation Metric for Terminology Consistency in MT (2022)

Abstract: The most widely used metrics for machine translation tackle sentence-level evaluation. However, at least for professional domains such as legal texts, it is crucial to measure the consistency of the translation of the terms throughout the whole text. This paper introduces an automated metric for the term consistency evaluation in machine translation (MT). To demonstrate the metric’s performance, we used the Czech-to-English translated texts from the ELITR 2021 agreement corpus and the outputs of the MT systems that took part in WMT21 News Task. We show different modes of our evaluation algorithm and try to interpret the differences in the ranking of the translation systems based on sentence-level metrics and our approach. We also demonstrate that the proposed metric scores significantly differ from the widespread automated metric scores, and correlate with the human assessment.

## introduction throughout the last decade, the quality of machine translation (mt) has improved significantly, and it is becoming a common phenomenon for various neural mt (nmt) systems to get better scores in manual direct assessment and other metrics than reference human translations @xcite . however, such figures are obtained when the mt outputs are evaluated on the sentence level (i.e., each sentence is assessed separately, without context); in document-level evaluation, human translations typically remain the best, although exceptions exist @xcite . we can explain this situation by the fact that most of the current state-of-the-art nmt systems translate documents sentence by sentence, which thus can provoke inconsistencies in the translation of different linguistic elements -from anaphoric pronouns to named entities and terminology. we focus on the latter. while term inconsistencies can be tolerable for the general spheres of communication, they are unacceptable for several professional domains, especially legal texts, where the coherent usage of terms is the ultimate characteristic. in the case of the term translation in the legal domain, the goal of the mt system can be split into several parts: in this paper, we present a novel metric that focuses on the consistency and unambiguity of terms, whereas measuring the third parameter, adequacy, is delegated to the mainstream automated metrics such as bleu @xcite and chrf @xcite . our proposed metric can be applied automatically, and it needs a small amount of human preprocessing and annotation (for instance, it does not require reference translation of the sentences). however, it can include manually tuned parameters as a variable. in section 2, we describe the background in the field of the term consistency in mt; in section 3 we introduce the algorithm of our metric; in section 4 we present the data on which the metric is applied; in section 5 we discuss the results and compare them to the widespread automated metrics in mt. limitations of our method are highlighted in section 6. ## background scholars have been drawing attention to documentlevel consistency for over a decade. for instance, @xcite presents a number of discourserelated phenomena (such as pronoun use and verb tense modeling) that should be taken into account, as well as an overview of the metrics that were designed to catch the consistency (by that moment, they did not correlate with human judgments much). since this time, there have been various experiments in enhancing the sentence-level mt models for better consistency, by a variety of means, from hierarchical approaches @xcite to postediting the output sentences @xcite . notably, the main focus of the proposed systems tends to be on the discourse-related features of texts, such as verb forms, anaphora, ellipsis, named entities, etc. @xcite , rather than on the terminology consistency. there has also been progress in designing the evaluation for lexical consistency in domainspecific spheres. for example, the creators of sao wmt test suite @xcite point out that the most accurate evaluation for the audit reports is performed manually by professionals in the field, while neither the automated metrics nor the direct evaluation by non-experts gives valuable information about the ranking of the systems' quality. the same authors in 2020 introduced the concept of the "markables": the linguistic elements to which the human annotators have to pay special attention @xcite . in the paper, they considered the domains of sublease, news, and audit, and the main markables were the crucial terms in the document. the research reaffirms that the automated metrics such as bleu are not very informative with respect to term consistency, while the non-professional annotators cannot spot the domain-specific inconsistencies; however, the additional annotation of the "markables" allows even the "lay" annotators to keep in mind the necessary terms, which makes the manual annotation more accurate and informative. another notable research by @xcite presents the ideas for automated metrics for the term consistency evaluation, namely, exact match accuracy, window overlap, and ter with bigger penalties for terms. the results of this approach, tested on the domain of medical texts about covid-19, show a correlation with human professional judgments; however, for most of the metrics, reference translations or at least term dictionaries are necessary. thus, the relevance of designing more automated metrics in the field is still valid. ## metric before explaining the metric in detail, we will reiterate our aim. our first objective is to reward translation consistency (i.e., penalize the one-to-many correspondences in source-to-target term pairs). secondly, we want translated terms to be unambiguous (i.e., we should penalize the many-to-one correspondences in source-to-target term pairs). optionally, we also want to include adequacy in our estimation (i.e., penalize the inappropriate translation of the term); otherwise, we will rely on the widely accepted metrics for adequacy. finally, we want the algorithm to be as automatic as possible, i.e., to avoid the necessity of human annotation on any level. to meet these demands, we introduce the following pipeline. ## data we used the data from the elitr agreement test suite to test the metric. the test suite consists of various short agreement documents, namely, 18 purchase agreements, 13 lease and sublease agreements, and two agreements on renting or using the software. all documents have czech as the source language and english as the target language; only for three files, the reference english translations are provided. as the mt outputs, we used the results of seven mt systems that took part in 2021 and 2022 competitions on this test suite. detailed information about the systems is presented in @xcite and ?, and the test suite texts are available online. foot_1 ## results and discussion in this section, we firstly comment on the absolute scores of the different variants of the proposed metric; secondly, we compare the ranking of the mt systems by our metric and by the ones represented in the findings of wmt21 and wmt22. ## limitations and perspectives as was stated, we proceed with testing our metrics, both "extensively" (on more data) and "intensively" (by tweaking the inner parameters of the metric itself). regarding the "extensive" analysis, we firstly should retrieve the automatic metrics obtained for the elitr agreement corpus and compare them to our findings. secondly, we should test our method on other language pairs or at least on the opposite english-to-czech direction. the second priority covers a more "intensive" analysis of the metric. the method that we suggest is based on several automated (or semi-automated) steps. for each of the steps (keyword extraction, word alignment, manual restriction of the term translations), different approaches and algorithms can be used. so far, we have tested the yake @xcite and keybert foot_2 keyword extractors for the first step. we compared their performance on the legal text outside the main elitr collection (this means that, for regex-based extractor, we created the templates based on the elitr connection and applied it to the testing text). tentative analysis shows that for the considered text, regex term extractor demonstrates the best performance, with 100% precision and 64% recall (7 out of 11 terms). both yake and keybert output an excessive number of false positive results, thus showing a dramatic decrease in precision (best performance -yake with 1-token keyword retrieval, 35%). the recall scores for these algorithms decrease as well: the comparable result is performed only by yake (54% for 1-token keyword retrieval), while the 2-token length yake shows 36% and keybert shows 9%. this can lead us to the conclusion that the regex term extraction is the best algorithm. however, if we apply these extractors to different texts of a similar domain -audit report (retrieved from another elitr repository, foot_3 ) we will see that the regex keyword extraction outputs no terms at all. the reason is that the terms in this report are introduced only in parentheses, with no additional explicit hints (such as "hereinafter referred as. . . ") in the legal texts. both machine learning-based algorithms, in contrast, manage to catch at least some of the necessary terms. this drives us to the conclusion, that for the robustness of the regex-based term extraction, we should take into account different "strategies" of introducing the terms in the document (sometimesby parentheses, sometimes -by additional phrases). this means that, before evaluating a new collection of the exact text, we still need some human effort to understand the strategy of the term marking there. another way for a bigger automatization can be using the combination of different keyword extraction algorithms, and choosing the terms through a majority vote or taking the union. finally, we can look at the problem of the term extraction and alignment from an opposite perspective: if there is no reliable combination of the automated algorithms for these two steps, we can use our metric semi-manually: the steps 2-3 from section 3 will be completely handed over to human annotators, and their results will be processed automatically by steps 4-5. of course, such implementation will be more time-and effort-consuming, but, firstly, it should still be faster than other manual evaluation approaches such as mqm, secondly, it will give us a model results of term extraction and alignment, against which we will compare the automated algorithms. the last notable limitation of the proposed approach is rooted in linguistic issues. although the legal texts are very consistent in using the same term for the same concept, there regularly appear cases of "legitimate" homonymy, where two terms can denote the same concept. this usually occurs when two or more antecedents can be referred to separately or by one term. the example is the following sentence: x, hereinafter referred to as "seller", and y, hereinafter referred to as "buyer", together also as "contracting parties".... such ambiguity (when person x can be both referred as "seller" and as "contracting parties") may cause the problems even within the correct translation, if in the original the chosen formulation would be "the seller and the buyer", and in the target language it would be chosen as "contracting parties". the current metric does not have any capacity to capture this feature of the legal language domain. ## conclusion we have presented the metric for evaluating the terminology consistency of the automatically translated texts. among its main advantages is its ability to be automatized and its relative simplicity of interpretation. we have tested our metric on the texts from the legal domain in the czech-to-english translation pair, and we have obtained the results that, according to preliminary estimates, correlate with human document-level judgements and statistically differ from those of the automated metrics such as bleu or chrf. we are continuing our analysis to understand the scope of our metric's functionality and test it on other language pairs. we publish our code of the project online at the github page foot_4 of the institute of formal and applied linguistics, charles university. we will appreciate feedback on the current algorithm, and we are open to discussion and suggestions on its improvement.