Machine Translation Evaluation beyond the Sentence Level (2018)

Abstract: Automatic machine translation evaluation was crucial for the rapid development of machine translation systems over the last two decades. So far, most attention has been paid to the evaluation metrics that work with text on the sentence level and so did the translation systems. Across-sentence translation quality depends on discourse phenomena that may not manifest at all when staying within sentence boundaries (e.g. coreference, discourse connectives, verb tense sequence etc.). To tackle this, we propose several document-level MT evaluation metrics: generalizations of sentence-level metrics, language-(pair)-independent versions of lexical cohesion scores and coreference and morphology preservation in the target texts. We measure their agreement with human judgment on a newly created dataset of pairwise paragraph comparisons for four language pairs.

## introduction automatic machine translation (mt) evaluation is a crucial technique that accompanied the development of machine translation systems over the last two decades. it allows replacing accurate, but prohibitively slow manual evaluation by a fast and replicable automatic evaluation routine approximating human judgment. so far, the most attention has been paid to the evaluation metrics that work c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. with text on the sentence level and most of the mt systems work at sentence level as well. the recent advances in neural machine translation @xcite demonstrated that the stateof-the-art systems, are not too far from the humanlevel quality on the sentence level. translating paragraphs or even entire documents is thus becoming a new challenge for mt systems. while this progress is underway, one also needs to assess the translation quality at the paragraph level. quality of coherent text translation depends on discourse phenomena that cannot be resolved within sentence boundaries. for instance, the correct sequence of events in the text or the correct placement of gendered pronouns needs to be retained in the target language text to provide a correct translation. recent experiments with incorporating a broader context into neural machine translation @xcite brought only a modest improvement. as these approaches were evaluated using only sentence-level metrics, some important properties of the models might have been missed. another important motivation for developing paragraph-or document-level metrics is the growing popularity of reinforcement learning in neural mt, optimizing the model directly towards a given metric @xcite @xcite . if we want to take advantage of this setup at the paragraph level, more elaborated metrics are necessary. in this paper, we propose several paragraphlevel mt evaluation metrics. we evaluate how these metrics agree with human judgment while deciding which translation is better when only a single paragraph of text is used for the comparison on four different language pairs. because of the lack of annotated data, we create our own dataset consisting of the system outputs submitted to the shared translation tasks of the workshop on machine translation (wmt) between 2014 and 2016 @xcite @xcite . the dataset with anonymized paragraph translation ratings will be published with the final version of this paper. the remainder of the paper is organized as follows: section 2 summarizes the previous work, section 3 introduces the paragraph-level level metrics, section 4 describes the evaluation dataset. in section 5, we describe the experiments we conducted to estimate agreement of the proposed metrics with human judgment. ## previous work there have been a few attempts so far to measure translation quality beyond the sentence level. with most of the mt frameworks still translating sentence by sentence, there was no urgent need to measure quality at higher levels. the fact that the standard mt scoring methods such as bleu (bilingual evaluation understudy) @xcite , seem to correlate well with human judgment further supported and established that practice. with the advent of high-quality sentence-level machine translation @xcite @xcite , one of the next challenges is to translate entire paragraphs and documents consistently, i.e. in a lexically coherent and pragmatically appropriate manner. argumentative structure of text, consistency of lexical choice, and the right 'tone' for its pragmatic intent are the next problems to focus on. simple n-gram matching (as with bleu) and/or allowing for certain word order and synonym variants (as with meteor (lavie and agarwal, 2007)), will likely not be able to capture the aforementioned linguistic phenomena that are crucial for the coherence of the entire text. more aggravatingly, both bleu and meteor heavily rely on comparison against one (or sometimes up to 4) human reference translations. these are however not usually available for an entire document. the blue score is technically a corpus-level metric because it computes the brevity penalty over the whole corpus. nevertheless, it does not make use of cross-sentence information in a particularly useful way. @xcite empirically showed that enforc-ing the one-sense-per-discourse hypothesis by repeating the same words in an mt output can potentially improve the mt quality. @xcite proposed measuring the semantic similarity of previously seen words in a text in order to capture lexical cohesion of documents in the target language. lexical cohesion relates to word choice, that wong and kit measure by tracking collocation and reiteration (of word stems), additionally allowing for synonyms, near-synonyms and superordinates (for collocation). we take on this approach as well and provide a language-independent variant in section 3.1. @xcite on the other hand, viewed the document-level mt evaluation as a ranking problem. they built an mt system that relies on regression models to find bleu-like numbers for good translations at the document-level which are then ranked higher than others. similarly to what we will find below, soricut and echihabi have shown that an averaged bleu score over a document is a useful indicator of actual good translation quality and can be used as a feature to find pseudo-reference translations (coming from a secondary mt system) that in turn can be used to estimate the quality of the former mt system. similarly, scarton and specia (2014) are concerned with quality estimation at the document level, especially when no human reference translations are available. they use a mix of pseudoreference scores, as @xcite , together with the lexical cohesion features by @xcite . they take the word form repetitions to make the metric languageindependent, while we rely on word embeddings that account for richer encoding of synonyms, antonyms etc. than just pure repeated mentions. the main discursive features scarton and specia use are lsa scores. they rely on spearman rank correlation of the word vector of a current sentence compared to all sentences of the document. whereas both soricut and echihabi's and scarton and specia's papers need human reference translations or at least pseudo-references for training their regression models, our metrics below can be deployed fully automatically and rely mostly on a monolingual (but automatic) word aligner and freely available, automatic syntactic and semantic parsers. @xcite and miculi-cich @xcite use f -score based metrics for pronoun translation evaluation. in sections 3.2 we take a similar approach from computing coreference preservation. besides the approaches presented above, there have also been a few attempts to measure translation quality for certain discourse phenomena in isolation. @xcite have developed a metric to measure improvements on mt for discourse connectives, whereas for example @xcite and @xcite specifically looked at measuring translation quality for verb tense. although these approaches have presented interesting results, they can unfortunately not point to the overall translation quality of an entire paragraph. ## implemented paragraph-level metrics we implement two sets of metrics. the first ones operate on the paragraph level and are mostly generalizations of existing mt evaluation metrics (see section 3.1). the second set of metrics relies on monolingual word alignment between the reference paragraph and the translation hypothesis (see section 3.2). word alignment allows us to measure linguistically motivated statistics about the translation. nevertheless, alignment errors can pose the danger of bringing additional noise to the evaluation. moreover, word alignment is only an approximation of what we would really need for thorough document level statistics which would be phraselevel alignment. in order to find linguistic features (especially entities, coreference and morphology) for the metrics described in the following section, we have been analyzing the respective texts with the google cloud natural language api foot_0 . ## dataset unlike sentence-level mt evaluation which can benefit from evaluation campaigns like the wmt tasks of annual metrics evaluation @xcite , there is no dataset consisting of human judgments on machine translation quality beyond the sentence level. even the metrics that were discussed in section 2 were only evaluated against human judgments collected at the sentence level. in order to evaluate our metrics reliably, we created a new dataset consisting of pairwise paragraph comparisons of machine translation outputs that have been rated by several human annotators per pair. the paragraphs are extracted from the freely accessible test sets provided for the @xcite . our rated data sets will be made available publicly with the final version of this paper. ## experiments we evaluated the metrics proposed in section 3 on the collected datasets on english to german and english to french translation directions. for every metric, we computed the proportion of cases when the paragraph annotated as the better one has also been assigned the higher score, i.e. which of the two system outputs provides a better entire paragraph translation when comparing to the reference. all the paragraphs were also evaluated with the standard sentence-level metrics (bleu, me-teor, ter) foot_2 . the detailed results are presented in if we interpret the annotator agreement as probability that all three annotators agree, we can factorize this probability into two steps: first that two agreed (and thus did the majority vote) and that the third annotator agreed with them. therefore, we can estimate the probability of the third annotator agreeing with the majority vote as a square root of the annotator agreement. these are presented in the first line of the main finding of the analysis is that the agreement of both the traditional sentence-level metrics and the proposed metrics with the human judgment is relatively low in pairwise comparison. in fact, only a small majority of the pairwise comparisons is done correctly. this particular finding contradicts the training techniques based on the reinforce algorithm @xcite where the update rule explicitly contains the pairwise comparison. moreover, it is not clear whether there is a room for improvement given that for good translation systems, the performance of the automatic metrics is on par with the estimated human agreements. the other interesting result is that it is possible to estimate which translation is better almost equally well when focusing only on a particular phenomenon (coreference, lexical cohesion, morphology) as with metrics that should capture the translation quality holistically (meteor, bleu). the metrics based on morphological analysis achieved better performance on paragraph pairs consisting of good translations. it might be so because the morphological analysis is more likely to fail in case of malformed translation outputs where the monolingual alignment is more difficult, because the hypothesis is different from the reference. a similar trend can also be observed for coreference preservation. the blanc score used for coreference evaluation is an average of f 1 scores of estimating correctly the coreference links and its complement-non-link relations. often, a better aggrement was achieved with the score computed only over the non-link relations which are much denser than the coreference links. we hypothesize this makes the score more robust to alignment errors. ## conclusions the presented study focused on two main new contributions. first, we implemented an entire package of automatic paragraph-level mt quality metrics that are language-(pair)-independent and track mt quality at different levels throughout entire paragraphs. our extensions of the meteor and lexical cohesion scores thereby showed promising results for most adequately and consistently measuring paragraph-level mt quality. we also experimented with more linguistically motivated scores, such as coreference preservation that could be interesting for future experiments, once the alignment of pronouns and referential expressions is more reliable. second, we prepared a dataset of human judgments on pairwise comparisons of mt quality at the paragraph level which can be used for new metrics evaluation. the dataset consists of system translations from english to czech, french, german and russian submitted to wmt in recent years. for each language pair, 400 pairwise comparisons of randomly selected paragraphs and another 400 pairs of more similar, high-quality translation pairs have been rated humanly for paragraph translation quality. future work will try to improve the monolingual alignment. better performance of parsers and coreference resolvers would indirectly also help the presented metrics. integration of pseudoreferences (where no human reference translations are available) and training an ensemble of all the metrics in our package can also be a promising direction.