Are Automatic Metrics Robust and Reliable in Specific Machine Translation Tasks? (2018)

Abstract: We present a comparison of automatic metrics against human evaluations of translation quality in several scenarios which were unexplored up to now. Our experimentation was conducted on translation hypotheses that were problematic for the automatic metrics, as the results greatly diverged from one metric to another. We also compared three different translation technologies. Our evaluation shows that in most cases, the metrics capture the human criteria. However, we face failures of the automatic metrics when applied to some domains and systems. Interestingly, we find that automatic metrics applied to the neural machine translation hypotheses provide the most reliable results. Finally, we provide some advice when dealing with these problematic domains.

## introduction machine translation (mt) assessment is an open research question. the most accurate methods require a manual evaluation of the mt system. unfortunately, this is a difficult and costly process, being unaffordable while developing new mt engines. therefore, protocols for automatic evaluation of mt are required. the most common approach for evaluating the mt quality is to compare the system hypotheses with one or more reference sentences and compute a quality score. c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. a significant research effort has been spent on enhancing the automatic metrics. for instance, a shared task is running since 2008, as part of the conference on machine translation (wmt). although several metrics have been proposed, the literature is nowadays dominated by bleu (bilingual evaluation understudy) @xcite and, to a lesser extent, by ter (translation edit rate) @xcite . despite their usefulness, those metrics may diverge, sometimes leading to deceiving conclusions. this is the case of unconventional tasks or domains (e.g @xcite ). this work aims to shed some light on these behaviors, by conducting a human evaluation of mt outputs that produce inconsistencies in the metrics. more precisely, we study the correlation between human judgment and automatic evaluation on three problematic domains and for three different mt systems. we analyze the strengths and flaws that each automatic metric conveys, giving some advice for future research. the main contributions of this paper are the following: this paper is structured as follows: in section 2, we review relevant literature in the field of mt evaluation. section 3 provides a brief summary of the metrics under study in this work. section 4 explains the methods used to evaluate mt. in section 5, we describe the experimental setup. we show and discuss the results of our evaluation in section 6. finally, we conclude in section 7 by highlighting the main lessons learned from this work. ## related work as stated in the previous section, the automatic evaluation of mt quality is a key element for the effective development of mt. therefore, it has been studied from long ago @xcite . from here, a large amount of metrics have been proposed. among them, the most widely used, especially in the academia, is the aforementioned bleu. nonetheless it is also widely accepted that bleu suffers from several limitations when correlating with human judgments @xcite and can be fooled with bad translations @xcite . other metrics are also common in the literature. this is the case of ter, meteor @xcite , word error rate (wer) @xcite or nist @xcite . despite these efforts, the automatic assessment problem remains open, being organized several evaluation campaigns @xcite and shared tasks @xcite . due to the fragility and ambiguity of the existing metrics, several works attempted to perform a fine-grained evaluation of different mt systems or technologies. with the recent irruption of the neural machine translation (nmt) paradigm, a natural question arises: is nmt better than classical phrase-based statistical machine translation (pb-smt) systems? several works aimed to answer this question. @xcite performed an extensive comparison of nmt and pb-smt systems, measuring several facets of the translation, such as similarity, fluency or reordering. error analyses of nmt and pb-smt have also been reported, either automatic @xcite or manual @xcite . the conclusions were alike: nmt handled better verbs and nouns reordering, while the translation of proper nouns was worse. however, it is still uncertain whether the nmt paradigm works better in situations with scarce data, as pointed out by @xcite . a solution to this issue is to add monolingual data. the usage of synthetic data in nmt has reported excellent results in terms of bleu @xcite ; but a study on the importance of adding synthetic data in nmt with respect to the human perception of translation is still missing. ## automatic evaluation of machine translation in the context of this paper, the goal of automatic metrics is to assign scores to mt outputs in a way that they correlate with a human evaluation of the translation quality. in this section we briefly describe the eight metrics compared in this work. these are the most common metrics used for evaluating mt. ## methodology in this section we describe the human evaluation protocol applied in our work. we also describe how we computed the correlation across metrics. ## results and discussion in this section, we present and discuss the results obtained from our experimentation. we analyze all metrics according to the domain and to the translation technology. firstly, we show the overall metrics for the three different mt systems in the three different domains. ## conclusions in this work, we studied the behavior of automatic metrics in several translation systems for different domains. since the metrics provided contradictory results, we conducted a human evaluation, based on the da protocol. next, we computed the correlation of the automatic metrics with respect to the human criteria. our findings were that automatic metrics were closer to the human as more structured and contextual the task was. when evaluating tasks with short sentences (e.g. samples from a printer manual), the correlation of the automatic metrics with respect to the human greatly fell. we also found that the automatic metrics evaluated surprisingly well nmt systems, while failing in the evaluation of classical pb-smt systems. finally, we also found that the metrics were clustered, even in these specific domains, according to their nature, n-gram-based or error-based. therefore, we recommend to always give errorbased and n-gram-based metrics when reporting results on mt quality. as future work, we intend to develop a metric capable to complement the existing ones, especially when dealing with the aforementioned short and simple corpora.