Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale (2020)

Abstract: Automatic evaluation of language generation systems is a well-studied problem in Natural Language Processing. While novel metrics are proposed every year, a few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation, despite their known limitations. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models by demonstrating important failure cases on multiple datasets, language pairs and tasks. Our experiments show that metrics (i) usually prefer system outputs to human-authored texts, (ii) can be insensitive to correct translations of rare words, (iii) can yield surprisingly high scores when given a single sentence as system output for the entire test set.

## introduction human assessment is the best practice at hand for evaluating language generation tasks such as machine translation (mt), dialogue systems, visual captioning and abstractive summarisation. in practice, however, we rely on automatic metrics which compare system outputs to human-authored references. initially proposed for mt evaluation, metrics such as bleu @xcite and meteor @xcite are increasingly used for other tasks, along with more task-oriented metrics such as rouge @xcite for summarisation and cider @xcite for visual captioning. @xcite remark that it is not sufficient to conclude on the usefulness of a natural language generation system's output by solely relying on metrics that quantify the similarity of the output to human-authored texts. previous criticisms concerning automatic metrics corroborate this perspective to some degree. to cite a few, in the context of @xcite state that human judgments may not correlate with bleu and an increase in bleu does not always indicate an improvement in quality. further, @xcite challenge the stability of the common practices measuring correlation between metrics and human judgments, the standard approach in the mt community, and show that they may be severely impacted by outlier systems and the sample size. in the context of image captioning, @xcite claim that the consensus-based evaluation protocol of cider actually penalises output diversity. similar problems have also been discussed in the area of automatic summarisation with respect to rouge @xcite . nevertheless, automatic metrics like these are a necessity and remain popular, especially given the increasing number of open evaluation challenges. in this paper, we further probe bleu, meteor, cider-d and rouge l metrics ( § 2) that are commonly used to quantify progress in language generation tasks. we also probe a recently proposed contextualised embeddings-based metric called bertscore @xcite . we first conduct leave-oneout average scoring with multiple references and show that, counter-intuitively, metrics tend to reward system outputs more than human-authored references ( § 3.1). a system output perturbation experiment further highlights how metrics penalise errors in extremely frequent n-grams ( § 3.2.1) while they are quite insensitive to errors in rare words ( § 3.2.2). the latter makes it hard to understand whether a model is better than another in its ability to handle rare linguistic phenomena correctly. finally, we design ( § 3.3) an adversary that seeks to demonstrate metrics' preference for frequent n-grams by using a single training set sentence as system output for the entire test set. we observe strikingly high scores, such as the sentence "a man is playing" obtaining a bleu score of 30.6, compared to 47.9 of a strong model, on a captioning task. we hope that our observations in this paper will lead the community towards formulation of better metrics and evaluation protocols in the future ( § 4). ## automatic metrics in this section, we briefly describe the metrics we use in our experiments. to compute bleu, me-teor, cider and rouge-l, we provide pre-tokenised hypotheses and references to coco-caption utility foot_1 . for bertscore, we use its official release foot_2 . ## experiments this section describes our probing experiments and results foot_3 for different tasks and language pairs.