Perturbation C heck L ists for Evaluating NLG Evaluation Metrics (2021)

Abstract: Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG tasks, we observe that the human evaluation scores on these multiple criteria are often not correlated. For example, there is a very low correlation between human scores on fluency and data coverage for the task of structured data to text generation. This suggests that the current recipe of proposing new automatic evaluation metrics for NLG by showing that they correlate well with scores assigned by humans for a single criteria (overall quality) alone is inadequate. Indeed, our extensive study involving 25 automatic evaluation metrics across 6 different tasks and 18 different evaluation criteria shows that there is no single metric which correlates well with human scores on all desirable criteria, for most NLG tasks. Given this situation, we propose CheckLists for better design and evaluation of automatic metrics. We design templates which target a specific criteria (e.g., coverage) and perturb the output such that the quality gets affected only along this specific criteria (e.g., the coverage drops). We show that existing evaluation metrics are not robust against even such simple perturbations and disagree with scores assigned by humans to the perturbed output. The proposed templates thus allow for a fine-grained assessment of automatic evaluation metrics exposing their limitations and will facilitate better design, analysis and evaluation of such metrics. Our templates and code are available at https://iitmnlp.github.io/EvalEval/

## introduction as the number of tasks and benchmarks for nlg have increased @xcite , the challenges in evaluating nlg systems have also continued to grow @xcite @xcite . one reliable way of evaluating nlg systems is to collect human judgements. however, this is a time consuming and expensive process @xcite @xcite . hence, automatic evaluation metrics such as bleu @xcite which are quicker to compute have become popular, despite being less reliable @xcite . the survey by @xcite shows that more than 35 automatic evaluation metrics have been proposed for nlg since 2014, however, there is no careful evaluation of the ability of such metrics to assess the quality of the output of an nlg system on multiple desired criteria. for example, consider the task of dialog evaluation, where humans are asked to score the output on multiple criteria such as fluency, adequacy, coherence, informativeness, engagingness, consistency, etc. contrast this with automatic evaluation metrics such as bleu, bleurt @xcite , deb @xcite , adem @xcite , etc., which assign a single score to the output. what does this score indicate? more specifically, does a low deb score indicate that the output is not fluent or does it indicate that the output is fluent but not coherent or neither fluent nor coherent? hence, a single overall score assigned by automatic evaluation metrics is not very informative in deciding which aspects of model improvement should one focus on. the question then is why do current automatic evaluation metrics produce only a single overall score? this is simply because of a conscious choice made while designing automatic evaluation metrics. in particular, current works only focus on evaluating whether the scores assigned by the proposed metric correlate well with the overall quality scores assigned by humans as opposed to all relevant criteria. in this work, we make a case for shifting the focus to all relevant criteria while evaluating such metrics. to this end, we first do a systematic study original sentence: cameron is the director of titanic change names: kate is the director of titanic (incorrect) the above results highlight a lacunae in the evaluation of automatic evaluation metrics wherein their ability to assess the output on multiple criteria is not evaluated. in this work, we propose a flexible framework which allows a systematic evaluation of the capabilities of an automatic evaluation metric. in particular, we propose checklist-style templates @xcite which evaluate the robustness of the metrics to certain perturbations targeting specific criteria. we illustrate this idea with an example in for the 6 nlg tasks mentioned earlier, we create 34 such perturbation templates covering 18 different evaluation criteria.we then instantiate these templates to create large-scale test cases. for every perturbation, we also collect human judgements to understand how much would a human change his/her score when shown such a perturbed output. we find that for several perturbations, the scores assigned by automatic evaluation metrics do not agree with the scores assigned by humans, thereby indicating that current automatic evaluation metrics are not robust to such perturbations (i.e., they do not really evaluate the desired criteria). overall, we believe that the proposed templates provide a better framework for a more fine-grained evaluation of automatic evaluation metrics which goes much beyond computing correlations with human scores. ## criteria used in human evaluations the goal of this work is to carefully evaluate automatic evaluation metrics with a focus on their ability to capture the diverse set of criteria used by humans while assessing nlg systems. to begin with, we describe the criteria that the output of an nlg system must satisfy for the 6 nlg tasks that we consider in this work, viz., machine translation (mt), dialog generation (dg), automatic summarisation (as), question generation(qg), data-to-text generation (d2t) and image captioning (ic). over the years, different works have proposed different criteria for evaluating nlg systems. in this work, we consider a popular set of criteria for each task as summarised in @xcite and presented in ## perturbation checklists so far we have established that if automatic evaluation metrics are to be used as a substitute for human evaluations as a whole, then they should be capable task criteria perturbation unmodified sentence perturbed sentence jumbling word order we play badminton every evening. we badminton every evening play. subject-verb disagreement he doesn't know how to bake. he don't know how to bake. fluency spelling errors make the most of every opportunity presented to you. make the most of evry opportunity presented to you. replace with synonyms the mangoes are delicious . the mangoes are tasty . contractions we are going to embark on an adventure. we're going to embark on an adventure. all tasks invariance numerals to words the flight will be delayed by 2 hours. the flight will be delayed by two hours. dropping out words or phrases i was being followed. i followed. add extra text this book is so inspiring. this book is so inspiring, i forgot . mt adequacy negation/antonyms it will rain on monday. it will not rain on monday. informativeness use hyponyms to create misinformation the girl my brother andy met through myspace turned out to be completely made up . the girl my friend andy met through myspace turned out to be completely made up. flow / coherence reorder sentences the pandemic was spreading uncontrollably. vaccines are being developed and tested rapidly. vaccines are being developed and tested rapidly. the pandemic was spreading uncontrollably. as referential clarity replace nouns by pronouns the pandemic was spreading uncontrollably. vaccines are being developed rapidly. it was spreading uncontrollably. they are being developed rapidly. drop question word when was he born ? was he born? answerability change question to assertive statement who is the director of titanic? the director of titanic is james cameron. mask a few words and predict how could tesla run the experiments ? how could tesla run to the beach ? qg relevance perturb nouns why did mary go to the school ? why did raj go to the market ? negate a previous statement by same speaker bot: i enjoy having your daughter in my class. user: i'm glad to hear that. i don't enjoy having your daughter in my class. making sense add extra non-sensible text do you know where dr. xyz lives? yes, my father is my grandmother's father repeat previous utterances do you know dr.xyz? do you know where dr. xyz lives? avoid repetition repeat phrases i like ice creams i like ice creams, ice creams listening replace with "can you repeat?" i need to book a taxi i'm sorry, can you repeat? dg relevance map random responses i am new to coding . i am scared of snakes . change gender two girls are playing with a doll two boys are playing with a doll correctness change attributes a small boy playing with a red ball a tall boy playing with a green ball drop objects/noun a small boy playing with a red ball a small playing with a red ic thoroughness repeat (append) object a lady riding a horse. a lady riding a horse and a lady . use hyponyms beethoven was a german musician beethoven was a german architect correctness change numbers the cricketer was born in 1990 . the cricketer was born in 1950 . drop phrases a small boy playing with a red ball a small boy playing with a data coverage repeat phrases beethoven was a german musician beethoven was a german musician and german musician . ## experimental setup we first do a coarse grained evaluation of several metrics by computing their correlations with the scores assigned by humans for multiple criteria. note that unlike existing studies which study such correlations for a small number of metrics (typically, n-gram based metrics) for a specific task (say, mt) and a single criteria (typically, overall quality), we do a more comprehensive study involving a combination of 6 tasks, 25 metrics and multiple criteria. apart from this coarse grained evaluation which simply looks at correlations, we also do a more fine-grained evaluation of the robustness of these metrics to different criteria-specific perturbations as summarised in ## results and discussion we now discuss the results of our experiments. ## related work some of the related work, particularly the relevant datasets, human evaluation criteria, and automatic metrics were already discussed earlier and hence not covered again here. we refer the readers to two recent surveys @xcite for a detailed overview of automatic evaluation metrics as well as related work on criticising the use of automatic evaluation metrics. we mention a few such important works here. bleu is one of the most widely analysed metric with several studies showing that it does not correlate well with human judgements for machine translation @xcite . this issue of poor correlations of metrics with human judgements has been reported on not just bleu, but also on various other metrics, across several nlg tasks including question generation @xcite , data-to-text generation @xcite , dialogue generation @xcite , and summarisation @xcite . apart from poor correlations, @xcite criticize the automatic metrics for abstractive summarization since they don't check for factual inconsistencies in the summaries. similarly wiseman et al. (2017) discuss the lack of a reliable measurement of faithfulness in the context of data-to-text generation. in case of dialogue, several n-gram-based and embedding-based metrics have been shown to fall short in capturing the diversity of the valid responses @xcite . the alternative of trained metrics, such as adem have been shown to be susceptible to adversarial attacks @xcite . similar to the main message of our work, some recent works have also called for a more robust evaluation of automatic evaluation metrics @xcite . ethayarajh and jurafsky (2020) also critically examine the current approaches towards nlp leaderboards and point towards having multiple metrics along different dimensions such as fairness, efficiency, robustness, etc. ## conclusion we conduct a large-scale study involving 6 tasks, 25 automatic evaluation metrics and 18 human evaluation criteria and observe that (i) different criteria such as fluency, coverage, etc are often not correlated and (ii) existing metrics have a low correlation with most criteria across different tasks. based on these observations, we suggest an alternative framework for evaluating evaluation metrics which goes beyond computing correlations with the human scores for overall quality. more specifically, we propose perturbation templates which allow a more fine-grained evaluation of such metrics and help in understanding their strengths and more importantly their limitations. we hope that future work on designing evaluation metrics will use our perturbation checklist for evaluating the effectiveness of the proposed metric in assessing different relevant criteria. ramakrishna vedantam, c. lawrence zitnick, and devi parikh. 2015. cider: consensus-based image description evaluation. in ieee conference on computer vision and pattern recognition, cvpr 2015, boston, ma, usa, june 7-12, 2015, pages 4566-4575. ieee computer society. sam wiseman, stuart m. shieber, and alexander m. rush. 2017. challenges in data-to-document generation. in proceedings of the 2017 conference on empirical methods in natural language processing, emnlp 2017, copenhagen, denmark, september 9-11, 2017, pages 2253-2263. association for computational linguistics. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. in 8th international conference on learning representations, iclr 2020, addis ababa, ethiopia, april 26-30, 2020. openreview.net. wei zhao, maxime peyrard, fei liu, yang gao, christian m. meyer, and steffen eger. 2019. moverscore: text generation evaluating with contextualized embeddings and earth mover distance. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, emnlp-ijcnlp 2019, hong kong, china, november 3-7, 2019, pages 563-578. association for computational linguistics. wanzheng zhu and suma bhat. 2020. gruen for evaluating linguistic quality of generated text. in proceedings of the 2020 conference on empirical methods in natural language processing: findings, emnlp 2020, online event, 16-20 november 2020, pages 94-108. association for computational linguistics.