Are LLM -based Evaluators Confusing NLG Quality Criteria? (2024)

Abstract: Some prior work has shown that LLMs perform well in NLG evaluation for different tasks. However, we discover that LLMs seem to confuse different evaluation criteria, which reduces their reliability. For further verification, we first consider avoiding issues of inconsistent conceptualization and vague expression in existing NLG quality criteria themselves. So we summarize a clear hierarchical classification system for 11 common aspects with corresponding different criteria from previous studies involved. Inspired by behavioral testing, we elaborately design 18 types of aspect-targeted perturbation attacks for fine-grained analysis of the evaluation behaviors of different LLMs. We also conduct human annotations beyond the guidance of the classification system to validate the impact of the perturbations. Our experimental results reveal confusion issues inherent in LLMs, as well as other noteworthy phenomena, and necessitate further research and improvements for LLM-based evaluation.

## introduction with the emergence of powerful large language models (llms) such as chatgpt, llm-based evaluators have been widely used for various natural language generation (nlg) tasks @xcite @xcite . in evaluation for common nlg tasks such as summarization @xcite , dialogue @xcite , and story generation @xcite , different aspects of quality (such as fluency and faithfulness) should be considered individually. traditional evaluation metrics are either incapable of evaluating specific aspects, like bleu @xcite and @xcite , or they can only roughly assess a single aspect, like @xcite . in contrast, llms can be treated as akin to human annotators, with various definitions of aspects contained in the prompt for flexible evaluation. some studies @xcite @xcite have shown llm-based evaluators have achieved comparable performance with humans in many nlg tasks, suggesting llms to become promising candidates for automatic evaluation. however, during the explorations of llm-based nlg evaluation, we observed two noteworthy phenomena that have not been revealed in previous work. first, the evaluation results from llms for a given aspect can achieve a higher correlation with human judgments on another clearly different aspect. second, the correlations between llmgenerated scores across different aspects are significantly higher than those between human judgments accordingly. these lead us to question the reliability of llm evaluations on required aspects, since llms seem to confuse different aspects. understanding these issues is inseparable from aspects themselves at first, which stem from hu-man evaluation for nlg tasks and are typically described by terms and definitions, forming corresponding specific criteria. through semi-structured interviews, @xcite revealed that if aspects for evaluation lacked clear conceptualization, human annotators might conflate different aspects, such as fluency and grammaticality. @xcite pointed out the long-standing confusion of terms and definitions in human annotations, resulting in incomparable evaluations. combining our investigation of previous work involving evaluation criteria, we believe that there are two distinct issues. the first is inconsistent conceptualization, where the definition is inconsistent with others for the same aspect but is clearly articulated. the second is ambiguous expression, where the definition is so vague that human annotators aren't sure what it really means. in figure 1 , we present an example of evaluation for fluency, where the criteria enclosed by the dashed box are selected from existing work and correspond to these two issues. therefore, we should reduce the influence of the issues within the evaluation criteria as much as possible, so as to reveal the actual performance of llms on nlg evaluation across aspects. we collect many existing criteria from previous papers involved, and summarize a clear hierarchical classification system for aspects that are most commonly used. for each aspect, we construct five criteria with descriptions of different levels of detail, including default, detailed, and simplified ones, to explore the corresponding effects. then, inspired by behavioral testing in nlp @xcite , we elaborately design a series of perturbation attacks based on the classification system to conduct targeted analyses on both proprietary llms (gpt-3.5 and gpt-4 foot_0 ) and specifically fine-tuned llms like prometheus @xcite . different from previous related work, each of our perturbations is designed for a specific aspect to better verify the variances in evaluation for aspects that are related or not. we also engage human annotators to check our perturbations and expected impacts to enhance the reliability of our attack tests. to sum up, our contributions and findings are as follows: ## preliminary study to explore the nlg evaluation capabilities of llms and potential issues, we conduct experiments with gpt-3.5 on the commonly-used summarization evaluation dataset summeval @xcite , attempting the evaluation forms introduced by @xcite . their work, as well as other studies @xcite @xcite , has explored ## methodology our findings in the preliminary study lead us to question whether llms can understand and execute the evaluation requirements represented by different criteria well. to conduct more in-depth explorations, we propose the fine-grained perturbation test inspired by behavioral testing @xcite , hoping to reveal their more actual capacities for nlg evaluation. in particular, instead of relatively coarse-grained perturbations in previous work, our perturbation attacks have been crafted to specifically target certain evaluation aspects without affecting evaluations for other unrelated ones. we formulate our approach as follows: we first collect a set of different common criteria denoted as @xmath0, @xmath1, 2, . . . m}, conceptually involving inclusive and non-inclusive relationships. and each of our perturbation attacks p j , @xmath2, 2, . . . n is applied to the original text x to generate the corresponding perturbed text p j (x). meanwhile, each perturbation is designed and expected to only reduce the text quality regarding the criteria for the originally targeted aspect and others whose scopes cover it. we define the set of criteria affected by the perturbation p j as c j t , and the rest in c are defined as c j f . and the distinction between these two groups is made more reliable based on our classification system and human annotations. then, to conduct the test, we prompt the model to evaluate all the perturbed texts, as well as the original text, in the form of scoring to check the expected two different evaluation behaviors: ## experiments we primarily display the experiments and analyze the results with gpt-3.5, and the performance of other llms is described in section 5.4. and to minimize the interference from criteria themselves, we first analyze the results with the detailed type flu. coh.gram. sim. read. fai. cont. hal. inf. ade. all. flu. of aspect definitions, which have also been confirmed through human judgments, to align most closely with our expectations. the main results are shown in ## discussions to investigate the failures of llms in our attack tests, we conduct some extended experiments with detailed aspect definitions. we retain only the definition or term, or even use the empty criterion, with the results presented in figure 30 and 31 in the appendix, which still exhibit convergence. it indicates that the improper sensitivity of llms to grammaticality and contradiction is likely derived from the default evaluation behaviors inherent in llms. they will be cumulative, regardless of whether the current criteria are unclear or unrelated to those two aspects. moreover, the detailed aspect definitions indeed have effects for aspects whose terms are not commonly-used in nlg evaluation, like non-hallucination. furthermore, we have attempted different empirical methods to intervene in llm-based evaluation to mitigate these issues. specifically, the most intuitive solution is using explicit instructions to require llms to ensure the evaluation relies solely on the given criteria. however, there are only slight and unstable improvements. so we further considered the ideas of chain of thought (cot) to decompose the evaluation process. one method was inspired by multidimensional quality metrics (mqm), which first identified relevant issues based on the criteria, then conducted the evaluation based on the identified issues and their severity. another related method required llms to provide the preliminary evaluation, then utilized llms themselves to check if the evaluation strictly adhered to the given criteria, and finally offered an improved version. moreover, we also attempted to provide more comprehensive background knowledge for llms, such as by including other aspect definitions in the prompt. although these methods show varying degrees of improvement, none of them has a generally significant effect, and we believe that they do not fundamentally solve the reliability issues in llm-based evaluation, which are quite stubborn and challenging and necessitate more systematic research. to cover a more diverse range of descriptions of quality criteria, we also conduct human evaluation and llm-based evaluation with descriptions selected from existing papers. we find that ambiguous expressions play a similar role in both human evaluations and llm-based evaluation as less informative descriptions designed by us. inconsistent conceptualizations (e.g. a description mixing fluency and grammaticality) can alter human judgments on related aspect-targeted perturbations, and a similar but weaker effect exists in llm-based evaluation. ## conclusions in this work, we conduct fine-grained pertubation attack tests guided by the classification system and human judgments on llms to reveal their actual performance in nlg evaluation. our findings can be concluded as follows: 1) the performance of llms in our perturbation tests deviates significantly from expectations, with both unawareness and oversensitivity in some aspects.