The Generative AI Paradox in Evaluation: “What It Can Solve, It May Not Evaluate” (2024)

Abstract: This paper explores the assumption that Large Language Models (LLMs) skilled in generation tasks are equally adept as evaluators. We assess the performance of three LLMs and one open-source LM in Question-Answering (QA) and evaluation tasks using the TriviaQA (Joshi et al., 2017) dataset. Results indicate a significant disparity, with LLMs exhibiting lower performance in evaluation tasks compared to generation tasks. Intriguingly, we discover instances of unfaithful evaluation where models accurately evaluate answers in areas where they lack competence, underscoring the need to examine the faithfulness and trustworthiness of LLMs as evaluators. This study contributes to the understanding of “the Generative AI Paradox” (West et al., 2023), highlighting a need to explore the correlation between generative excellence and evaluation proficiency, and the necessity to scrutinize the faithfulness aspect in model evaluations.

## introduction there has been a growing emphasis on the need for automatic evaluation to reduce costs in the assessment of free-form text generation, which traditionally required human evaluation. recently, with the performance of llms such as gpt-4 on linguistic tasks approaching or even exceeding humanlevel @xcite , and the improvement in their ability to follow instructions @xcite , there has been a surge in research on using llms for model evaluation. beyond using llms as evaluators when there is a golden set of answers @xcite , we focus on adapting llms for reference-free evaluation to meet the needs of recent long-form text evaluation. the assumption that an llm skilled in a specific generation task inherently possesses the capability to evaluate that task should be approached with caution. human evaluators tasked with assessing a certain activity are presumed to possess both a comprehensive understanding and the capability to execute said task. accordingly, the deployment of an llm as an evaluator often implies the same assumption. nonetheless, as highlighted in @xcite , there exist scenarios where an llm, despite exhibiting generative skills surpassing human experts, can still make basic mistakes in certain tasks -the kind of errors typically not made by human non-experts. this phenomenon, referred to as "the generative ai paradox", underscores a critical aspect of llm performance. this paper seeks to investigate the extent to which llms, when demonstrating superior generative abilities in a specific task, can effectively function as evaluators of that task. we use an open domain question-answering (qa) task as a case study. in this context, llm's free-form outputs represent "generation", while evaluating responses to the same qa pairs signifies "understanding". this investigation evaluates the performance of three llms and one open-source lm in qa and evaluation tasks, utilizing the triviaqa dataset @xcite . our analysis reveals a marked discrepancy in performance, with llms showing reduced effectiveness in evaluative tasks compared to their generative counterparts. notably, we identify instances of unfaithful evaluation, where models proficiently assessed answers in areas beyond their expertise. this study emphasizes the importance of critically examining llms' faithfulness and trustworthiness in their evolving evaluation roles. ## related work reassessing the capabilities of llms recent studies have raised questions about the inferred capabilities of llms based on their task performance. @xcite suggest that llms do not necessarily develop systematic problem-solving abili- generation triviaqa evaluation nigel hawthorne was born in , warwickshire (•••) coventry nigel hawthorne was born in , england. coventry evaluation : incorrect q. where in england was actor nigel hawthorne born? a. coventry case 1. generation correct, evaluation incorrect the two musical notes that have no flats are (•••) b and e. the only two musical notes that have no flats are b and e. evaluation : incorrect q. what are the only two musical notes which have no flats? a. c and f case 2. generation incorrect, evaluation correct correct correct incorrect incorrect figure 1: examples of gpt-4's generative ai paradox in evaluation. case 1 demonstrates a paradox where the generation is correct but the evaluation is incorrect, while case 2 shows the opposite paradox with the generation being incorrect but the evaluation being correct. ## generative ai paradox in evaluation figure 1 demonstrates the seemingly paradoxical behavior of a generative model. in case 1, gpt-4 correctly generates an answer in a qa scenario, but in an evaluation scenario, it erroneously judges the same answer. in this first case, while the model ef-ficiently performs the generation task of free-form qa, it fails to properly evaluate the qa pair despite the task being "easier", as it involves a selective question. this suggests that a model's competence and its qualities as an evaluator may not be aligned or correlated as one would typically expect. in the second case, gpt-4 generates incorrect answers during the generation process, yet it is evaluated as correct. this paradoxical phenomenon occurs when the model accurately evaluates problems for which it lacks competence in the task. as a result, there is a need to thoroughly examine the reliability and trustworthiness of the model's evaluation, which are crucial aspects of the evaluation process. among these aspects, we specifically focus on determining whether the model's scores are based on its actual knowledge, emphasizing the concept of faithfulness. it's important to note that our exploration does not aim to provide definitive evidence regarding the faithfulness of model-generated evaluation. instead, our goal is to investigate this phenomenon by analyzing a specific example. thus, we measure the performance of the evaluation by asking the following questions: ## analysis the following sections present the findings derived from a case-by-case analysis of the three factors: human evaluation label, model evaluation label, and evaluation accuracy.