Finding Blind Spots in Evaluator LLM s with Interpretable Checklists (2024)

Abstract: Large Language Models (LLMs) are increasingly relied upon to evaluate text outputs of other LLMs, thereby influencing leaderboards and development decisions. However, concerns persist over the accuracy of these assessments and the potential for misleading conclusions. In this work, we investigate the effectiveness of LLMs as evaluators for text generation tasks. We propose FBI, a novel framework designed to examine the proficiency of Evaluator LLMs in assessing four critical abilities in other LLMs: factual accuracy, instruction following, coherence in long-form writing, and reasoning proficiency. By introducing targeted perturbations in answers generated by LLMs, that clearly impact one of these key capabilities, we test whether an Evaluator LLM can detect these quality drops. By creating a total of 2400 perturbed answers covering 22 perturbation categories, we conduct a comprehensive study using different evaluation strategies on five prominent LLMs commonly used as evaluators in the literature. Our findings reveal significant shortcomings in current Evaluator LLMs, which failed to identify quality drops in over 50% of cases on average. Single-answer and pairwise evaluations demonstrated notable limitations, whereas reference-based evaluations showed comparatively better performance. These results underscore the unreliable nature of current Evaluator LLMs and advocate for cautious implementation in practical applications.

## introduction large language models (llms) are gaining widespread acceptance as the gold standard for evaluation in numerous applications, thanks to their efficiency and significant reductions in cost & time compared to human evaluators @xcite @xcite @xcite ## fbi: meta-evaluation checklist we introduce fbi, a meta-evaluation benchmark designed to assess the capabilities of evaluator llms in examining the outputs of other llms across four distinct task abilities: (i) factual accuracy, (ii) reasoning ability, (iii) instruction following, and (iv) proficiency in long-form writing. each instance within the benchmark comprises a tuple (i, a gold , a perturb ), where i represents the input instruction or prompt given to the model, a gold denotes the correct or gold answer, and a perturb signifies a perturbed version of the gold answer. the perturbed answers, a perturb , are generated by introducing specific types of errors across each of the four task abilities ( llm evaluators can accurately identify and account for these errors in the perturbed answers. the perturbations are based on perturbation categories carefully crafted by human annotators, informed by the prevalent failure modes in current llms @xcite @xcite . these human annotators are graduate students who are well aware of the typical errors made by llms. such human oversight is used throughout the benchmark's development, from prompt selection ( § 3.1) to defining perturbation categories ( § 3.2) and creating the perturbations ( § 3.3). to ensure a high standard of accuracy and reliability, all perturbations within fbi undergo rigorous manual vetting( § 3.4). ## strategies for using evaluator llms in this section, we outline the prompting strategies employed by evaluator llms benchmarked on fbi. an evaluator llm, f (•), takes the input instruction, llm generated response and an evaluation prompt, p eval , as input, and is required to generate a score and an optional explanation. to make the evaluation more robust, the evaluator may also be provided with additional information specifying the axes of evaluation, rubrics, rules, and other criteria. our study focuses on 3 evaluation paradigms: (i) single-answer scoring ( §4.1), ( ii axis+rules @xcite : extending the axis+rubric strategy defined in sec §4.1, this strategy involves choosing the better response along the designated axis ax. the evaluator is prompted with the instruction i, two model responses a 1 , a 2 , details about the axis ax, and detailed rules for evaluation -represented as f (p eval , ax, r, i, a 1 , a 2 ) → (exp, verdict). ## experiments we use gpt-4-turbo as our primary evaluation model, given its widespread adoption @xcite @xcite . we also extend our analysis to other proprietary models -gemini-1.5-pro ideally, except for score-invariant perturbations, the evaluator should penalize the score of the perturbed answer. for pairwise comparison paradigm ( § 4.2), we include our "gold" answer as one of the responses, requiring the evaluator to select the best response between the "gold" and the "perturbed" answer. here, we measure the percentage of times the evaluator does not choose the gold answer as our metric. to mitigate position bias @xcite , we conduct each evaluation twice, swapping the order of the gold and perturbed responses. for reference-guided single answer scoring paradigm ( § 4.3), the gold answer serves as the reference. here, we measure the percentage of times the evaluator awards a perfect score to the perturbed answer as our metric. ## conclusion we propose fbi, a novel framework designed to evaluate the proficiency of evaluator llms in assessing four critical abilities: factual accuracy, instruction adherence, coherence in longform writing, and reasoning proficiency, through targeted perturbations. our comprehensive study, involving 2400 perturbed answers across 22 categories and using three evaluation paradigms (singleanswer, pairwise, and reference-guided evaluation), reveals significant shortcomings in current evaluator llms. our findings show that even the most advanced models failed to identify quality drops in over 50% of cases on average. while reference-based evaluations performed relatively better, single-answer and pairwise evaluations demonstrated notable limitations. these results underscore the unreliable nature of current evaluator llms and advocate for cautious implementation in practical applications. we hope that the fbi framework will be further extended and used for continued meta-evaluation of evaluator llms.