How Reliable Are Automatic Evaluation Methods for Instruction-Tuned LLM s? (2024)

Abstract: Work on instruction-tuned Large Language Models (LLMs) has used automatic methods based on text overlap and LLM judgments as cost-effective alternatives to human evaluation. In this paper, we perform a meta-evaluation of such methods and assess their reliability across a broad range of tasks. In evaluating how well automatic methods align with human evaluations, correlation metrics are the most commonly employed method despite their inherent limitations when dealing with ties and different scales. To address these shortcomings, we use Pairwise Accuracy as an alternative to standard correlation measures. We observe that while automatic evaluation methods can approximate human ratings under specific conditions, their validity is highly context-dependent. Specifically, the simple ROUGE-L metric correlates very well with human ratings for short-answer English tasks but is unreliable in free-form generation tasks and cross-lingual scenarios. The effectiveness of the more advanced method of using GPT-4 as a judge diminishes significantly if reference answers are not included in the prompt, which is the scenario where this method has the potential to provide the most value compared to other metrics. Our findings enhance the understanding of how automatic methods should be applied and interpreted when developing and evaluating instruction-tuned LLMs.

## introduction a key strength of the current generation of large language models (llms) is their capacity to learn new tasks from instructions, either in-context @xcite @xcite or in a dedicated fine-tuning phase @xcite . the field has also seen the development of methods to adapt llms to new languages, for example, through continued fine-tuning † equal contribution @xcite , alignment with translation pairs @xcite , and instruction tuning on additional languages @xcite . the gold standard for evaluating generative tasks is human annotation, but this scales poorly due to high costs and time constraints. consequently, the most common approach for assessing generative llms is using automated evaluation techniques. among these, two popular methods are measuring text overlap with rouge-l @xcite and utilizing existing llms as automatic judges @xcite ; however, these methods only approximate human judgment, prompting questions about their reliability. while previous research has found that automatic evaluation methods correlate well with human assessments @xcite , it is important to recognize that these findings generalize over tasks of very different types and in different languages. additionally, correlation measures may not provide reliable estimates of alignment with human ratings, as they are limited in their ability to deal with ties and constant scores, which are common in human annotations @xcite . in this paper, we provide a thorough analysis of two widely-used automatic methods for approximating human judgments, rouge-l and llmas-a-judge. additionally, we experiment with bertscore, a semantic text similarity measure, to assess its potential utility. we study the reliability of the three measures across a broad range of english-language tasks. we also perform experiments on swedish as an initial study on the reliability of these metrics across languages. instead of using correlation measures, we employ pairwise accuracy @xcite to quantify the alignment with human ratings. our overall goal is to increase our understanding of the reliability of automatic evaluation methods and to establish guidelines regarding their appropriate usage. our contributions can be summarized as follows: ## related work we start by reviewing the research on the automatic evaluation of generated text, the use of llms as evaluators, and the methods applied to assess the alignment of metrics with human preferences. ## methodology in this section, we present the data and instructiontuned models used in our study. we provide an overview of the automatic metrics we employed to assess model performance and detail our approach to conducting a meta-evaluation of these metrics. ## results and analysis in this section, we present a comparative analysis of the evaluation methods in terms of their alignment with human assessments. ## conclusions and future work this study provides insights into the methods we use to evaluate language model generations, focusing on when automatic metrics align with human annotators and what the best metric is under different scenarios. we are the first to do a broader meta-evaluation study where we compare gpt-4as-a-judge and traditional metrics with a methodology that allows for reliable comparisons between metrics. we recommend using pairwise accuracy (pa) with tie calibration for meta-evaluation. this method effectively handles ties, which are prevalent when using human and gpt-4 ratings, making it a reliable tool for assessing metric performance against human ratings. our main finding is that gpt-4 shows strong alignment with human judgments for short-answer tasks, but only when gold references are provided. the reliability drops significantly without gold references, as the model is overly positive compared to human evaluations. the issue is particularly evident in free-form tasks, which are tasks where llmas-a-judge could be the most valuable and where gold labels are typically not available. when gold references are available, we observe that gpt-4 is too strict compared to humans, relying to much on the gold label. for these type of tasks, even though llm-as-a-judge is often applied to them, human evaluations still remain the gold standard. rouge-l performs comparably to gpt-4-gold for short-answer tasks, offering a cost-effective alternative in scenarios where the use of gpt-4 is limited by cost or time constraints. for long-answer tasks, while bertscore demonstrates strong performance, it does not fully replace the need for gpt-4-gold. these metrics provide valuable insights but vary in effectiveness depending on the specific task. evaluating non-english outputs, such as swedish, presents additional challenges. there is a significant drop in alignment for gpt-4-no-gold, which highlights that gpt-4 as a judge is less reliable for languages other than english. while this is true for swedish, we expect these findings to be more sever for lesser-resourced languages and those less similar to english. future work should focus on expanding this study to more languages. as we have observed, there is a large variation in alignment with human ratings for all metrics across task types. previous research identifies strong correlations with human annotators, but that is often the average over tasks. our findings underscores the necessity of task-specific evaluation metrics rather than relying on general averages which can obscure important nuances in metric alignment with human annotators. furthermore, while gpt-4 is a valuable tool for evaluation short-answer tasks when gold references are available, alternatives like rouge-l and bertscore can be effective for most tasks types, offering cost-efficient and reliable evaluations.