R e IFE : Re-evaluating Instruction-Following Evaluation (2025)

Abstract: The automatic evaluation of instruction following typically involves using large language models (LLMs) to assess response quality. However, there is a lack of comprehensive evaluation of these LLM-based evaluators across two dimensions: the base LLMs and the evaluation protocols. Therefore, we present a thorough meta-evaluation of instruction following, including 25 base LLMs and 15 recently proposed evaluation protocols, on 4 human-annotated datasets, assessing the evaluation accuracy of the LLM-evaluators. Our evaluation allows us to identify the best-performing base LLMs and evaluation protocols with a high degree of robustness. Moreover, our evaluation reveals key findings: (1) Base LLM performance ranking remains largely consistent across evaluation protocols, with less capable LLMs showing greater improvement from protocol enhancements; (2) Robust evaluation of evaluation protocols requires many base LLMs with varying capability levels, as protocol effectiveness depends on the base LLM used; (3) Evaluation results on different datasets are not always consistent, so a rigorous evaluation requires multiple datasets with distinctive features. We release our meta-evaluation suite ReIFE, which provides the codebase and evaluation result collection for over 500 LLM-evaluators, laying groundwork for future research in instruction-following evaluation.

## introduction the ability to follow human instructions has become an important evaluation aspect for large language models (llms), indicating their alignment with human users @xcite . recently, due to their better correlation with human judgments compared with traditional evaluation metrics, the llms themselves are often used as judges of the model output quality for generative tasks including instruction following @xcite @xcite . these llm-based evaluation methods are an essential component of the most widely used automatic benchmarks for instruction-following evaluation, such as alpacaeval @xcite and @xcite , where a strong llm is used to evaluate the quality of model responses. moreover, they can be used as reward models for instruction fine-tuning of llms in both distillation and self-improvement settings @xcite . however, recent studies have identified various limitations of llm-based evaluation methods, including low self-consistency rates in their predictions, positional biases, and a preference for their own outputs @xcite @xcite . therefore, the evaluation of llm-based evaluations is critically important. such evaluations of evaluation methods, or meta-evaluation, usually involve comparing the automatic evaluation results against human evaluation @xcite @xcite . these evaluations of llm-evaluators assess two dimensions: @xcite the capabilities of base llms in performing the evaluation task and (2) the effectiveness of evaluation protocols -the methods by which base llms are used to perform evaluation, e.g., pairwise comparison as in alpacaeval or pointwise scoring as in mtbench. 2 existing work @xcite @xcite often lacks comprehensiveness in one or both of these dimensions, and more thorough evaluations are needed. we argue that the following two directions are crucial for a more comprehensive, rigorous evaluation of llm-evaluators for instruction following: @xcite including a diverse set of base llms determine the most common word in the text: humans are created in the image of god, from a spiritual perspective and from a physical perspective. the most common word in the text is 'humans', which appears twice. llama yi gemma tülu glm qwen mistral ... 25 open base llms base llm evaluation protocol metric enhanced propose at most three important questions about what constitutes a good output for the instruction: äã does the output indicate the most common word? öã can there be more than one most common word? multi-role debate complex {debate_history} claim your choice at the end of your debate turn: scientist critic writer multi aspect complex here's the aspect-wise analysis you wrote earlier {per_aspect_analysis}. make a final decision: analysis accuracy diversity fluency safety gpt-4 reference complex consider this reference answer from a strong ai assistant: {gpt_ref prepair i: pointwise analysis complex {pointwise_analysis_a} {pointwise_analysis_b} provide a pairwise evaluation: ii: pairwise comparison protocol-consistency self-consistency a b a a evaluation protocols: ... metric cot reference ... 15 protocols for the evaluation of evaluation protocols -while various evaluation protocols have been proposed recently @xcite @xcite , metaevaluation studies of these evaluation protocols often lack scale in the number of llms used. for example, llmbar @xcite uses only 5 llms to compare different evaluation protocols. as a result, it remains unclear whether the improvements observed in recently introduced evaluation protocols are robust and generalizable across base llms with varying performance levels. therefore, we aim to conduct an evaluation with a larger and more diverse set of base llms to ensure a more rigorous examination of the evaluation protocols. based on these goals, we present an in-depth meta-evaluation with the following components: our large-scale meta-evaluation, as outlined in figure 1 , enables a rigorous examination of the current progress in llm-based instruction-following evaluation, providing a solid foundation for developing evaluation protocols and evaluating base llms' evaluation capabilities. we make our metaevaluation suite reife publicly available to facilitate future research, which contains the codebase and evaluation result collection for over 500 llmevaluators, and we summarize our key findings: findings (1) when used in conjunction with 15 evaluation protocols, llama-3.1-405b @xcite is the best open-source base llm we evaluated ( recent work has proposed various advanced llm-based evaluation methods. for example, finegrained or decomposition-based approaches, such as chain-of-aspects (gong and mao, 2023) and branch-solve-merge @xcite , can guide llms to perform structured analysis by identifying fine-grained differences and providing detailed rationales. agent-based methods, like prd @xcite and @xcite , employ multi-role debate settings to bring diverse perspectives to the evaluation process. other techniques include probability-weighted scoring @xcite , reference-based evaluation @xcite , and self-consistency decoding @xcite . our study investigates the effectiveness of these advanced evaluation protocols on a larger scale, assessing their performance across multiple datasets and base llms. related studies have also explored fine-tuning llms as evaluators for various evaluation tasks including instruction-following evaluation @xcite , such as prometheus @xcite . however, we choose to exclude them from the majority of our evaluation since our focus is on generic llms with various evaluation protocols, while the fine-tuned llms usually require a fixed evaluation protocol. ## evaluation settings of reife in reife, we evaluate llm-based instructionfollowing evaluations along two dimensions: base llms and evaluation protocols (figure 1 ), using human evaluations as the gold standard. below, we outline the settings of this evaluation. ## baselines we first establish baselines for base llms and evaluation protocols regarding evaluating instructionfollowing for our further investigations. ## analysis in §5, a total number of 375 llm-evaluators are evaluated, combining 25 base llms and 15 evalua- dataset llm protocol acc. b.acc. nat. llama-3.1-405b swap&synthesize 98.0 94.0 adv. llama-3.1-405b gpt4-reference 87.8 83.1 mt. ## conclusion in this work, we conducted a large-scale metaevaluation of instruction following, examining 25 open-source base llms and 15 evaluation protocols while identifying the best-performing llmevaluators over 4 datasets. we found that a reliable evaluation of base llms' evaluation capabilities can likely be achieved with a single evaluation protocol due to the stability of their performance across different protocols. however, evaluating evaluation protocols should involve a diverse group of base llms, as they can significantly impact the evaluation protocols' effectiveness. we hope that our findings and meta-evaluation suite, reife, can pave the way for future studies in this direction.