CAMIE val: Enhancing NLG Evaluation through Multidimensional Comparative Instruction-Following Analysis (2025)

Abstract: With the rapid development of large language models (LLMs), due to their strong performance across various fields, LLM-based evaluation methods (LLM-as-a-Judge) have become widely used in natural language generation (NLG) evaluation. However, these methods encounter the following challenges: (1) distinguishing instruction-following ability, (2) being applicable across diverse NLG tasks, and (3) identifying low-quality outputs. To address these issues, we propose CAMIEval, a multidimensional comparative evaluation method based on instruction-following. Specifically, we define three fundamental dimensions of instruction-following: relevance, factuality, and adherence. Subsequently, we introduce a concrete Chain-of-Thoughts (ConcreteCoT) process to enhance the accuracy of evaluations. In addition, we trained a “regrettable model” RegretLM to generate low-quality outputs, which helps the evaluator better identify the potential shortcomings of the candidate output by comparing low-quality outputs with reference outputs. Through this comparison, the evaluator can generate instruction-specific dimensions that complement the fundamental dimensions, forming a more comprehensive evaluation metric system. Experiments on two NLG evaluation benchmarks demonstrate that CAMIEval consistently outperforms existing methods in terms of correlation with human evaluations, providing a general and accurate framework for evaluating the outputs of LLMs.

## introduction evaluating the quality of texts generated by language models has always been a challenging but essential task in natural language processing (nlp). an evaluation method more closely aligned with human judgment can provide effective feedback, including scores, explanations, and improvement suggestions. such an approach will help better understand the actual performance of language models and promote their continuous improvement @xcite . in recent years, with the rapid advancement of large language models (llms) @xcite @xcite @xcite , llm-based evaluation methods, known as llm-as-a-judge @xcite , have gradually become mainstream in natural language generation (nlg) evaluation @xcite . these methods leverage powerful llms for pairwise evaluation or pointwise scoring, following established evaluation processes or predefined evaluation criteria @xcite @xcite @xcite . a range of studies have demonstrated that these llm-based evaluation methods significantly outperform traditional n-gram-based @xcite or text-embeddingbased methods @xcite . in many tasks, their performance matches or even surpasses that of human evaluation @xcite @xcite . however, as shown in figure 1 , the current evaluation methods exhibit significant shortcomings in evaluating the outputs of the ever-evolving llms: suggest a snack that you can make in under 5 minutes. ## methodology camieval is a multidimensional nlg evaluation method based on comparative analysis, beginning with instruction-following evaluation. in this section, we will provide a detailed introduction to camieval, as shown in figure 2 . ## experiment we evaluate camieval on two benchmarks, llmbar @xcite and instrusum @xcite , which respectively represent pairwise evaluation of instruction-following and pointwise scoring of instruction controllable summarization. ## analysis the effect of concretecot. to mitigate description bias, we introduce the concretecot process to concretize the fundamental dimensions for evaluating the instruction-following performance. we compared the performance without using concrete-cot in pairwise evaluation, as shown in the effect of dimensions in camieval. in section 1 and 2, we discussed that manually defining evaluation dimensions is a labor-intensive process. furthermore, the currently predefined evaluation dimensions fail to distinguish the quality of outputs from current llms and are inapplicable to various downstream nlg tasks (dimension bias). to address these issues, we introduced a two-stage approach to generate and concretize fundamental and instruction-specific dimensions. to validate the effectiveness of our approach, we adopted the predefined dimensions of coherence, consistency, fluency, and relevance, as proposed in summeval @xcite , and applied them in the in-strusum benchmark. as demonstrated in the effect of regretlm. in order to capture the low-quality features of llm's outputs and ob- ## conclusion in this paper, we introduced camieval, a novel evaluation method for natural language generation that addresses critical shortcomings in current approaches, such as dimension bias, description bias, and insufficient focus on instructionfollowing. by defining fundamental dimensions and leveraging a concretecot process, along with a "regrettable model" regretlm to generate low-quality outputs for comparative analysis, camieval provides a general and precise evaluation framework. experimental results on the llmbar and instrusum benchmarks demonstrate that camieval significantly outperforms existing methods, achieving higher correlations with human evaluations and offering a versatile solution for evaluating the performance of llms.