C ritique LLM : Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation (2024)

Abstract: Since the natural language processing (NLP) community started to make large language models (LLMs) act as a critic to evaluate the quality of generated texts, most of the existing works train a critique generation model on the evaluation data labeled by GPT-4’s direct prompting. We observe that these models lack the ability to generate informative critiques in both pointwise grading and pairwise comparison especially without references. As a result, their generated critiques cannot provide fine-grained distinguishability on generated texts, causing unsatisfactory evaluation performance. In this paper, we propose a simple yet effective method called Eval-Instruct, which can first acquire pointwise grading critiques with pseudo references and then revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings, including pointwise grading and pairwise comparison with / without references. After fine-tuning on these data, the resulting model CritiqueLLM is empirically shown to outperform ChatGPT and all the open-source baselines and even achieve comparable evaluation performance to GPT-4 in system-level correlations of pointwise grading. We also demonstrate that our generated critiques can act as scalable feedback to further improve the generation quality of strong LLMs like ChatGPT.

## introduction recently, large language models (llms) @xcite @xcite have been improved rapidly and approached human-level performance on various natural language processing * equal contribution † work done when these authors interned at zhipu ai. ‡ corresponding author 1 the codes are available at https://github.com/ thu-coai/critiquellm . traditional evaluation metrics, usually based on n-gram overlap between generated texts and reference texts (such as bleu @xcite and rouge @xcite ), have limited effectiveness. recent works mostly resort to model-based evaluation metrics, especially llm-based ones @xcite @xcite . since most of the best-performing llms such as @xcite and @xcite can only be accessed via openai apis, researchers start to automatically collect evaluation data by directly prompting gpt-4 and train their own evaluation models, aiming to avoid potential risks of commerical apis, such as high cost, unstable usage, and data leakage @xcite @xcite . however, we argue that these evaluation models are still struggling to generate informative critiques in different evaluation tasks including pointwise grading and pairwise comparison. especially in the challenging reference-free setting, these models tend to generate general critiques without finegrained distinguishability on generated texts, causing unsatisfactory evaluation performance @xcite . in this work, we propose a simple yet effective method called eval-instruct, which can automatically construct informative instruction-tuning data for different evaluation tasks and settings, including pointwise grading and pairwise comparison with / without references. our main idea is to fully utilize referenced pointwise grading critiques, which are shown to possess rich information with the assistance of references and elaborate prompt design @xcite , to construct evaluation data for other tasks and settings. specifically, after acquiring pointwise grading critiques with pseudo references via gpt-4, we devise a multi-path prompting method including two strategies: 1) pointwise-to-pairwise prompting aims to inject pointwise grading critiques into pairwise critiques and enrich them with more information about the respective quality of text pairs. 2) referenced-to-reference-free prompting is targeted at removing direct comparison with references in referenced critiques, while keeping other details to improve the specificity of reference-free critiques. the evaluation data in different tasks and settings can be acquired via different paths consisting of these two strategies. and we also design a cross validation mechanism to improve the data quality of reference-free pairwise comparison because both of the two paths reach this task. after fine-tuning on the data of all the tasks and settings, the resulting model critiquellm is empirically shown to outperform all the opensource baselines and even achieve comparable performance with gpt-4 in system-level correlations of pointwise grading. we also show the potential of critiquellm to act as effective feedback to enhance the performance of llms like chatgpt. our main contributions are as follows: ## related work evaluation is a long-standing task in nlp, which becomes more challenging with the rapid development of llms @xcite . currently, there are mainly two lines of work on llm evaluation, including nlu-style and nlg-style evaluations. nlu-style evaluation methods utilize natural language understanding (nlu) tasks such as multi-choice qa to measure the performance of llms via simple objective metrics (such as accuracy and f1 score) @xcite @xcite , which may deviate from the common usage of llms and may not exactly reflect the ability of llms in generating responses for user queries. nlg-style evaluation methods extend metrics for natural language generation (nlg) tasks and expect to apply them to the measurement of llm's performance, which are the main focus of this paper. compared with early metrics that depend on the n-gram overlap between generated texts and reference texts @xcite @xcite , recently proposed metrics based on state-of-the-art llms like gpt-4 (openai, 2023) are shown to be strong evaluators due to the encouraging effectiveness of llms and the simplicity of formulating evaluation tasks as instruction-following tasks @xcite @xcite @xcite . since most of the state-of-the-art llms can only be accessed via apis, researchers start to automatically collect evaluation data by directly prompting gpt-4 and train their own evaluation models to provide stable and effective evaluations at a lower cost @xcite @xcite . the concurrent works similar to ours are the llms specially trained for evaluation tasks like pandalm @xcite , judgelm @xcite , and auto-j @xcite . for comparison, our work is the first attempt to deal with the challenge of uninformative critique generation which commonly appears in recent llmbased evaluation models especially without references. instead of prompting gpt-4 directly, our proposed eval-instruct can fully utilize the connection among different evaluation tasks and settings to construct informative evaluation data, which are empirically shown to improve the quality of generated critiques. ## experiment 4.1 dataset we adopt three benchmark datasets on open-ended instruction following, which involve various nlp tasks in llm's real-world scenarios foot_3 . the datasets also cover all the evaluation tasks and settings in this paper. the statistics are shown in alignbench @xcite ): this benchmark includes 8 categories of instruction following tasks dataset task setting #models #samples / #pairs length alignbench pointwise r / r-f 8 3,200 274 pairwise r / r-f 8 1,600 293 auto-j (eval-p) pairwise r-f 6 1,392 372 llmeval pairwise r-f 11 1,530 283 and 8 llms for generation. it provides an evaluation dataset with human-annotated scores on the quality of generated texts. in addition to using human-annotated scores for measuring pointwise grading performance, we also follow the original paper to sample text pairs of the same query for pairwise comparison foot_4 , whose label is automatically determined by their pointwise scores. auto-j (eval-p) @xcite : this benchmark provides 1,392 pairwise comparison data, each of which contains a user query, two llmgenerated texts, and a human-annotated preference label. these data involve 58 real-world scenarios and 6 model families for generation. ## conclusion we present an evaluation data construction method called eval-instruct, which can automatically construct informative evaluation data in both pointwise grading and pairwise comparison with / without references. after fine-tuning on the data from eval-instruct, the resulting model critiquellm can beat chatgpt and all the open-source baselines, and perform comparably with gpt-4 in systemlevel correlations of pointwise grading. cri-tiquellm can also provide scalable feedback which can improve the generation quality of llms.