I nstruct E val: Instruction-Tuned Text Evaluator from Human Preference (2024)

Abstract: This paper explores to construct a general text evaluator based on open-source Large Language Models (LLMs), a domain predominantly occupied by commercial counterparts such as GPT-4. Recognizing the limitations of open-source models like Llama in evaluative tasks, we introduce InstructEval, a general multi-aspect text evaluator developed through instruction tuning of open-source LLMs. To overcome the shortage of annotated resources for multi-aspect evaluations, InstructEval combines extensive open Human Preference Modeling (HPM) datasets with a small set of multi-aspect annotated data.This approach not only enhances effectiveness in overall evaluation tasks but also exhibits improved performance in multi-aspect evaluation tasks.As demonstrated by our extensive experiments, InstructEval achieves comparable or superior performance to commercial LLMs like ChatGPT or GPT-4 in terms of both overall and multi-aspect evaluation.

## introduction recent advancements in llms, exemplified by renowned models like chatgpt, have showcased their impressive zero-shot capabilities in a generative manner, empowering them to effectively handle a wide range of arbitrary human instructions @xcite . nonetheless, assessing the text quality presents a significant challenge due to the complexity of tasks and the necessity for multi-aspect evaluation @xcite . in recent research, leveraging the generalization capabilities of large language models (llms) as evaluators for text generation tasks has gained popularity. for example, a popular method involves using pairwise comparisons, with models like gpt-4 serving as benchmarks for identifying superior * work is done during an internship at baidu inc. † corresponding author. 🤖 0.75 ##instruction: are there precious gemstones on the moon, given its different geological history? in this paper, we aim at investigating the construction of a general multi-aspect text evaluator based on open-source llms. constructing such an evaluator is challenging mainly due to the following two reasons: (1) limited resources: training a text evaluator is hindered by the scarcity of annotated samples. (2) complexity of evaluation instructions: an effective text evaluator is required to comprehend intricate task instructions and as-sess text samples across multiple aspects. to address these challenges, we propose to fine-tune llms using publicly available human preference modeling (hpm) datasets. these datasets are designed to capture human preferences regarding responses to various questions and instructions. despite their human-assessed nature, hpm datasets represent valuable labeled resources, offering a diverse and rich set of instructions, as outlined in to put this insight into practice, we propose instructeval, a multi-aspect text evaluator that leverages instruction tuning on human preference datasets. initially, we establish a standardized prompt format, termed "eval-instruct", designed to unify human preference modeling (hpm) data from diverse sources. this format is composed of three integral parts: the task definition, the aspect definition, and the target sample. such a structure allows for the clear delineation and decomposition of both explicit and implicit evaluation criteria within the hpm data. subsequently, we engage in the fine-tuning of llms using the entirety of available open-source hpm data structured in the eval-instruct format. this process substantially enhances the llms' capabilities in assessing textual quality comprehensively. to further refine the llms' proficiency in complex multi-aspect evaluations, we construct a small set of multi-aspect evalinstruct for augmentation. this set is generated through a methodical process of random sampling, combining various task instructions and aspect definitions, and employing gpt-4 for the creation of specific assessment preferences. through joint finetuning of llms using both the publicly available hpm data and our constructed multi-aspect labeled data, instructeval not only demonstrates improved efficacy in evaluating the overall quality of text samples but also exhibits enhanced performance in multi-aspect evaluation tasks. extensive experimentation in both overall helpfulness and multi-aspect evaluation demonstrate that our proposed method, instructeval, achieves comparable or even superior performance to chat-gpt or gpt-4, despite fine-tuning with only 7b or 13b parameters of llama. furthermore, we con-duct a thorough analysis to examine the contributions of each resource in enhancing the evaluation capabilities of instructeval. this analysis allows us to gain valuable insights into the specific benefits provided by each resource. our contributions can be summarized into three folds: ## preliminaries and related works our work is closely related to the three domains discussed below, and there are subtle connections among these three domains as well. instruction tuning instruction tuning is a technique employed to assist llm in better comprehend and respond to a wide range of diverse instructions provided by humans @xcite . formally, with each task @xmath0, instruction tuning trains a model by maximizing: human preference modeling (hpm) hpm, often referred to as reward modeling (rm) in the literature on reinforced learning from human feedback (rlhf), leverages human preference samples to train an evaluation model @xcite . the main goal of hpm is to ensure that the evaluation model aligns with human judgment and accurately assesses the quality or performance of given samples. hpm is usually performed in a pairwise manner: given a pair of output samples (y w , y l ), where y w is the sample human preferred over y l , hpm trains the evaluation model by maximizing: text evaluation models previous text evaluation mainly focus on assessing a single task @xcite or a set of tasks in generation @xcite . benefiting from the powerful generalization ability of llms, current text evaluation models have the capacity to assess any given task. an evaluation model m should evaluate the quality of sample y based on task requirement t and the required evaluation aspect a i by outputting a score s @xmath1, a i ). recent works focus on how to prompting general llms for text evaluation. gptscore @xcite and g-eval @xcite study how to better use gpts for multi-aspect text evaluation. @xcite investigate the possibility of using gpt-4 as a replacement for human experts as evaluators and discovered its positional biases. ## instructeval in our proposed instructeval, our target is to construct a llm with the ability to evaluate not only from overall quality but also based on specific aspects defined. llms should provide objective evaluations even in situation where a particular task and aspect that never appeared in training set. to achieve this objective, our initial step proposes a standardized prompt format called "evalinstruct", which explicitly defines evaluation task and aspects. subsequently, we finetune a base llm using hpm samples to equip the llm with the capability to evaluate the overall aspect of the given task. in order to further augment the llm's capacity for specific aspect assessments, we create an additional set of samples evaluated in diverse aspects. these additional samples are then added to our training set, contributing to the training of our final instructeval. intuitively, the design of an instructeval can be deconstructed into three fundamental components: input format, scoring format and instruction tuning. in the subsequent section, we will provide a detailed introduction to each part. ## conclusion and future research this paper has addressed the challenges of text evaluation with llms. the major insight of this study lies in leveraging hpm resources for text evaluation. by utilizing the abundant hpm datasets, the study demonstrates the potential of fine-tuning llms to evaluate specific aspects and overall quality in a zero-shot manner. additionally, this paper propose to use a multi-aspect instruct set constructed by gpt-4, to enhance the performance of multi-aspect evaluation. the insights gained from the analysis of resource contributions in this paper can guide future research in designing more effective evaluation methodologies. understanding the specific benefits provided by different resources can inform the selection and utilization of appropriate datasets for evaluation tasks.