GPTS core: Evaluate as You Desire (2024)

Abstract: Generative Artificial Intelligence (AI) has enabled the development of sophisticated models that are capable of producing high-caliber text, images, and other outputs through the utilization of large pre-trained models.Nevertheless, assessing the quality of the generation is an even more arduous task than the generation itself, and this issue has not been given adequate consideration recently.This paper proposes a novel evaluation framework, GPTScore, which utilizes the emergent abilities (e.g., in-context learning, zero-shot instruction) of generative pre-trained models to score generated texts. There are 19 pre-trained models explored in this paper, ranging in size from 80M (e.g., Flan-T5-small) to 175B (e.g., GPT3).Experimental results on four text generation tasks, 22 evaluation aspects, and corresponding 37 datasets demonstrate that this approach can effectively allow us to achieve what one desires to evaluate for texts simply by natural language instructions.This nature helps us overcome several long-standing challenges in text evaluation–how to achieve customized, multi-faceted evaluation without model training. We make our code publicly available.

## introduction the advent of generative pre-trained models, such as gpt3 @xcite , has precipitated a shift from analytical ai to generative ai across multiple domains @xcite . take text as an example: the use of a large pre-trained model with appropriate prompts @xcite has achieved superior performance in tasks defined both in academia @xcite and scenarios from the real world @xcite . while text generation technology is advancing rapidly, techniques for evaluating the quality of these texts lag far behind. this is especially evident in the following ways: in this paper, we demonstrated the talent of the super large pre-trained language model (e.g., gpt-3) in achieving multi-aspect, customized, and training-free evaluation (fig. 1-(c) ). essentially, it skillfully utilizes the pre-trained model's zeroshot instruction @xcite and in-context learning @xcite ability to deal with complex and ever-changing evaluation needs while solving multiple evaluation challenges that have plagued many years. specifically, given a text generated from a specific context (e.g., source text in text summarization) and a desirable evaluation aspect (e.g., fluency), the high-level idea of the proposed framework is that gptscore input rel 0.9 inf 0.8 ... generate a summary for the following text. task specification aspect definition ... the details provided by the generated text are consistent with the details in the source text. rel ... inf text: ... summ: ... evaluation sample {task_specification} {aspect_definition]} text: {text} tl;dr: {summ} template demonstrated samples inf rel text: ... summ: ... evaluation protocol scoring inf generate a relevant summary with consistent details for the following text. text: {text} tl;dr: {summ} text: {text} tl;dr: {summ} ... text: {text} tl;dr: {summ} text: {text} tl;dr: {summ} rel demo sample the higher-quality text for a specific aspect will be more likely generated than unqualified ones, where the "likely" can be measured by the conditional generation probability. how to perform an evaluation as the user desires? as illustrated in fig. 2 , to capture users' true desires, an evaluation protocol 2 will be initially established based on (a) the task specification, which typically outlines how the text is generated (e.g., generate a response for a human based on the conversation); (b) aspect definition that documents the details of desirable evaluation aspects (e.g., the response should be intuitive to understand); (c) demonstrated samples: a handful of well-labeled samples are required to teach the model which sample is qualified. subsequently, each evaluation sample will be presented with the evaluated protocol with optionally moderate exemplar samples, which could facilitate the model's learning. lastly, a generative pre-trained model will be used to calculate how likely the text could be generated based on the above evaluation protocol, thus giving rise to our model's name: gptscore. given the plethora of pre-trained models, we instantiate our framework with different backbones: gpt2 (radford et al., 2019), opt @xcite , flan-t5 @xcite , and gpt3 (instructionbased @xcite ) due to their superior capacity for zero-shot instruction and their aptitude for in-context learning. experimentally, we ran through almost all common natural language generation tasks in nlp, and the results showed the power of this new paradigm. the main observations are listed as follows: (1) gptscore performs better when instructed by the 2 to better understand how to design the evaluation protocols, we give all the evaluation protocols for the different tasks and aspects studied in this work in the appendix f. our main contributions in this paper are: (1) we propose a newly generated text scoring framework, gptscore, which utilizes the emergent ability of large language model to achieve multiaspect, customized, and training-free evaluation. single-aspect evaluator refers to evaluators designed to evaluate the quality of a specific aspect or overall of the generated text. for example, deam @xcite and quan-tidce @xcite were proposed for the evaluation of the coherence of the dialogue system; several evaluators @xcite @xcite are designed for the evaluation of the consistency of text summarization. multi-aspect evaluator refers to one evaluator handle several evaluation aspects by using different input and output text pair @xcite , different prompt designed by the aspect name @xcite , @xcite , different formulas @xcite . unlike @xcite which only consider the vague aspect description, we fully considered exhaustive aspect definition and their relationship. emergent ability recent works have revealed various emergent abilities of generative pre-trained language models, such as, in-context learning @xcite , chain-of-thought reasoning @xcite , and zero-shot instruction @xcite . these abilities allow large language models to achieve good performance without training. what is the difference between our work and bartscore? @xcite demonstrates the feasibility of using the probability of text generation as a text quality score, and fine-tuning is required to achieve better performance. however, the model fine-tuning cost a lot, and it is hard for us to fine-tune a scoring model for each task and each domain. in this work, we focus on proposing a new framework that allows the generated text evaluation to achieve customizable, multi-faceted, and train-free evaluation. to achieve this target, (1) we utilized the emergent ability of language models, such as in-context learning, zero-shot instruction et al., to build the gptscore. ## generative pretraining score (gptscore) the core idea of gptscore is that a generative pre-training model will assign a higher probability of high-quality generated text following a given instruction and context. in our method, the instruction is composed of the task description d and the aspect definition a. specifically, suppose that the text to be evaluated is @xmath0, h 2 , • • • , h m }, the context information is s (e.g., source text or reference text), then gptscore is defined as the following conditional probability: in our work, we treat each token equally. t (•) is a prompt template that defines the evaluation protocol, which is usually task-dependent and specified manually through prompt engineering. few-shot with demonstration the generative pre-trained language model can better perform tasks when prefixed with a few annotated samples (i.e., demonstrations). our proposed framework is flexible in supporting this by extending the prompt template t with demonstrations. ## experiment results in this work, we focus on exploring whether language models with different structures and sizes can work in the following three scenarios. ( a ) vanilla (val): with non-instruction and non-demonstration; (b) instruction (ist): with instruction and non-demonstration; (c) instruc-tion+demonstration (idm): with instruction and demonstration. we studied four text generation tasks introduced in sec. 4.2. due to the limited space, we moved the results and analysis of the machine translation task into the appendix a. significance tests to examine the reliability and validity of the experiment results, we conducted the significance test based on bootstrapping. 5 our significance test is to check (1) whether the performance of ist (idm) is significantly better than val, and values achieved with the ist (idm) settings will be marked † if it passes the significant test (p-@xmath1.05). (2) whether the performance of idm is significantly better than ist, if yes, mark the value with idm setting with ‡. average performance due to space limitations, we keep the average performance of gpt3-, gpt2-, opt-, and ft5-based models. the full results of various variants can be found in appendix g. ## conclusion in this paper, we propose to leverage the emergent abilities from generative pre-training models to address intricate and ever-changing evaluation requirements. the proposed framework, gptscore, is studied on multiple pre-trained language models with different structures, including the gpt3 (175b). gptscore has multiple benefits: customizability, multi-faceted evaluation, and train-free, which enable us to flexibly craft a metric that can support 22 aspects on 37 datasets without any learning process yet attain competitive performance. furthermore, demonstrate that gptscore achieves the goal of "evaluate as you desire". this work opens a new way to audit generative ai by utilizing generative ai.