Is C hat GPT a Good NLG Evaluator? A Preliminary Study (2023)

Abstract: Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community. Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics. However, the ability of ChatGPT to serve as an evaluation metric is still underexplored. Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric. In this report, we provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail, we regard ChatGPT as a human evaluator and give task-specific (e.g., summarization) and aspect-specific (e.g., relevance) instruction to prompt ChatGPT to evaluate the generated results of NLG models. We conduct experiments on five NLG meta-evaluation datasets (including summarization, story generation and data-to-text tasks). Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases. In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets. For the meta-evaluation datasets which are created greatly depending on the reference and thus are biased, the ChatGPT evaluator might lose its effectiveness. We hope our preliminary study could prompt the emergence of a general-purposed reliable NLG metric.

## introduction pre-trained large language models (llms; e.g., gpt-3.5, chatgpt and gpt-4), which are performed through chatting (or asking) with it, have obtained promising results on various natural language understanding (nlu) and natural language generation (nlg) downstream tasks @xcite @xcite @xcite @xcite . for example, @xcite show that chat-gpt can attain the comparable understanding abil-1 ity to some fine-tuned bert-style models on nlu tasks while failing to surpass current task-specific nlu models. @xcite prove that chatgpt can achieve good performance and even surpasses some full-shot models on several datasets through a multi-turn question-answering manner. for nlg tasks, @xcite claim that chatgpt performs competitively with commercial translation products (e.g., google translator) on high-resource european languages. @xcite demonstrate that chatgpt can balance well between informativeness and conciseness, and generate great cross-lingual summaries. although impressive performance on these tasks in terms of automatic evaluation metrics has been shown, it is still not clear whether chatgpt can evaluate the quality of textual generations as a human does. recently, using pre-trained language models as nlg evaluation metric, e.g., moverscore @xcite , bertscore @xcite , comet @xcite , bleurt @xcite , bartscore @xcite and mauve @xcite , receives increasing attention since it offers a decent human-related judgment from a deep semantic perspective. given the powerful ability of chatgpt as an intelligent conversational llm, researchers also attempt to investigate whether it can evaluate the translation quality as a human evaluator @xcite . however, the automated assessment of the general generation quality of nlg models still remains underexplored. in this report, we aim to answer the following research question: is chatgpt a good nlg evaluator? to this end, we regard chatgpt as a human evaluator and give task-specific (e.g., summarization) and aspect-specific (e.g., relevance) instruction to prompt chatgpt to evaluate the generation of nlg models. as the example shows in figure 1 , we also attempt different scoring criteria and whether to provide golden references in the prompts to systematically test the reliability of the chatgpt evaluator. we conduct experiments on five widely-used nlg meta-evaluation datasets (including summarization, story generation and data-to-text tasks). experimental results show that chatgpt exhibits a high correlation with human judgment in most cases especially for the story generation task, indicating its potential as an nlg metric. in addition, we find that the chat-gpt evaluator is sensitive to the prompts, and for different tasks or aspects, the prompts should be carefully designed. moreover, the creation method of the meta-evaluation datasets has a significant influence on the effectiveness of different evaluation metrics. if a meta-evaluation dataset is created greatly depending on the reference, the similarity between model generation and references serves as a strong signal to reflect human judgments, where simple similarity-based metrics (e.g., rouge) can achieve very strong performance. therefore, the chatgpt evaluator might lose its effectiveness in such situations. our main contributions are concluded as follows: • to our knowledge, we are the first to utilize chat-gpt as a general nlg evaluation metric to study its correlations with human judgments. ## chatgpt for nlg evaluation in this section, we discuss how to prompt chatgpt to serve as a reference-free nlg metric ( § 3.1) or a reference-based nlg metric ( § 3.2) to evaluate the generation quality of nlg models. we take the news summarization task as an example, and give the details of the prompt templates. ## conclusion in this technical report, we explore a research question: "is chatgpt a good nlg evaluator?". to this end, we design task-specific as well as aspectspecific prompts to guide chatgpt to perform as an nlg metric. experimental results on five widely-used meta-evaluation datasets, covering text summarization, story generation and data-to-text tasks, show the potentiality of chatgpt as an nlg metric. chatgpt achieves the new state-of-theart correlations (with human judgments) on sum-meval and openmeva meta-evaluation datasets, and obtains competitive results on newsroom and bagel datasets. in addition, we also find that the lexical biases involved in the meta-evaluation datasets would influence the effectiveness of nlg metrics, and might lead to the limited performance of the chatgpt evaluator. besides, the performances of chatgpt as an nlg evaluator are sensitive to the format of the prompt, for different tasks and aspects, the prompt should be carefully designed. we believe that chatgpt will exceed its current performance and provide a reliable nlg metric for the research community in the near future.