Can Large Language Models Be an Alternative to Human Evaluations? (2023)

Abstract: Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans. However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms. Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation. We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation. We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: open-ended story generation and adversarial attacks. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer. We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.

## introduction human evaluation is an important method to understand the performance of an nlp model or algorithm @xcite . we rely on human evaluation because there are certain aspects of texts that are hard to evaluate using automatic evaluation metrics; thus, researchers resort to humans to rate the quality of the output of nlp models. while human evaluation is prevalent and indispensable in nlp, it is notoriously unstable @xcite . @xcite has shown that the quality of workforces in human evaluation can have a detrimental effect on the evaluation result, making it impossible to compare the performance among different systems. reproducibility is another issue in human evaluation since it is hard to recruit the same human evaluators and rerun the same evaluation. even if the same workers are recruited, the workers that have seen the task before are likely to produce a different evaluation result the next time because they have already done the task. while human evaluation is used to better assess nlp systems and has some advantages over automatic evaluation metrics, the drawbacks of human evaluation somewhat make it difficult to reliably evaluate nlp systems. to resolve some of the drawbacks, we take advantage of large language models (llms). llms are large models that are trained to model human languages using self-supervised learning @xcite and further using special training procedures to improve the performance on unseen tasks and better follow natural language instructions @xcite . the ability to perform a task just given the task instructions motivates us to ask if these llms can perform what humans do in human evaluation. to answer this question, we feed in the llm with the same instruction, sample, and question used in human evaluation, and take the sequences generated by the llm as the llm's answer to the question. this process is shown in figure 1 , and we call this process llm evaluation. to test if llm evaluation yields meaningful results, we conduct llm evaluation on two different nlp tasks: evaluating the quality of stories in openended story generation and the quality of sentences generated by adversarial attacks. we summarize our findings and contribution as follows: ## example task 1: open-ended story generation we first use open-ended story generation to demonstrate the usefulness of llm evaluation. ## example task 2: adversarial attack as another application, we use llm evaluation to rate the texts generated by adversarial attacks. ## discussions in this paper, we propose to use llm for evaluating the quality of texts to serve as an alternative to human evaluation. to demonstrate the potential of llm evaluation, we use llms to rate the quality of texts in two distinct tasks: open-ended story generation and adversarial attacks. we show that even if llms have exceptional zero-shot in-context learning ability, they are not always suitable to be used for llm evaluation. still, we find that the best instructgpt and chatgpt can rate the quality of texts like human experts on the two tasks we used as examples. overall, the results in this paper demonstrate that llm evaluation has the potential to be used to evaluate nlp systems and algorithms.