LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models (2024)

Abstract: Current developments in large language models (LLMs) have enabled impressive zero-shot capabilities across various natural language tasks. An interesting application of these systems is in the automated assessment of natural language generation (NLG), a highly challenging area with great practical benefit. In this paper, we explore two options for exploiting the emergent abilities of LLMs for zero-shot NLG assessment: absolute score prediction, and comparative assessment which uses relative comparisons between pairs of candidates. Though comparative assessment has not been extensively studied in NLG assessment, we note that humans often find it more intuitive to compare two options rather than scoring each one independently. This work examines comparative assessment from multiple perspectives: performance compared to absolute grading; positional biases in the prompt; and efficient ranking in terms of the number of comparisons. We illustrate that LLM comparative assessment is a simple, general and effective approach for NLG assessment. For moderate-sized open-source LLMs, such as FlanT5 and Llama2-chat, comparative assessment is superior to prompt scoring, and in many cases can achieve performance competitive with state-of-the-art methods. Additionally, we demonstrate that LLMs often exhibit strong positional biases when making pairwise comparisons, and we propose debiasing methods that can further improve performance.

## introduction with the current rapid advances in generative ai, pre-trained models are increasingly utilized in a range of nlp tasks, necessitating reliable evaluations of these models. human evaluation, where annotators critically assess the quality of the outputs of natural language generation (nlg) systems, has been the gold standard approach @xcite @xcite ; comparative assessment prompts an llm to compare candidates in a pairwise manner, and the comparisons are subsequently converted into scores or ranks. @xcite . however, human evaluation has its drawbacks, and is notably labor-intensive, time-consuming, and costly. as such, automating the evaluation process and assessing nlg systems without human intervention is highly desirable. though there has been considerable progress in automatic evaluation methods, many proposed approaches have certain restrictions that limit their effectiveness. a large body of existing work use evaluation methods designed for particular tasks and attributes @xcite @xcite , for example, measuring the consistency of summaries @xcite . though effective within their domain, these approaches are not extensible to different nlg aspects and cannot be used by practitioners wishing to evaluate systems on inputs or properties that are less common. the recent development in the emergent abilities of llms @xcite has enabled llms to achieve impressive zero-shot performance for a slew of language tasks. this has led to general prompt-based assessment approaches, such as prompt-scoring where an llm is probed to score outputs on a particular aspect @xcite . these approaches are often only effective with massive llms with 175b+ parameters, which may limit the applicability of the approach, especially when access is limited to api access. with the insight that for humans, it is often easier to select which of two options is better than it is to score options independently, we question whether pairwise comparisons may be more effective at leveraging the impressive emergent ability of llms. in this work, we consider llm comparative assessment, where an llm is prompted to compare pairs of nlg candidates and predict which one is better. we demonstrate empirically that comparative assessment performs much better than promptscoring for flant5 and llama style models, and enables moderate-sized open-source llms to achieve near (or above) state-of-the-art performance across a range of nlg language tasks, for a diverse set of attributes. our approach is general and can be applied to a diverse range of tasks and textual attributes, is simple and requires minimal prompt engineering. further, we demonstrate that pairwise llm comparisons often exhibit strong positional biases, where the ordering of candidates impacts the decisions. we introduce a simple debiasing method and empirically illustrate that debiasing can provide further performance improvements, especially when large biases are present. our contributions are 1) we are the first work that comprehensively analyzes pairwise comparative assessment for nlg evaluation; 2) we demonstrate that comparative assessment is far more effective than prompt-scoring for moderately-sized llms, and yields performance that is state-of-theart for particular attributes; 3) we demonstrate that positional bias impacts comparative decisions, and introduce a method to debias llms which leads to performance boosts, especially when only a subset of comparisons are considered. ## conclusions this paper investigates llm comparative assessment, a simple zero-shot approach to nlg evaluation. we demonstrate that for moderately sized llms, comparative assessment outperforms absolute scoring, and is an effective automatic assessment, achieving near state-of-the-art performance for a range of nlg evaluation tasks. furthermore, we show that llms are prone to have positional bias that could impact their decisions, however, we introduce a simple debiasing approach that leads to performance boosts, especially for biased systems.