Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLM s (2023)

Abstract: In recent years, large language models (LLMs) have drawn significant attention due to their impressive emergent capabilities that were not observed in earlier language models. One emerging area where LLMs have been widely used in recent times is the utilization of LLMs as the evaluator of the texts generated by various generative models. In this paper, we also explore the possibility of whether LLMs are reliable in assessing the factual consistency of summaries generated by text generation models. We first propose a new approach to evaluate the factuality score using LLMs by utilizing the same LLM to perform all steps in the question-answering-based factuality scoring pipeline. Subsequently, we study the performance of various LLMs to directly score the factuality. Our evaluation is conducted in traditional benchmarks by comparing their correlation with human annotations. Contrary to expectations, our findings revealed that none of the factuality metrics showed any significant correlations (e.g., coefficient scores greater than 0.3) to human evaluations of factuality for GPT-4, PaLM-2, and Claude-2, with the only exception being GPT-3.5 in two subcategories of factuality. Nonetheless, our findings are consistent across almost all factual error types, suggesting a fundamental limitation in the ability of current LLMs to assess factuality.

## introduction text summarization has significantly advanced through the utilization of pre-trained language models @xcite @xcite @xcite . however, a persistent concern with current models is their frequent inability to maintain factual consistency with the original documents they intend to summarize @xcite . consequently, establishing the factual accuracy of a summary continues to be the key for the evaluation of summa-rization models @xcite . to resolve this issue, recent studies have utilized techniques like natural language inference, questionanswering, or syntactic dependency as factuality evaluation metrics @xcite . however, as highlighted by @xcite , none of these automatic factuality metrics demonstrate a considerable correlation (i.e., fails to achieve a correlation score above 0.3) with human evaluations, pointing to the limited efficacy of these measures. the emergence and subsequent advancements of llms, such as chatgpt foot_0 , have transformed the landscape of natural language processing (nlp). chatgpt-like llms @xcite @xcite have displayed impressive progress across a broad spectrum of nlp tasks, from text classification to generation, language translation, and beyond @xcite . given the capabilities of these llms, our research explores the possibility of utilizing llms for the critical task of factual consistency evaluation @xcite @xcite @xcite . to assess the factual consistency of a model, one common approach is the utilization of a questionanswering (qa) pipeline @xcite . traditionally, the evaluation of factuality using qa systems has involved the use of separate, distinct models for each of the following tasks: answer selection, question generation, and question answering @xcite . however, this approach involves the intricate task of coordinating between these disparate models, potentially resulting in inefficiencies in real-world scenarios. additionally, these models may fail to capture the comprehensive context necessary for optimal factuality evaluation. in response to these challenges, we propose a novel approach that substitutes the separate models with a singular and unified model using llms. in addition, we explore another approach where llms prompt: qa-based factuality metric via llms prompt: llm-based factuality scoring # answer selection and question generation: from the following text, generate a question that can be answered within 1 or 2 words and also generate an answer that is either a noun phrase/named entity. text: tom went to a baseball game tonight. output: { "question": "when did tom go to a baseball game?", "answer": "tonight" } text: @xcite output: were directly asked to assess the factuality of a given summary. meanwhile, we also address the potential risk of inaccurate high correlation measures @xcite by considering partial correlations, which are adept at controlling for confounding variables. in sum, this paper investigates the following research questions (rq): rq 1: can the qa-based factuality metric be improved by utilizing llms? rq 2: can llms directly generate reliable faithfulness scores? meanwhile, one major limitation in factuality evaluation is the lack of common benchmarks. this makes the comparison of various factuality metrics quite difficult. to address this issue, various benchmarks have been introduced recently for factual consistency evaluation, such as sum-meval @xcite and frank @xcite . these benchmarks are designed to evaluate various metrics on their ability to capture factual errors in abstractive summarization. among the available benchmarks, the frank benchmark is the largest one consisting of humanannotated factuality scores of summaries from diverse datasets. more specifically, it is a compilation of two datasets, cnn-dm @xcite and xsum @xcite , amalgamating outputs from nine distinct models across these datasets (5 models for cnn-dm and 4 models for xsum). in total, the dataset comprises 2250 human-annotated judgments on different types of factual errors of model outputs. in addition, this benchmark addresses the false measurement of high correlations in various factuality metrics by introducing the partial correlation coefficients. in this paper, we also utilize the frank benchmark to evaluate the factual consistency of modelgenerated summaries by leveraging llms as the evaluator. our paper diverges from that of @xcite in several key aspects. notably, our research employs the frank dataset, encompassing the cnn-dm and xsum datasets. in contrast, @xcite base their findings on the summeval and newsroom datasets. additionally, our study presents results using partial correlation as opposed to the straightforward correlation employed by @xcite . this metric is adept at controlling for confounding variables, potentially mitigating the risk of inaccurate high correlation measures @xcite . ## methodology in this section, we present our methods: (i) using llms as qa-based factuality metric, and (ii) using llms for direct factuality scoring. below, we first present these methods. in this paper, contrary to the traditional approach of utilizing separate models to perform each task that makes the qa-based factuality evaluation process very complicated, we propose one single llm to be used as the qa-based factuality metric evaluator to perform all steps. for prompt construction, we first evaluate various prompts in some samples and then select the one for our experiment that performs the best. we show our selected prompt for this task that we use in our experiments in in our prompt, we leverage the in-context learning principle and provide an associated example with our prompt to the llms to perform the first two tasks: initial answer selection and question generation. since both the initial answer and the questions are required to be generated from the given summary (making both the question and the answer to have some dependencies between them), we unify these two steps together by asking the llm to generate both the answer and the question simultaneously from the given summary. this makes the first two steps of the qa-based pipeline to be more efficient. afterward, the generated question and the article are given as input to the llm to generate the final answer. the evaluation process of the qa-based factuality metric depends on finding the similarity between the initially selected answer and the final answer. the higher the similarity, the more faithful the summary is being considered. ## experiments in this section, we first present the llms that we study in this paper, followed by defining the evaluation metrics and finally the experimental results. ## conclusion the central objective of this research was to assess the effectiveness of various llms, specifically gpt-3.5, gpt-4, and palm-2 in the evaluation of factuality in text summarization tasks. in addition to directly using llms to evaluate the factuality of a summary, we also introduce a novel approach that utilizes one single llm to perform various steps of the qa-based factuality scoring pipeline. contrary to expectations, our findings revealed that none of the approaches showed a significant correlation (with a coefficient greater than 3) to human evaluations of factuality for most llms, with the only exception happening while directly generating the llm faithfulness scores by gpt-3.5 in two subcategories of factuality: prede and circe. nonetheless, the result is consistent across almost all factual error types, suggesting a fundamental limitation in the ability of current llms to effectively assess factuality. while previous studies, such as @xcite , indicated the potential of automatic metrics in assessing factuality, our findings suggest that it is essential to consider possible dataset biases @xcite . in some contexts, not accounting for confounding variables associated with system and the dataset properties might influence the perceived correlations. to provide a more nuanced perspective, we recommend utilizing partial correlation coefficients to control for these variables. our study calls for an exploration into the inherent deficiencies of current language models in maintaining factual consistency and sheds light on the necessity for developing more accurate and comprehensive models and methods for factuality evaluation. in the future, we will study the factuality evaluation capabilities of llms using other benchmarks @xcite , as well as on noisy datasets @xcite @xcite @xcite , alongside investigating new approaches, such as the utilization of few-shot learning @xcite , other prompting strategies @xcite , and whether fine-tuning opensource llms @xcite for factuality evaluation leads to a better factuality evaluator.