T ofu E val: Evaluating Hallucinations of LLM s on Topic-Focused Dialogue Summarization (2024)

Abstract: Single document news summarization has seen substantial progress on faithfulness in recent years, driven by research on the evaluation of factual consistency, or hallucinations. We ask whether these advances carry over to other text summarization domains. We propose a new evaluation benchmark on topic-focused dialogue summarization, generated by LLMs of varying sizes. We provide binary sentence- level human annotations of the factual consistency of these summaries along with detailed explanations of factually inconsistent sentences. Our analysis shows that existing LLMs hallucinate significant amounts of factual errors in the dialogue domain, regardless of the model’s size. On the other hand, when LLMs, including GPT-4, serve as binary factual evaluators, they perform poorly and can be outperformed by prevailing state-of-the-art specialized factuality evaluation metrics. Finally, we conducted an analysis of hallucination types with a curated error taxonomy. We find that there are diverse errors and error distributions in model-generated summaries and that non-LLM based metrics can capture all error types better than LLM-based evaluators.

## introduction recently, the field of automated text summarization has been increasingly inclined toward using large language models (llms) to evaluate model outputs @xcite @xcite . given the consequential nature of this trend, we ask: are llms up to the task of evaluating model outputs? while recent work on news summarization has shown that llms' performance at evaluating the factual consistency of generated news summaries is promising (luo et al., 2023; 1 we release the benchmark dataset with expert annotations at github.com/amazon-science/tofueval. ## related work factual consistency evaluation benchmarks in text summarization, there have been significant efforts to collect human-annotated data for assessing the effectiveness and correlation of different evaluation metrics with human judgments in detecting hallucianted contents in generated summaries @xcite @xcite ). 2 our proposed benchmark tofueval aligns with these efforts but differs from prior work as follows (summarized in detecting hallucinations common automatic metrics for text summarization such as rouge @xcite , bleu @xcite , and bertscore @xcite have poor correlations with human judgement on factual consistency @xcite @xcite . therefore, a few non-llm-based metrics have been developed to detect factuality errors @xcite @xcite @xcite . more recently, llms have been shown to have superior zero-shot performance at factual consistency evaluation under certain evaluation settings, highlighting their potential as state-of-the-art factual consistency evaluators @xcite . as hallucinations from more recent models are harder to detect @xcite , we re-evaluate non-llm-based and llm-based factuality metrics using llm-generated summaries within the context of our dialogue summarization benchmark to-fueval. we find that non-llm-based metrics can surpass most llm-based evaluators. nevertheless, all automated factuality metrics still perform quite poorly, underlining the challenging nature of the problem and the substantial room for improvement in automated factual inconsistency detection. ## tofueval benchmark our topic-focused dialogue summarization benchmark tofueval is constructed as follows: (1) sample documents from two publicly available dialogue summarization datasets (section 3.1); (2) create a variety of topics for sampled documents (section 3.2); and (3) generate topic-focused summaries with various llms (section 3.3). the resulting benchmark contains 100 dialogues and 15 llm-generated summaries per dialogue; (4) lastly, we provide fine-grained human annotations on the topics and the generated summaries for dimensions including factual consistency, relevance, and completeness (section 3.4). the dataset construction pipeline is illustrated in figure 1 . ## results: llms as summarizers we show the error rate in generated summaries in we show the error rates for main-topic summaries separately from those for marginal-topic summaries. we highlight the lowest and second lowest error rates. see examples of annotated summaries in figure 2 : error distribution over factually inconsistent summary sentences for tofueval (left) and for each summarizer over main/marginal topics (right). see error distributions over all summary sentences for each summarizer over main/marginal topics in appendix figure 5 . llms we studied make a significant amount of factual errors, especially on the summary level. we further investigate the distribution of different hallucination types in tofueval with our curated error taxonomy. note that our taxonomy closely resembles that of @xcite , which is based on the samsum dialogue dataset @xcite . due to the complexity of the long dialogues in tofueval, we extend the taxonomy of @xcite with new error types, such as reasoning error and stating opinion as fact. a summary of our curated error taxonomy for the benchmark is provided in figure 3 . we leverage the error taxonomy to enrich all binary factual inconsistency annotations in the benchmark. additional details about the taxonomy curation process and error-type annotation can be found in appendix g. llms tend to produce more factually inconsistent summaries when prompted to focus on a marginal topic, especially with extrinsic information error. as shown in figure 2 , when prompting models for summaries about marginal topics, all summarizers generate significantly more extrinsic information. we find that when the topic is barely mentioned in the document, models try to rely on their knowledge to make informed inferences about the topic, bringing unsupported information into the summary. an exception is gpt-3.5-turbo, which generates far fewer extrinsic information for marginal topics. compared to other summarizers that generate unsupported sentences, we find that gpt-3.5-turbo often handles requests for marginal-topic summaries by including off-topic content or occasionally explicitly saying, "the topic is not discussed in the document". 5 more findings can be found in appendix d. ## results: llms as evaluators we now move on to consider the use of llms as evaluators of factual consistency rather than as summarizers. we first present an evaluation of their performance at making binary factual consistency predictions for both summaries and summary sentences (section 5.1). we then provide an error-type analysis, investigating which error types models fail to detect well (section 5.2). finally, given that llms have the ability to generate critiques definition example explanation extrinsic information error type reasoning error tense/modality error stating opinion as fact nuanced meaning shift contradiction mis-referencing a property or an event in the summary sentence can be found in the source material, but are associated with the wrong entity the summary sentence entails a proposition that's mentioned in the source material not as a fact, but as someone's opinion the summary sentence makes one or more wrong inferences from information in the source document the tense or modal (e.g. can, may, must) used in the summary sentence does not match the tense/modality of the source document the summary sentence contradicts the source material the summary sentence twists information from the source material in a subtle way the current fleet is over budget and overdue, with additional requirements added over time. government intervention may be needed as airlines have forfeited the right to selfregulate. general motors, ford, and daimler chrysler are planning to eliminate a combined total of 300,000 jobs by 2008. a judge has allowed thousands of students who did not pass to potentially graduate. some airlines argue that factors like weather and labor problems are beyond their control, but experts disagree. the trial of dr. conrad is set to begin on monday. it is not the current fleet that is "over budget and overdue", but the one they are intending to replace the current fleet with. it is the opinion of trippler, and not a fact, that the airlines have "forfeited" the right to self-regulate. 30,000 (general motors) + 30,000 (ford) + 40,000 (daimler chrysler) = 100,000 jobs, not 300,000 jobs. per source, the ruling issued by the judge could allow student to graduate, not has allowed. experts don't disagree that weather is beyond the airlines' control. the trial is set to resume, not begin on monday. president obama has called for reforms in the procurement process. the summary sentence contains new information not grounded in the source document the document does not explicitly mention that president obama has called for reforms in the procurement process. of model output and provide explanations for their factual consistency judgments @xcite , we consider the accuracy of model-generated explanations by comparing them to human-written explanations (appendix h). evaluator selection for a comprehensive comparison, we include the following non-llm based sota factuality metrics: summac-zs, summac-cv @xcite , qafacteval @xcite , and alignscore @xcite . we also include the following proprietary and open-source llms as factual consistency evaluators: (1) gpt-4 (openai, 2023); (2) gpt-3.5-turbo; (3) vicuna-13b/33b @xcite ; (4) wizardlm-13b/30b @xcite . for all llms in the aforementioned list we used a zero-shot configuration, as certain llms in this list are unable to accommodate few-shot evaluations due to input length constraints. in any case, it has been observed that few-shot examples do not consistently yield superior outcomes in comparison to zero-shot scenarios @xcite . more details about model selection are in appendix b. ## conclusion we have proposed a new factual consistency evaluation benchmark tofueval for topic-focused dialogue summarization with llm-generated sum-maries. we find that llms serving as summarizers make numerous and diverse hallucinations in their summaries. furthermore, by measuring balanced accuracy and analyzing models' error types, we conclude that it is still challenging for both llm-based evaluators and existing state-of-the-art non-llm based factuality metrics to detect a wide range of hallucinations in dialogue, although the latter exhibit slightly better performance.