Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends (2024)

Abstract: Recent advancements in large language models (LLMs) have significantly advanced the capabilities of summarization systems.However, they continue to face a persistent challenge: hallucination. While prior work has extensively examined LLMs in news domains, evaluation of dialogue summarization has primarily focused on BART-based models, resulting in a notable gap in understanding LLM effectiveness.Our work seeks to address this gap by benchmarking LLMs for dialogue summarization faithfulness using human annotations,focusing on identifying and categorizing span-level inconsistencies.Specifically, we evaluate two prominent LLMs: GPT-4 and Alpaca-13B.Our evaluation reveals that LLMs often generate plausible, but not fully supported inferences based on conversation contextual cues, a trait absent in older models. As a result, we propose a refined taxonomy of errors, introducing a novel category termed “Contextual Inference” to address this aspect of LLM behavior. Using our taxonomy, we compare the behavioral differences between LLMs and older fine-tuned models. Additionally, we systematically assess the efficacy of automatic error detection methods on LLM summaries and find that they struggle to detect these nuanced errors effectively. To address this, we introduce two prompt-based approaches for fine-grained error detection. Our methods outperform existing metrics, particularly in identifying the novel “Contextual Inference” error type.

## introduction considerable progress has been made in summarization using large language models (llms) @xcite . however, the challenge of so-called "hallucinations", characterized in this context as statements in summaries figure 1: in the example provided, gpt-4 infers that the speakers are discussing "their son." although this inference seems plausible given the circumstantial evidence in the conversation, it lacks direct evidence. in prior research, news articles have been the main testbed for llm-generated summary evaluation @xcite . dialogue summarization remain less explored, with prior works mostly focused on smaller fine-tuned models @xcite @xcite . in this work, we close the evaluation gap, focusing our analysis on llm summaries of chit-chat style dialogues. we obtain fine-grained inconsistency annotations for summaries generated (zero-shot) by two prominent llms (gpt-4 @xcite and alpaca-13b @xcite ) and across two summarization datasets @xcite and @xcite ). in the domain of dialogues, a further gap exists in understanding the differences between summaries generated by llms and those generated by smaller fine-tuned models. in the news domain, prior work has found that llm-generated summaries have fewer inconsistencies @xcite . work done by @xcite , also in the news domain, notes varying error distributions across different model categories. in our work in the dialogue domain, we compare differences in error rates and analyze the categories of errors for summaries of dialogues with fine-tuned models versus summaries with llms. as in the news domain, we find that llm-generated summaries have fewer inconsistencies. surprisingly, our analysis reveals that over 30% of llm-generated summaries contain inconsistencies, contrasting sharply with the inconsistency rate of less than 5% in gpt-generated news summaries @xcite . to further elucidate the differences between llms and fine-tuned models, we annotate spans with error categories. previous work has primarily relied on part-of-speech-based tags for error classification @xcite @xcite . however, complexities inherent in llm-generated summaries, often lengthier and more intricate, do not neatly align with error categories based solely on part of speech, warranting alternative strategies for a more meaningful categorization. hence, our work proposes a refined taxonomy integrating existing error types. we further introduce a new error category specific to llm behavior: "circumstantial inference." this category stems from the observation that llms frequently produce statements that appear plausible based on circumstantial (but not direct) evidence in the dialogues, an aspect hitherto unexplored. in particular, llms tend to produce statements that may be circumstantially implied based on contextual cues in the conversation but not explicitly stated as seen in figure 1 . although these inferences are not directly stated and can be inherently unsupported, they can still be useful in some instances, especially when summarizing ambiguous dialogues. however, the appropriateness of such inferred details varies depending on context and domain, highlighting the need for further investigation. in addition, there is limited understanding regarding the automatic detection of the mentioned error types. therefore, we systematically evaluate the performance of state-of-the-art error detectors on llm-generated dialogue summaries. we also introduce two prompt-based methods for fine-grained error detection, which notably outperform all prior state-of-the-art error detectors, particularly in identifying the newly introduced error type, "circumstantial inference." in summary, our primary contributions are as follows: ## human evaluation: zero-shot prompted dialogue summaries we aim to compare the difference in consistency of zero-shot prompted llm-generated dialogue summaries with smaller fine-tuned model-generated summaries. 2 to accomplish this, we conduct human annotations to identify inconsistent spans generated by both @xcite and alpaca-13b @xcite . specifically, we direct annotators to spot inconsistencies in summaries, marked by spans that lack evidence in the source text or distort information from it. our evaluation is carried out on dialogue summarization datasets previously used for benchmarking finetuned summarization models. ## automatic error detection prior research has demonstrated a shift in error category distributions when summarizing with models of different generations, resulting in varied trends in the performance of automatic error detectors @xcite . this section aims to address the following key questions: a) do factuality metrics perform similarly on summaries generated by large language models (llms) compared to older models (in our study, ft-summ)? specifically, we investigate whether detecting factual errors in llm-generated summaries poses greater challenges. b) which error types across various model categories can factuality metrics identify, and what are the associated failure modes? drawing upon our error taxonomy, we analyze the ability of metrics to detect different error categories, with a specific focus on their performance in identifying circumstantial inferences, an aspect not previously evaluated. ## conclusion in summary, our work is the first to comprehensively assess large language model (llm) performance in dialogue summarization, revealing considerable inconsistencies that underscore the ongoing challenges in this area. our findings emphasize the prevalence of circumstantial inferences, in summaries generated by gpt-4 and alpaca-13b, indicating llms' proficiency in language understanding but their tendency to introduce conceptual inferences. moreover, we demonstrate that existing metrics struggle to detect these nuanced errors effectively. consequently, we advocate for performance evaluations on benchmarks utilizing newer models to better capture the capabilities and limitations of automatic metrics, given the evolving error distributions and types of newer llms compared to ft-summ models. furthermore, our incorporation of two prompt-based methods shows promising progress in identifying circumstantial inference errors, although further research is required to improve performance.