FIZZ : Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document (2024)

Abstract: Through the advent of pre-trained language models, there have been notable advancements in abstractive summarization systems. Simultaneously, a considerable number of novel methods for evaluating factual consistency in abstractive summarization systems has been developed. But these evaluation approaches incorporate substantial limitations, especially on refinement and interpretability. In this work, we propose highly effective and interpretable factual inconsistency detection method FIZZ (Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document) for abstractive summarization systems that is based on fine-grained atomic facts decomposition. Moreover, we align atomic facts decomposed from the summary with the source document through adaptive granularity expansion. These atomic facts represent a more fine-grained unit of information, facilitating detailed understanding and interpretability of the summary’s factual inconsistency. Experimental results demonstrate that our proposed factual consistency checking system significantly outperforms existing systems. We release the code at https://github.com/plm3332/FIZZ.

## introduction with the development of pre-trained language models, abstractive summarization systems using these language models have made remarkable progress in generating fluent and natural summarizations @xcite . however, one of the notable challenges these systems confront is the hallucination, causing language models to generate summaries that are factually inconsistent with the given article @xcite @xcite . recognizing the significance of this issue, various evaluation metrics have been introduced to detect these errors, starting from tra- ## related work summarization factual consistency evaluation a multitude of metrics designed to evaluate summarization factual consistency are currently being refined by leveraging nlp pipelines originally developed for disparate tasks, including qa-based evaluation, parsing-based methods, llm-based prompting, and nli-based metrics. ## fizz the overall flow of our proposed system fizz is presented in figure 2 . our method first begins with the application of a coreference resolution model to a given (document, summary) pair, resulting in a new pair of texts (document, summary) where coreferences have been resolved (section 3.1). fol-lowing this, we proceed to generate atomic facts from the coreference-resolved summary leveraging llms as a zooming-in approach for the summary (section 3.2). using the generated atomic facts, we compute the score of each atomic fact with the nli system (section 3.3). finally, we propose a granularity expansion method, which is a way of zooming out the documents, to compute the score for the summaries that contain high abstractiveness more accurately. ## conclusion in this work, we propose a novel method, fizz, in detecting summary factual inconsistency. our approach decomposes summaries into atomic facts and conducts a sentence-wise comparison with the document, and achieves state-of-the-art performance on the aggrefact benchmark dataset. also, our proposed system has a higher interpretability due to its ability to precisely identify which parts of a summary are factually inaccurate by breaking it down into atomic facts. furthermore, we analyze the necessity and significance of coreference resolution and granularity expansion in the context of summary factual consistency checking.