Summary Factual Inconsistency Detection Based on LLM s Enhanced by Universal Information Extraction (2025)

Abstract: Automatic text summarization has a potential flaw that affects the factuality of summaries. Recently, Large Language Models (LLMs) have been introduced as detectors for factual inconsistencies in summaries. However, LLM-based methods rely on reasoning capabilities and face challenges in terms of efficiency and explainability. We focus on decoupling LLMs’ information extraction and reasoning capabilities to address prominent challenges, and propose a novel framework, UIEFID (Universal Information Extraction-enhanced Factual Inconsistency Detection). Our idea is to define a self-adaptive structured schema to guide fine-tuned LLMs in extracting unified structured information from documents and summaries, ultimately detecting the origins of inconsistencies in extraction information. The evaluation on 5 open-source models shows that UIEFID not only enhances the detection accuracy on the AGGREFACT benchmark but also significantly reduces redundant reasoning.

## introduction automatic text summarization compresses extensive source documents into brief summaries, covering diverse content like news articles, source codes, and cross-lingual text @xcite . previous summarization models are categorized into two main paradigms @xcite . extractive models select and combine significant sentences or passages, ensuring faithfulness but potentially lacking coherence and conciseness. abstractive models generate novel content, producing more natural and fluent summaries but risking faithfulness issues. recently, large language models (llms) have demonstrated remarkable capability and open new possibilities for enhancing abstractive summarization @xcite . nevertheless, llms cannot fully guarantee the faithfulness of generated summaries. in response to factual inconsistency in abstractive summarization, leveraging llms as detectors is gaining increasing popularity @xcite . however, two main challenges exist in llm-based methods: (1) efficiency. current methods heavily rely on the reasoning capability of llms. this implies that whether these inconsistencies are explicit or implicit, llms typically use extensive reasoning to analyze the discrepancies between summaries and source documents @xcite . especially for long documents, llms struggle to deliver detection results concisely. recent research @xcite has demonstrated that llms excel in processing structured information. we focus on decoupling llms' information extraction and reasoning capabilities to address prominent challenges. by introducing universal information extraction (uie) @xcite , we facilitate llms to identify patterns, extract key information, and transform unstructured textual data into structured data for more refined detection. consequently, we propose a novel detection framework, uiefid (universal information extraction-enhanced factual inconsistency detection). our framework obtains a fine-tuned llm with excellent information extraction capability, enabling it to extract corresponding structured information from both summaries and documents efficiently. by analyzing the discrepancies within structured information, llm can pinpoint and annotate the origins of inconsistencies, subsequently feeding them back into further refined reasoning to retrieve substantiating evidence from the document to validate inconsistencies. our work emphasizes the information extraction capabilities of llms rather than relying solely on their zero-shot or few-shot reasoning capabilities. by leveraging uie, our approach can effectively extract structured information that facilitates easy comparison and verification while directing the trajectory and scope of reasoning. this enhances both the efficiency and explainability of the detection. our contributions are threefold: