Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization (2025)

Abstract: Detecting factual inconsistency for long document summarization remains challenging, given the complex structure of the source article and long summary length. In this work, we study factual inconsistency errors and connect them with a line of discourse analysis. We find that errors are more common in complex sentences and are associated with several discourse features. We propose a framework that decomposes long texts into discourse-inspired chunks and utilizes discourse information to better aggregate sentence-level scores predicted by NLI models. Our approach shows improved performance on top of different model baselines over several evaluation benchmarks, covering rich domains of texts, focusing on long document summarization. This underscores the significance of incorporating discourse features in developing models for scoring summaries for long document factual inconsistency.

## introduction current state-of-the-art summarization systems can generate fluent summaries; however, their ability to produce factually consistent summaries that adhere to the source content or world knowledge remains questionable. this phenomenon is known as factual inconsistency, one type of "hallucination" problem @xcite @xcite @xcite . a rigorous line of research approaches this problem by developing models to detect unfaithful summary content, including utilizing pre-trained models such as natural language inference (nli) @xcite @xcite and question answering (qa) @xcite models. such approaches are tested on rich benchmark datasets, such as true @xcite , summac @xcite , and aggrefact @xcite , etc. however, such benchmark datasets only include short documents (< 1000 words) and summaries with a few sentences. while the methods mentioned above perform well with short texts, they struggle with longer documents @xcite . recent nli work addresses this by selecting the input and breaking down the summary. lengthy summaries are split into individual sentences or more minor atomic claims, while small chunks of the source document are extracted as premises. this approach reduces the task to multiple short evaluations, which are then aggregated to provide a summary-level label @xcite @xcite . out of the existing nli-based methods, align-score demonstrated superior performance on multiple benchmarks. it breaks the input document into continuous chunks of text to tackle the input restriction. however, this exhaustive approach may break the structure of the context (section and paragraph split), thus reducing the chances that the summary sentence can be correctly verified with its factual consistency. on the other hand, most factuality evaluation metrics aggregate the sentencelevel aligning scores through averaging or selecting the minimum, disregarding that sentences are not equally important @xcite . for instance, people can remember the big picture more easily but struggle to retain low-level details when retelling a story. the natural questions would be: do system-generated summaries carry a similar pattern? if so, how can we utilize the text organization information to help detect the inconsistencies between the summary and the source document? in this work, we study the factual inconsistency problem through the lens of discourse analysis. by analyzing the structure (here we use rhetorical structure theory (rst) @xcite ) of the original articles and the summaries, we uncover the importance of preserving the article structure and studying the connections between discourse structure and the factual consistency of model-generated summaries. our analysis shows that complex sentences built by multiple elementary discourse units (edus, the basic units used in the discourse theory) have a higher chance of containing errors, and we also find several discourse features connected to the factual consistency of summary sentences. motivated by the analyses mentioned above, we propose a new evaluation method, structscore, based on the nli-based approaches to better detect factual inconsistency. our algorithm includes two steps: (1) leveraging the discourse information when aggregating the sentence-level alignment scores of the target summary and (2) decomposing the long input article into multiple discourseinspired chunks. we tested our proposed approach on multiple document summarization benchmarks, including aggrefact-ftsota split, diver-summ, longsciverify, longeval, and a nonscientific domain dataset legalsumm with a focus on long document summarization. our proposed approach obtained a performance gain on multiple tasks. 1 to sum up, two research questions are addressed: 1. how and what discourse features are connected to the factual inconsistency evaluation? 2. can our discourse-inspired approach improve the detection performance on long document summarization? ## datasets this section describes the datasets used to explore our research questions. we begin with the discourse analysis dataset, which includes sentencelevel fine-grained labels of errors introduced in @xcite , enabling systematic analysis of the relationships between different features and their labels. we then discuss the benchmark datasets, which provide summary-level labels in either binary or continuous scores, and evaluate our approach and baselines on them. ## discourse analysis preliminaries discourse analysis with rhetorical structure theory (rst) is helpful for different downstream tasks, such as argument mining @xcite ), text simplification @xcite , ai-generated text detection @xcite , and summarization @xcite . rst predicts tree structures on the grounds of underlying coherence relations that are primarily defined in speaker intentions @xcite . the discourse tree comprises lower-level elementary discourse units (edus), each corresponding to a phrase within a sentence. these units are then integrated into more complex structures, such as sentences and paragraphs, to form the full discourse tree. discourse labels (i.e., elaboration, contrast, condition, etc.) are assigned as the relation between nodes. additionally, a nuclearity attribute is assigned to every node of the discourse tree, aiming to encode the relative importance between the pairs of sub-trees (nucleus roughly implying primary importance and a satellite means supplemental). we first parse the summaries from the datasets as mentioned earlier in section 3 with an opensourced dmrst model @xcite , following similar work which utilizes the same model for discourse parsing @xcite @xcite . in the following paragraphs, we propose and verify multiple hypotheses that inspired our discourse-structure-aware factual inconsistency detection approach. figure 1 summawe first conduct discourse analysis on parsed summary sentences ( §4.1) and exploit the source document's discourse structure ( §4.2). motivated by the findings, our proposed approach is introduced in §5.2 and §5.1. ## structscore in this section, we describe the structscore framework. the lower right part of figure 1 presents motivations for each module. ## experimental details we adapt mainstream evaluation setups for each benchmark. for diversumm, we apply an 80/20 test/dev split by stratifying the labels for each subtask. for aggrefact, we use their released val/test split. for longsciverify, longeval and legalsumm, we use them as test sets. baselines one of our baselines is alignscore @xcite , an nli-based metric that computes the aggregated inference score between a source article and generated summaries. we included infuse @xcite , which sets the sota on diversumm, minicheck ft5 (minicheck-flant5 checkpoints) @xcite ) that is a best-performing non-llm factchecker over multiple benchmarks, and long-docfactscore @xcite which claimed to work well on factuality validation of lengthy scientific article summaries. our experiment notes that minicheck did not work well over long summaries due to its design objectives of short-statement fact-checking. we thus introduce mc-ft5 (sent), which computes the individual summary sentences' scores using minicheck and reports their average as the final summary score. we additionally include the gpt4o (openai et al., 2024) as the llm fact-checker, using a prompt adopted from minicheck-7b (bespoke-mc-7b) 9 , the sota fact-checking model on the llm-aggrefact benchmark @xcite . unless otherwise noted, we reran the baseline models on our datasets using the original authors' released code and checkpoints. implementation details are provided in appendix e. our approach we re-utilized baseline models to compute the scores between context chunks and summary sentences, including alignscore @xcite , minicheck-ft5 (sent) and in-fuse @xcite , and experimented with below settings to apply our proposed approaches: the reweighting and segmentation can not be applied to longdocfactscore, as it produced negative scores on all enumeration of source-target sentence pairs, which does not utilize the structural information. infuse utilizes the ranked list of entailment scores for all document sentences associated with each summary sentence. thus, the segmentation approach does not affect. ## results overall performance rq1. we observe that the re-weighting algorithm improves prediction performance on different baselines (rows 5-6, 11-12, 17-18). for long source documents, the re-weighting approach consistently improves or closely matches gov, axv, csm splits in diversumm and the axv split in longsciverify (lsv-axv) and legals performance. noticeably, alignscore with reweighting scored the best on legals. on the other hand, for both xsm and cnd in aggrefact-ftsota, the re-weighting algorithm does not help much. we posit that the short summary length (1-3 sentences) has minimally structured information, so the scores will not change much. for mnw and qms, the short summaries in qms (averaging 3 sentences) reduce the effectiveness of the re-weighting algorithm. moreover, mnw's non-factual sentences often receive high prediction scores, which our re-weighting approach tends to amplify, leading to a drop in performance. we also observe a slight performance drop on lsv-pub and longeval-pub for alignscore and in-fuse, potentially due to the different document structure of scientific articles from the medical domain. these observations also suggest potential future work for a dynamic weighting algorithm based on the document structure and domain knowledge. in rq3. combining both approaches is not universally beneficial across all scenarios. when both individual approaches contribute positively, the combined structs generally achieves better performance, as seen in row 8 on axv, csm, and row 14 on axv. however, when one component causes a performance drop, combining both often leads to weaker overall performance than the stronger component alone. for instance, on gov, row 8 performs worse than row 5, likely due to the segmentation in row 7, making the model less accurate. similarly, row 14 performs slightly better than row 11 on lsv-pub, but row 13's improvement does not translate into better performance gains when combined with row 12. differences in evaluation metrics (auc vs. correlation) and dataset sizes may also have influenced these outcomes (i.e., row 14 does not improve much on longeval-pub while rows 12 and 13 have larger gains). ## conclusion in this work, we approach the factual inconsistency detection of long document summarization through the lens of discourse analysis. we find that discourse factors, with regard to sentence structure, are related to the factual consistency of sentences. we further propose a framework that leverages the source document structure and introduces re-weighting the sentence-level predictions on top of different nli-based models, achieving performance gains across multiple long-document summarization evaluation datasets, including scientific articles and legal documents.