Interpretable Automatic Fine-grained Inconsistency Detection in Text Summarization (2023)

Abstract: Existing factual consistency evaluation approaches for text summarization provide binary predictions and limited insights into the weakness of summarization systems. Therefore, we propose the task of fine-grained inconsistency detection, the goal of which is to predict the fine-grained types of factual errors in a summary. Motivated by how humans inspect factual inconsistency in summaries, we propose an interpretable fine-grained inconsistency detection model, FineGrainFact, which explicitly represents the facts in the documents and summaries with semantic frames extracted by semantic role labeling, and highlights the related semantic frames to predict inconsistency. The highlighted semantic frames help verify predicted error types and correct inconsistent summaries. Experiment results demonstrate that our model outperforms strong baselines and provides evidence to support or refute the summary.

## introduction prior work @xcite @xcite formulates the problem of factual inconsistency detection as a binary classification task, which predicts whether a summary is consistent with the source document. however, these approaches have two drawbacks. first, they cannot predict the types of factual errors made by a summary and thus provide limited insights into the weakness of summarization systems. although recent studies @xcite @xcite have manually inspected the types of factual errors in summaries, there is no existing work on automatic detection of fine-grained factual inconsistency. second, existing models typically cannot explain which portions of the document are used to detect the inconsistency in the input summary. in order to verify and correct an inconsistent summary, humans still need to read the entire source document to find the supporting evidence. @xcite introduce an auxiliary task to extract the supporting spans in the document for inconsistency detection, which requires expensive ground-truth labels of supporting spans. to address the first limitation, we propose the fine-grained factual inconsistency detection task. the goal is to predict the types of factual inconsistency in a summary. we show examples of different factual error types in to solve the second challenge, we further introduce an interpretable fine-grained inconsistency detection model (finegrainfact) that does not require any label of supporting text spans, inspired by how humans verify the consistency of a summary. when humans annotate the factual error types of a summary, they first identify facts in the document that are relevant to the summary and then determine the factual error types in the summary. following this intuition, our model first extracts facts from the document and summary using semantic role labeling (srl). we consider each extracted semantic frame as a fact since a semantic frame captures a predicate and its associated arguments to answer the question of "who did what to whom". after fact extraction, a document fact attention module enables the classifier to focus on the facts in the document that are most related to the facts in the summary. by highlighting the facts in the document with the highest attention scores, our model can explain which facts in the document are most pertinent to inconsistency detection. experiment results show that our model outperforms strong baselines in detecting factual error types. moreover, the document facts highlighted by our model can provide evidence to support or refute the input summary, which can potentially help users to verify the predicted error types and correct an inconsistent summary. ## task definition the goal of the fine-grained inconsistency detection task is to predict the types of factual errors in a summary. we frame it as a multi-label classification problem as follows. given a pre-defined set of l factual error types {e 1 , . . . , e l }, a document d, and a summary s, the goal is to predict a binary vector @xmath0, 1} l where each element y i indicates the presence of one type of factual errors. we follow the typology of factual error types proposed by @xcite , which include intrinsic noun phrase error, extrinsic noun phrase error, intrinsic predicate error, and extrinsic predicate error. the definitions and examples of these error types are presented in ## our finegrainfact model the model architecture is illustrated in figure 1 . ## related work factual consistency metrics. qa-based consistency metrics @xcite @xcite involve generating ques-tions from the given document and its summary, and then comparing the corresponding answers to compute a factual consistency score. entailmentbased consistency metrics @xcite @xcite utilize a binary classifier to determine whether the contents in a system summary are entailed by the source article. in contrast, our model is a multi-label classifier that detects the types of factual errors in a summary. moreover, our model leverages srl to encode the facts in the input document and summary, enabling users to interpret which facts in the document are most relevant to the inconsistency detection. ## conclusion in this paper, we present a new task of fine-grained inconsistency detection, which aims to predict the types of factual inconsistencies in a summary. compared to the previous binary inconsistency detection task, our new task can provide more insights into the weakness of summarization systems. moreover, we propose an interpretable finegrained inconsistency detection model, which represents facts from documents and summaries with semantic frames and highlights highly relevant document facts. experiments on the aggrefact-unified dataset show that our model can better identify factual error types than strong baselines. furthermore, results on the fever 2.0 dataset validate that the highlighted document facts provide evidence to support the inconsistency prediction.