Phrase-Level Localization of Inconsistency Errors in Summarization by Weak Supervision (2022)

Abstract: Although the fluency of automatically generated abstractive summaries has improved significantly with advanced methods, the inconsistency that remains in summarization is recognized as an issue to be addressed. In this study, we propose a methodology for localizing inconsistency errors in summarization. A synthetic dataset that contains a variety of factual errors likely to be produced by a common summarizer is created by applying sentence fusion, compression, and paraphrasing operations. In creating the dataset, we automatically label erroneous phrases and the dependency relations between them as “inconsistent,” which can contribute to detecting errors more adequately than existing models that rely only on dependency arc-level labels. Subsequently, this synthetic dataset is employed as weak supervision to train a model called SumPhrase, which jointly localizes errors in a summary and their corresponding sentences in the source document. The empirical results demonstrate that our SumPhrase model can detect factual errors in summarization more effectively than existing weakly supervised methods owing to the phrase-level labeling. Moreover, the joint identification of error-corresponding original sentences is proven to be effective in improving error detection accuracy.

## introduction the quality, particularly the fluency, of automatically generated abstractive summaries has improved significantly @xcite @xcite with methods that benefit from large-scale pre-trained language models. however, recent studies @xcite have pointed out that more than 30% of the generated summaries are inconsistent with the source documents owing to unintentionally introduced factual errors, which affect the reliability and usability of summarization systems. existing approaches for evaluating inconsistency in summarization can be roughly divided into two categories. one is the unsupervised approach that relies on an external natural language understanding (nlu) system, such as natural language inference (nli) @xcite @xcite or question answering (qa) @xcite @xcite systems, to validate a summary. although this approach can benefit from the ever-progressing nlu technologies, the configuration and performance of an inconsistency detection system would be inevitably constrained by the underlying systems. therefore, we adopt another approach, namely the weakly supervised approach, which employs a synthetic dataset that contains automatically generated errors @xcite . the key to the success of this approach lies in the quality and quantity of the synthetic dataset. it is necessary for such a dataset to contain a variety of errors that are likely to be produced by a common summarization system. however, as argued by @xcite , the error distributions that are produced by existing methods are considerably different from those produced by actual summarization models. figure 1 overviews our proposal for the localization of factual inconsistency errors in summarization. we propose to create a synthetic dataset by applying sentence fusion and compression operations in addition to paraphrasing. thus, recurring summarization errors are expected to be reproduced. the resulting synthetic dataset is then used as weak supervision to train an error localization model called sumphrase. this model jointly localizes errors in a summary and their corresponding sentences in the source document 1 . in the dataset creation process, a single sentence or a pair of sentences is first selected from a source document. a hypothesis sentence is then generated from the selected sentence or sentence pair, which is expected to contain various intrinsic factual errors. thereafter, the hypothesis sentence is labeled at the phrase level, where each label is computed from dependency arc-level labels. finally, the dataset for training the sumphrase model is created by organizing the phrase-level labeled data. furthermore, the detection of factual errors in a summary can be combined with the identification of their corresponding sentences in the source document. this joint approach not only improves the usability of the error detection system but also improves the error detection performance in a multitask learning setting @xcite . the contributions of this study are as follows: ## related work we review the approaches for inconsistency detection in summarization by classifying them into the 1 the dataset and code are available at https:// github.com/taka2946/sumphrase following two categories. ## methodology this section first details the process for creating a synthetic dataset that intentionally accommodates erroneous sentences. our proposed model for jointly detecting errors in a summary and identifying corresponding sentences in the source document is then described. ## experimental setup the cnn/dailymail dataset @xcite was used in the experiments. ## results and discussion this section discusses the efficacy of the proposed method by referring to the results. appendix d presents and discusses some detection results. ## conclusions we propose a neural model called sumphrase for detecting errors in an abstractive summary. the model was empirically proven to be effective as it outperformed other weakly supervised models. this significant performance is primarily attributed to the dataset on which the model was trained. we created a synthetic dataset that contains factual errors likely to be produced by a common summarizer. these errors are labeled at the phrase level, as opposed to the dependency arc-level labels employed by existing models. the synthetic training dataset can also contribute to improving models that rely on an external nli or qa system @xcite . it can also be used to fine-tune these models or to post-edit errors @xcite . the model exhibited improved performance when jointly trained with the sub-task of localizing corresponding sentences in a summary sentence. this functionality may contribute to enhancing the explainability of an inconsistency error-detection system. our method can also contribute to generating negative samples required to train a summarization model that relies on contrastive learning @xcite . the electra-base-discriminator foot_7 , implemented in the huggingface transformers library @xcite . all experiments were conducted using an nvidia geforce gtx titan x gpu with 12 gb of memory. the time required to train the sumphrase model was approximately 40 h. for reliability, each number reported from our implementations in the present study is the average of three runs with different random seeds.