Factual Error Correction for Abstractive Summaries Using Entity Retrieval (2022)

Abstract: Despite the recent advancements in abstractive summarization systems leveraged from large-scale datasets and pre-trained language models, the factual correctness of the summary is still insufficient. One line of trials to mitigate this problem is to include a post-editing process that can detect and correct factual errors in the summary. In building such a system, it is strongly required that 1) the process has a high success rate and interpretability and 2) it has a fast running time. Previous approaches focus on the regeneration of the summary, resulting in low interpretability and high computing resources. In this paper, we propose an efficient factual error correction system RFEC based on entity retrieval. RFEC first retrieves the evidence sentences from the original document by comparing the sentences with the target summary to reduce the length of the text to analyze. Next, RFEC detects entity-level errors in the summaries using the evidence sentences and substitutes the wrong entities with the accurate entities from the evidence sentences. Experimental results show that our proposed error correction system shows more competitive performance than baseline methods in correcting factual errors with a much faster speed.

## introduction text summarization is a task that aims to generate a short version of the text that contains the important information for the given source article. with the advances of neural text summarization systems, abstractive summarization systems @xcite that generate novel sentences rather than extracting the snippets in the source are widely used @xcite . however, factual inconsistency between the original text and the summary is frequently observed in the abstractive summarization system @xcite @xcite ). as in the example of figure 1 , article: singer-songwriter david crosby hit a jogger with his car sunday evening, a spokesman said. the accident happened in santa ynez, california, near where crosby lives. crosby was driving at approximately 50 mph when he struck the jogger. the posted speed limit was 55. the jogger suffered multiple fractures, and was airlifted to a hospital in santa barbara, clotworthy said.,... system summary with factual error: don clotworthy hit a jogger with his car sunday evening. the jogger suffered multiple fractures and was airlifted to a hospital. ## related work with the advancement of pre-training language models such as bert @xcite and bart @xcite , abstractive summarization systems have adopted these models to use the rich information inherent in parameters. while these models improved the performance, the generated summaries are still often factually inconsistent with the source article. @xcite . to solve the factual inconsistency in abstractive summarization systems, fasum @xcite adopted graph attention network @xcite for generating the correction summary. @xcite studied contrast candidate generation and selection by ranking approach as a modelagnostic post-processing technique to correct the extrinsic hallucinations. another line of mitigating factual errors is to develop a post-editing system to fix the errors. @xcite presented a post-editing corrector module using a bart-based auto-regressive model. the study generated a corrupted summary to train the correction system by substituting the key information, such as an entity or a number, to construct a training dataset. @xcite also develop a seq2seq based error correction system in the claim of fever dataset @xcite by correcting the words after masking some words. different from seq2seq based previous works, we develop a faster retrieval based factual error correction system that does not generate the whole summary, only corrects the entity-level errors by substituting them with one of the entities in the article. ## experiments to evaluate the performance of the proposed error correction system, we measure the success rate of correction for the systems by comparing the correction results with the ground-truth summaries or conducting a human evaluation for the corrected summaries as follows. ## conclusion in this paper, we proposed an efficient factual error correction system rfec based on two retrieval steps. rfec first retrieves evidence sentences based on textual similarities between the summary and the article for detecting and correcting factual errors. then, if there is an entity that is a cause of factual errors, rfec substitutes it with one of the entities in the evidence sentences as a retrieval-based approach. experiments on two benchmark datasets demonstrate that our proposed method shows competitive results compared to example 1) -success evidence sentences: her husband, philippe braham, was one of 17 people killed in january's terror attacks in paris. one month after the terror attacks in paris, a gunman attacked a synagogue in copenhagen, denmark, killing dan uzan, who was working as a security guard for a bat mitzvah party.