Entity-level Factual Consistency of Abstractive Text Summarization (2021)

Abstract: A key challenge for abstractive summarization is ensuring factual consistency of the generated summary with respect to the original document. For example, state-of-the-art models trained on existing datasets exhibit entity hallucination, generating names of entities that are not present in the source document. We propose a set of new metrics to quantify the entity-level factual consistency of generated summaries and we show that the entity hallucination problem can be alleviated by simply filtering the training data. In addition, we propose a summary-worthy entity classification task to the training process as well as a joint entity and summary generation approach, which yield further improvements in entity level metrics.

## introduction many recent advances in deep neural networks have led to significant improvement in the quality of abstractive summarization @xcite @xcite . despite this progress, there are still many limitations facing neural text summarization @xcite , the most serious of which is their tendency to generate summaries that are not factually consistent with the input document; a factually consistent summary only contains statements that can be derived from the source document. recent studies show that about 30% of the summaries generated by neural network sequence-to-sequence models suffer from fact fabrication @xcite . unfortunately, the widely used rouge score is inadequate to quantify factual consistency @xcite . factual inconsistency can occur at either the entity or the relation level. at the entity level, a model generated summary may contain namedentities that never appeared in the source document. we call this the entity hallucination problem. for example, consider the following model generated summary: people in italy and the netherlands are more likely to consume fewer cups of coffee than those in the uk, a study suggests. in this paper, we propose a set of simple metrics to quantify factual consistency at the entitylevel. we analyze the factual quality of summaries produced by the state-of-the-art bart model @xcite on three news datasets. we then propose several techniques including data filtering, multi-task learning and joint sequence generation to improve performance on these metrics. we leave the relation level consistency to future work. ## related work large transformer-based neural architectures combined with pre-training have set new records across many natural language processing tasks @xcite @xcite . in particular, the bart model @xcite has shown superior performance in many text generation tasks including abstractive summarization. in contrast to encoder-only pre-training such as in bert @xcite or decoder-only pre-training such as in gpt-2 @xcite , bart is an encoder-decoder transformer-based neural translation model jointly pre-trained to reconstruct corrupted input sequences of text. several authors have pointed out the problem of factual inconsistency in abstractive summarization models @xcite @xcite . the authors in @xcite proposed to train a neural network model to classify if a summary is factually consistent with a given source document, similar to a natural language inference task. in the dialogue generation setting, authors in @xcite proposed using unlikelihood to surpress logically inconsistent responses. our work is complementary to such existing approaches as we focus on simple entity-level metrics to quantify and improve factual consistency. our goal of improving entity-level metrics of summaries is also related to controllable abstractive summarization @xcite , where a list of named-entities that a user wants to see in the summary can be passed as input to influence the generated summary. in contrast, our goal is to predict which entities are summary-worthy while generating the summary that contains them. in this view we are trying to solve a more challenging problem. ## entity-level factual consistency metrics we propose three new metrics that rely on off-theshelf tools to perform named-entity recognition (ner). 1 we use n (t) and n (h) to denote the number of named-entities in the target (gold summary) and hypothesis (generated summary), respectively. we use n (h∩s) to denote the number of entities found in the generated summary that can find a match in the source document. if a named-entity in the summary consists of multiple words, we consider it a match as long as any ngram of the named-entity can be found in the source document. this is meant to capture the situation where the named-entity can be shortened; for example, "obama " is a match for "barack obama" and "harvard" is a match for "harvard 1 we use spacy @xcite . university". when the match is at the unigram level, we make sure that it is not a stop word such as "the". we also make the match case-insensitive to accommodate casing variances. ## joint entity and summary generation: we also explore another generative approach to promote entity-level precision and recall metrics. in particular, instead of just generating the summary, we train the bart model to generate the sequence of summary-worthy named-entities, followed by a special token, and then the summary. we call this approach jaens (join salient entity and summary generation). similar to the multitask learning approach discussed earlier, jaens encourages the model to jointly learn to identify the summary-worthy named-entities while learning to generate summaries. since the decoder generates the salient named-entities first, the summaries that jaens generate can further attend to these salient named-entities through decoder self-attention.