X - FACTOR : A Cross-metric Evaluation of Factual Correctness in Abstractive Summarization (2022)

Abstract: Abstractive summarization models often produce factually inconsistent summaries that are not supported by the original article. Recently, a number of fact-consistent evaluation techniques have been proposed to address this issue; however, a detailed analysis of how these metrics agree with one another has yet to be conducted. In this paper, we present X-FACTOR, a cross-evaluation of three high-performing fact-aware abstractive summarization methods. First, we show that summarization models are often fine-tuned on datasets that contain factually inconsistent summaries and propose a fact-aware filtering mechanism that improves the quality of training data and, consequently, the factuality of these models. Second, we propose a corrector module that can be used to improve the factual consistency of generated summaries. Third, we present a re-ranking technique that samples summary instances from the output distribution of a summarization model and re-ranks the sampled instances based on their factuality. Finally, we provide a detailed cross-metric agreement analysis that shows how tuning a model to output summaries based on a particular factuality metric influences factuality as determined by the other metrics. Our goal in this work is to facilitate research that improves the factuality and faithfulness of abstractive summarization models.

## introduction in this work, we consider the task of automatic text summarization, i.e., the task of generating a concise summary of a given document that preserves its most salient information @xcite . this is a prominent task that has been applied across a variety of contexts and domains, such as the summarization of legal documents @xcite , automatic news summarization @xcite , and summarization of dialog between agents and clients @xcite . article: kremlin spokesman dmitry peskov said on thursday that the russian air force would continue its support of the syrian armed forces. he also urged washington to deliver on a pledge to separate moderate syrian opposition fighters from "terrorists" . . . the russian foreign ministry said a us refusal to co-operate would be a gift to "terrorists". the us and russia have been negotiating for months to try to secure a cessation of hostilities but the latest truce collapsed last week after only a few days and attacks on eastern aleppo have since intensified . . . bart: russia has said it will continue to support the syrian government despite a partial truce collapsing. rerank (factgraph): russia has said it will continue its air strikes in syria despite a partial truce collapsing. figure 1 : factual hallucinations on an xsum article by bart @xcite summarization model which are not supported by the article. here, we show the result of reranking, one of our factuality-aware methods, on this example that removes the hallucinations making the summary factually consistent with the article. most modern approaches to automatic text summarization can be characterized as either (a) extractive in that they select a subset of sentences from the longer source article to serve as the summary, or (b) abstractive in that they generate a summary that includes sentences and phrases not occurring in the source article. abstractive summarization is generally considered more challenging, as the aspect of summaries containing information outside their source articles adds complications to both training and automatic evaluation. one of the most notable challenges faced when designing abstractive summarization models is their proclivity for generating factual errors @xcite , i.e., for generating summaries that do not agree with the original document's text. the fact that these models struggle with factuality is somewhat unsurprising, as the lexical overlap metrics (e.g., rouge @xcite , bleu @xcite ) frequently used to evaluate and tune these models only compute the co-occurrence of n-grams between the reference accept as output factgraph 7.3 factgraph 92.4 . . . . . . and generated summary texts, and thus do not explicitly measure for factual correctness @xcite . in service of this issue, a large number of recent works have proposed metrics that are specifically targeted towards evaluating and improving the factuality of abstractive summarization @xcite @xcite @xcite . however, it is unclear how the factuality metrics introduced by these works agree with each other and how tuning summary generation with a particular metric influences how the summary is scored by other metrics. therefore, in this work, we perform a more comprehensive and unified analysis of how these metrics can be used to improve factual consistency and how they interact with one another. our analysis is performed on seven of the recent factuality metrics. we leverage the following three different strategies for fact-aware summarization using these metrics: (a) fact-aware training: a methodology to select factually consistent training data to fine-tune a summarization model; (b) posthoc reranking: reranks the output summaries with respect to the factuality metrics; and (c) fact-aware corrector: leverages the scores from the metrics to train a seq2seq model that modifies a hallucinated summary to a factual text generation output. while previous works have performed some of the above fact-aware methods usually on a single metric, this work provides an extensive and unified analysis of these methods optimized for various factuality metrics. in this work, we are primarily interested to understand how optimizing fact-aware summarization methods for each of these metrics affects the performance of other lexical and factual evaluation metrics and how that can be used to design an abstractive summarization pipeline that outperforms recent fact-aware summarization baselines. figure 1 gives an illustrative example. our contributions in this paper are as follows: (i) we present a comprehensive analysis of factaware summarization methods optimized for various factuality metrics and their impact on other metrics which has not been studied by previous works; (ii) from our analysis, we find that optimizing the method for lexical overlap metrics does not correlate well with factuality scores, whereas, optimizing for one of the factuality metrics can show gains for other factuality based metrics. ## fact-aware summarization in this section, we detail the three methods we use to optimize for each of the factuality metrics and in turn for analyzing the cross-metric agreement. figure 2 illustrates the three methods of fact-aware training (facttr), post-hoc reranking (rerank), and hallucination corrector (corrector). ## experimental results in this section, we outline the dataset we used, our evaluation metrics, and experimental results. ## related works abstractive summarization has been an important task in evaluating models' abilities to understand language @xcite . recently, multiple studies have shown that the quality of the summaries being generated has issues with factuality and being faithful to the source article. in other words, the recent summarization models hallucinate content that is mostly erroneous (almost 70%) in the context of the source article @xcite @xcite . such insights into the factual content in the summaries of models have opened up research efforts both in terms of developing models @xcite and improving existing evaluation metrics to capture factuality and faithfulness as a measure to improve the models @xcite @xcite . @xcite also study re-evaluation of factuality scores for abstractive summarization. models and approaches in improving factuality and faithfulness in summarization can be categorized into (a) post-hoc reranking-based approaches: topk summaries (modified) from the existing models are re-ranked based on models that ensure factuality and faithfulness @xcite @xcite from the summarization model is used to generate candidates by replacing entities (or quantities) of the same type mentioned in the original article. these candidate summaries are them re-ranked using multiple different models such as (a) classification models such as nli model @xcite and bart + linear layers @xcite ; (b) span prediction @xcite or sequence labeling models @xcite . in this category of approaches, @xcite and @xcite do not show any significant improvements in correcting factual errors, whereas the ones that generate candidates based on entities mentioned in the original article do have improvement with different factuality measures, with negative or no impact on the rouge scores. graph-based methods: this category of approaches includes not only the text of the article and the summary but a graph representation of the text using dependency parses and semantic parses such as openie @xcite and abstract meaning representation (amr) @xcite . specifically @xcite transforms the text summarization to graph summarization where the models are trained using amrs (the graph structure) of the source article and the summaries. ftsum @xcite uses triples extracted from text using openie and dependency parser as additional information for summarization. the model includes a dual encoder that encodes text and triples with the decoder using an attention mechanism between them to output the summary. openie is also used by fasum @xcite , where the extracted triples are used to form a graph structure that is encoded using graph attention networks. the model is built using a transformer where the decoder computes cross-attention over the nodes' embeddings from the knowledge graph in conjunction with the cross-attention over the encoder's embeddings. this methodology also includes a post-hoc fact corrected trained on synthetically generated data with entity replacement similar to post-hoc approaches @xcite . loss-based methods: the final category of methods focuses on addressing the problem of factuality by modifying the loss functions for training the model. loss truncation @xcite introduces distinguishability by introducing a surrogate loss based on negative entropy where the samples with the highest log loss will be removed. quals @xcite uses a combination of summarization and question-answering methods for producing a factually consistent summary. the model back propagates contrastive loss through both the summaries and qa pairs. constrained abstractive summarization (cas) @xcite introduces constrained decoding where the constraints are either manually or automatically added (key-phrases vocab from the source article). our fact-aware training which preprocesses and filters the training data is inspired by the loss truncation @xcite approach where the loss truncation learns to ignore training data with less distinguishability. article: united nations officials rarely use the words "genocide" and "ethnic cleansing" but they now say potentially both could envelop the world's youngest country. since violence flared in juba in july and spread to the previously peaceful southern equatoria states of south sudan, . . . "this has been unrelenting since july," said nasir abel fernandes, the unhcr's senior emergency coordinator in northern uganda. "the international community has to pay attention, and pressure the south sudanese leaders to stop this, as it's a massacre of civilians from both sides . . . sudan is doing to its own people . . . pegasus : south sudan's civil war could be the world's worst since world war two, the un has warned gold summary: for three years south sudan has tumbled deeper into self-inflicted chaos, and it now finds itself on the brink of something even more terrifying. facttrain (anli): south sudan's civil war is spiralling out of control. facttrain (sc): the un refugee agency (unhcr) has warned that south sudan's civil war could lead to "genocide" and "ethnic cleansing" rerank (factcc): south sudan's civil war has been described as "genocide" and "ethnic cleansing" ## limitations one of the primary limitations of this work is that fact-aware training uses hard thresholds to discard entire training examples, whereas there are abstractive summarization techniques that only modify the hallucinated content in the text while keeping the other portions intact. additionally, although this is a part of our future work, another contribution could have been a novel metric that can ensemble characteristics of all the factuality metrics. finally, in comparison to the baseline summarization models, fact-aware summarization models have larger inference times because of the pre-processing and post-processing required to choose relevant candidates either for training or as output.