Improving Factual Consistency of Abstractive Summarization via Question Answering (2021)

Abstract: A commonly observed problem with the state-of-the art abstractive summarization models is that the generated summaries can be factually inconsistent with the input documents. The fact that automatic summarization may produce plausible-sounding yet inaccurate summaries is a major concern that limits its wide application. In this paper we present an approach to address factual consistency in summarization. We first propose an efficient automatic evaluation metric to measure factual consistency; next, we propose a novel learning algorithm that maximizes the proposed metric during model training. Through extensive experiments, we confirm that our method is effective in improving factual consistency and even overall quality of the summaries, as judged by both automatic metrics and human evaluation.

## introduction recent advances in neural text generation have led to significant improvement in the quality of abstractive summarization @xcite @xcite . despite this progress, there are still many limitations facing neural text summarization @xcite , the most serious of which is the tendency to generate summaries that are not factually consistent with the input document; a factually consistent summary only contains statements that can be inferred from the source document. recent studies show that about 30% of the summaries generated by neural network sequence-to-sequence (seq2seq) models suffer from fact fabrication @xcite . the standard training approach for seq2seq learning has been maximizing the log likelihood of the target given the input sequences (mle). it has empirically performed well as a surrogate loss input: ## an efficient metric for factual consistency in order to improve factual consistency of summarization models, we must have a metric to quantify it. in addition, the metric needs to be computationally efficient so that we can incorporate it as part of the model training process. we first describe the qags protocol and then present our quals protocol. ## improving factual consistency through contrastive learning although quals can be computed more efficiently, using it in the training process is not straightforward because one would need to backpropagate through generated sumaries and qa pairs. we present our conseq (contrastive seq2seq learning) algorithm that can effectively maximize such metrics in training. to fix notation, @xmath0, . . . , x m denotes a sequence of input tokens; @xmath1, . . . , y n denotes a sequence of target output tokens; ŷ = ŷ1 , . . . , ŷn denotes a sequence of generated tokens from a seq2seq model via sampling, i.e. ŷ ∼ p θ (•|x), where θ is the parameter of the model. let r(ŷ, x) be the evaluation (in our case the quals) metric that we aim to maximize. ## related work several authors have pointed out the problem of factual inconsistency in abstractive summarization models @xcite @xcite . besides qags @xcite and factcc @xcite , another possible approach to quantify factual consistency is to rely on open information extraction (openie) and dependency parsing tools to identify and match the relations in an input document and its summary @xcite . however, the underlying openie tools are often not accurate enough to be used for this purpose. our proposed conseq algorithm is related to the unlikelihood training @xcite as both have positive and negative loss terms. the key difference is that in unlikelihood training, the negative loss serves as a regularization term, weighted by a hyperparameter α, in addition to the regular mle training. in contrast, our conseq is motivated from the reinforce algorithm and treats the positive and negative terms equally. furthermore, while the unlikelihood training uses all the ground truth sequences equally in the regular mle (positive) loss term, we construct the positive and negative sets by incorporating the reward function (e.g. quals) as discussed in sec. 3. in another related work, factual consistency metrics at the entity level have been proposed @xcite . the authors also investigated several techniques such as data cleaning, multitask learning and entity-augmented decoding to improve entity level factual consistency scores of abstractive summarization models. in contrast, the quals metric that we propose is more general, not limited to entities. another recent work tackles the hallucination problem in abstractive text summarization via post processing on the generated summary @xcite . specifically, entities of the generated summaries are swapped with other named entities of the same type found in the original document to form a set of candidate summaries. the final summary is determined by a ranking model trained to prefer the factually consistent summaries. ## conclusion in this paper we proposed to improve the factual consistency of abstractive summarization models. we first proposed an efficient evaluation protocol called quals to measure factual consistency. we then proposed a contrastive learning algorithm for seq2seq models called conseq to maximize quals during training. we demonstrated that our proposed method significantly improves the factual consistency of the current state-of-theart summarization model measured by automatic metrics as well as side-by-side human evaluation. in addition to improving factual consistency of summarization models, we believe that the conseq algorithm can have a wider impact on training seq2seq models in general to incorporate non-differentiable evaluation metrics into model training. experiments that the best results are obtained with only one outer iteration of algorithm 1.