Using Question Answering Rewards to Improve Abstractive Summarization (2021)

Abstract: Neural abstractive summarization models have drastically improved in the recent years. However, the summaries generated by these models generally suffer from issues such as: not capturing the critical facts in source documents, and containing facts that are inconsistent with the source documents. In this work, we present a general framework to train abstractive summarization models to alleviate such issues. We first train a sequence-to-sequence model to summarize documents, and then further train this model in a Reinforcement Learning setting with question-answering based rewards. We evaluate the summaries generated by the this framework using multiple automatic measures and human judgements. The experimental results show that the question-answering rewards can be used as a general framework to improve neural abstractive summarization. Particularly, the results from human evaluations show that the summaries generated by our approach is preferred over 30% of the time over the summaries generated by general abstractive summarization models.

## introduction although neural abstractive summarization has seen drastic improvements over the recent years @xcite @xcite , these systems still have multiple drawbacks. one such common drawback is that the generated summaries frequently fail to capture critical facts in source documents (low recall) @xcite . on the other hand, neural abstractive summarization models are known to generate content which are inconsistent with the source document @xcite . this is commonly known as hallucination @xcite . some studies @xcite claim that nearly 30% of the outputs of common abstractive summarization models suffer from this problem. figure 1 shows a source document, the ground truth summary and few summaries generated by neural models. in the generated summary 1, the model fails to capture some of the crucial facts in the original dialog, such as the play is translated. in generated summary 2, although the model successfully identifies the fact that the play is a translated, it incorrectly mentions that both charlie and curtis are performing. due to such common factuality related issues, neural abstractive summarization models are hardly usable in real-world applications @xcite . in this work, we propose a general framework to alleviate factuality related issues and improve the quality of the abstractive summarization by using question-answering(qa) based rewards. first, we train a sequence-to-sequence(seq2seq) summary generation model to take a document as the input and generate a summary as the output. next, we improve the precision and recall of the summary generation model using a qa framework as follows. to improve the precision of the model, we first generate questions and corresponding answers for each generated summary. next, we evaluate the answers that we get for the same questions from the ground truth summaries. if a generated summary contains factually incorrect information, this would lead to having different answers from the ground truth summary for some of the generated questions. we use the similarity of answers to calculate a reward to improve the precision. similarly, to improve the recall of the summarization model, we generate questions and corresponding answers from the ground truth summaries and evaluate the answers we obtain for the same questions from the generated summaries. if the generated summary does not contain some key information as captured in the ground truth summary, then this would lead to obtaining different answers from the ground truth summary for some of the generated questions. we use the similarity of answers to calculate a reward to improve the recall. the calculated rewards were used in a reinforcement learning (rl) based framework to improve the summary generation model. in figure 1 we show an example output from our approach, which does not contain the factuality related issues shown above. we evaluate the summaries generated by our approach using multiple automatic measures and human judgements, and show that the qa can be used as a general framework to improve abstractive summarization. in summary, our key contributions are: (1) we introduce a reinforcement learning framework, which uses qa rewards to improve the recall and precision of abstractive summarization. (2) the framework is evaluated on three commonly used transformer based summarization models on two public datasets. (3) the evaluation of generated summaries on several automatic measures and human judgements show the effectiveness of our method. in particular, the human judges prefer summaries generated by our approach more than 30% of the time, over the summaries generated by general abstractive summarization models. ## related work there have been previous work on improving the factual consistency of abstractive summarization models. @xcite used an approach with two encoders, one to encode the source document, and another to encode the facts, and a decoder to attend to the outputs of the two encoders when generating the summary. @xcite used openie to extract facts and used them in the form of knowledge graphs to improve abstractive summarization. @xcite used facts obtained from question-answering rewards to improve extractive summarization. @xcite used multichoice cloze rewards, in addition to the knowledge graphs to improve the factual consistency. @xcite incorporated entailment knowledge into abstractive summarization to improve factual correctness. there have been several work proposed to evaluate the factuality of summarization algorithms, as more common n-gram based metrics, such as rouge @xcite , are known to perform poorly for this purpose. most recent approaches proposed for evaluating the factuality are based on qa frameworks @xcite @xcite @xcite . the evaluation metrics proposed by the the above studies measure to which extent a generated summary provides sufficient information to answer questions posed on its ground truth summary and whether the questions generated on the generated summary can be answered by the ground truth summary. ## improving summarization with qa rewards in general, abstractive summarization models are trained to minimize the cross entropy loss of the reference summary at the word-level, which does not necessarily reward models for being factually accurate with high precision and recall @xcite . hence, to improve the factual accuracy of abstractive summarization, we propose a general framework which uses qa based rewards and rl based training. our proposed framework is illustrated in figure 2 , and below we describe the critical components of the framework. ## evaluation and results we evaluate our qa based summarization framework on three common neural abstractive summarization models: gpt-2 @xcite , bart @xcite and pegasus @xcite . the experiments are performed on two different abstractive summarization datasets: the documents in the xsum data are fed to the models unaltered. for samsum data, we first preprocess the conversations by replacing the personal names (ex: john) with unique tags (ex:<@xmath0, and then accumulate the utterances in each conversation as follows before feeding them to the models: <@xmath1.... in this implementation, we generate one qa pair per sentence in a summary. in addition to that, we also filter out answers that are long (over 5 words), as we believe such long answers do not correspond to the factuality, which is the focus of this study. the average number of qa pairs per summary are 2.5 and 1.4 for sam-sum and xsum datasets respectively. the qa based reward process is less expensive in this study, since the number of qa pairs generated are low compared to the studies that generate qa pairs on source documents (not on summaries). we evaluate each model, first, with general method of training: generate the summary given the document, then, with further rl based training with qa rewards that we propose. the hyper-parameters used in training are available in the section a of the appendix. evaluation with rouge scores: we first evaluate the models using the rouge scores. the obtained results are reported in tables 1 and 2 . each table contains two sections, where the first section shows the accuracy before training with qa based rewards, and the second section shows the results after rl based training with qa rewards. the results suggest that for both datasets, each model significantly improves (@xmath2.05) its summarization accuracy using our qa framework. factuality based evaluation: we evaluate the results obtained from our models using the factuality based evaluation framework proposed by @xcite . this measure provides better correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance) @xcite , and provides precision, recall and f1 for a generated summary given a reference. the results obtained on the two datasets are shown in ## conclusion we investigated the problem of low recall and precision of factuality in neural abstractive summarization models, and proposed a framework to alleviate this issue which uses qa based rewards. the proposed framework is evaluated on three commonly used transformer based summarization models and on two publicly available datasets. the automatic evaluations were performed using rouge scores, as well as question answering based evaluation framework and the results suggest that the our method improves the summarization accuracy and factuality. the human evaluation on the generated summaries also suggest that our approach produces summaries with significantly high factual consistency and quality. original document/dialog person_0: hey babe, what do you want for dinner tonight? person_1: gah, don't even worry about it tonight person_0: what do you mean? everything ok? person_1: not really, but it's ok, don't worry about cooking though, i'm not hungry person_0: well what time will you be home? person_1: soon, hopefully person_0: you sure? maybe you want me to pick you up? person_1: no no it's alright. i'll be home soon, i'll tell you when i get home. person_0: alright, love you. person_1: love you too.