Incorporating Question Answering-Based Signals into Abstractive Summarization via Salient Span Selection (2023)

Abstract: In this work, we propose a method for incorporating question-answering (QA) signals into a summarization model. Our method identifies salient noun phrases (NPs) in the input document by automatically generating wh-questions that are answered by the NPs and automatically determining whether those questions are answered in the gold summaries. This QA-based signal is incorporated into a two-stage summarization model which first marks salient NPs in the input document using a classification model, then conditionally generates a summary. Our experiments demonstrate that the models trained using QA-based supervision generate higher-quality summaries than baseline methods of identifying salient spans on benchmark summarization datasets. Further, we show that the content of the generated summaries can be controlled based on which NPs are marked in the input document. Finally, we propose a method of augmenting the training data so the gold summaries are more consistent with the marked input spans used during training and show how this results in models which learn to better exclude unmarked document content.

## introduction abstractive sequence-to-sequence summarization models have become very effective methods of easily generating summaries of input documents @xcite @xcite . previous work has demonstrated that conditioning the summary generation on salient document sentences results in higher-quality summaries and more controllable summarization models @xcite . salient sentences are typically identified during training by ## question-based salience we begin by describing how qa is used to identify salient spans of text in the input document and discuss the advantages of this approach. input document a british military health care worker in sierra leone has tested positive for ebola, a uk health agency said… an ebola outbreak has devastated parts of west africa, with sierra leone … being the hardest hit… where did a british military health care worker test positive for ebola? sierra leone spokesperson: experts are investigating how the uk military health care worker got ebola. it is being decided if the military worker infected in sierra leone will return to england. there have been some 24,000 reported cases and 10,000 deaths in the latest ebola outbreak. ## a two-stage, span-based model next, we propose a two-stage, span-based model that can incorporate the qa-based salience signals into the learning procedure. the first of the two stages, the span selection component, classifies salient spans within the text. the second stage, the generation component, generates the summary given the document and the salient spans. the details of each component are detailed next. ## improving controllability via data augmentation although there is nothing to directly force the generation model to learn to include content based on the supervision provided by the salient spans, if the supervision is of high enough quality, we expect the model will learn to do so. indeed, we later show in §6.2 that this is true, thus the content of the summary can be controlled by which spans are marked as salient. however, it is also desirable for a controllable summarization model to also not include usain bolt rounded off the world championships sunday by claiming his third gold in moscow as he anchored jamaica to victory in the men's 4x100m relay. the fastest man in the world charged clear of united… usain bolt rounded off the world championships sunday by claiming his third gold in moscow as he anchored jamaica to victory in the men's 4x100m relay. the fastest man in the world charged clear of united … usain bolt wins third gold of world championship. anchors jamaica to 4x100m relay victory. jamaica double up in women's 4x100m relay. usain bolt wins third gold of world championship. anchors jamaica to 4x100m relay victory. jamaica double up in women's 4x100m relay. ## experimental setup datasets our experiments use three popular english single-document summarization datasets: cnn/dailymail @xcite , xsum @xcite , and nytimes @xcite . specific details on the sizes of the datasets can be found in appendix a. baselines & other work we compare the salient spans selected by our qa-based method against three baseline span selection methods. the first marks salient sentences by greedily selecting k sentences that maximize the rouge-2 score calculated against the gold summary, a popular method that is frequently used to train extractive summarization models @xcite as well as other two-step abstractive systems @xcite . the other two mark entities and nps as salient if they appear in the gold summaries as determined by lexical matching. we only mark the first occurrence of the phrases as salient since we found that worked better than marking all occurrences. additionally, we compare our results to bart (the original implementation and our own; lewis et al., 2020) since our models are built on top of it. we also compare to gsum @xcite , which uses salient sentence guidance that is similar to our baseline salient sentence method. gsum encodes the additional guidance signal separately from the input document and uses the document and guidance encodings to generate the summary. ## conclusion in this work, we proposed a method for incorporating qa-based signals into a summarization model by automatically marking document nps as salient based on whether a np's corresponding wh-question is answered correctly in the summary. we showed that incorporating this signal into our two-stage summarization model results in higher quality summaries than baseline methods of identifying salient spans. finally, we demonstrated that our data augmentation algorithm, which attempts to ensure the span supervision is consistent with the gold summaries, improves controllability by eliminating unmarked content from the output summaries.