Salience Allocation as Guidance for Abstractive Summarization (2022)

Abstract: Abstractive summarization models typically learn to capture the salient information from scratch implicitly.Recent literature adds extractive summaries as guidance for abstractive summarization models to provide hints of salient content and achieves better performance.However, extractive summaries as guidance could be over strict, leading to information loss or noisy signals.Furthermore, it cannot easily adapt to documents with various abstractiveness.As the number and allocation of salience content pieces varies, it is hard to find a fixed threshold deciding which content should be included in the guidance.In this paper, we propose a novel summarization approach with a flexible and reliable salience guidance, namely SEASON (SaliencE Allocation as Guidance for Abstractive SummarizatiON).SEASON utilizes the allocation of salience expectation to guide abstractive summarization and adapts well to articles in different abstractiveness.Automatic and human evaluations on two benchmark datasets show that the proposed method is effective and reliable.Empirical results on more than one million news articles demonstrate a natural fifteen-fifty salience split for news article sentences, providing a useful insight for composing news articles.

## introduction abstractive summarization seeks to generate concise descriptions about synoptic information of longer documents @xcite @xcite . tackling this task can provide users with improved dissemination and acquisition of more readable content in long documents. more concretely, it allows for enhanced selection, compression and retrieval of web-scale textual information that benefits other nlp tasks such as machine reading comprehension @xcite , mention linking @xcite , claim verification @xcite , and information extraction @xcite . abstractive summarization models are typically trained end-to-end using large collections of paired corpora of raw documents and human-written summaries to directly perform sequence-to-sequence generation. in terms of deciding what to include in the generated summaries, these models implicitly learn to capture the salient information from scratch. accordingly, recent literature has attempted to add auxiliary extractive salience guidance for abstractive summarization models to give them a higher-level understanding of input documents, among which, extractive summaries appear to provide the most effective guidance @xcite @xcite . methods following this strategy learn to first perform extractive summarization, then perform abstraction on top of the extractive summaries @xcite @xcite . however, incorporating extractive summaries as a form of guidance is evidently imperfect, even though it improves the overall performance of ab-stractive summarization in some cases @xcite : 1) extractive summaries are not reliable guidance. when there are too many summaryworthy sentences in the document, selecting a part of them may prone to information loss. when there are too few or no summary-worthy sentences, using the selected extractive summaries could be noisy and confusing to the model. 2) extractive summaries are not flexible to adapt to different cases. the number and allocation of salience content pieces can vary by documents. rather than extracting a fixed number of sentences, a flexible guidance should select salient content based on document properties. an imperfect selection process may also lead to further model biases, such as positional biases or length biases @xcite . as the summarization process can differ for distinct documents @xcite , a reliable guidance should allow flexible content selection, and be adaptive to documents with different abstractiveness. in this paper, we propose a novel summarization approach with a flexible and reliable salience guidance, namely season (salience allocation as guidance for abstractive summarization). salience is the degree to which a sentence contributes to the central idea of a document, and its allocation means how salience is distributed among all sentences in a document. to estimate the salience allocation, a linear classifier is trained on top of the encoder. this estimation is incorporated into the decoder with salience-aware cross-attention (saca). it provides the flexibility to decide how much signal to accept from the salience guidance to supervise the abstractive summarization. the ground-truth salience label is assigned to each sentence based on its similarity with the ground-truth summary. meanwhile, the number of salience degrees and their cut-off thresholds are decided based on the corpus to balance informativeness and prediction accuracy. to further improve the robustness of the summarization model, we apply label smoothing between adjacent salience degrees during training, and use the expectation of salience as a more robust salience estimation. the technical contributions of this work are three-fold. first, we develop a new method for abstractive summarization on transformer-based encoder-decoder architecture with the allocation of salience expectation as flexible guidance ( §3). our method provides reliable guidance that adapts well to articles in different abstractiveness ( §5.1). second, we show the effectiveness and reliability of our proposed method comparing to the existing methods in both automatic ( §4.2) and human evaluation ( §5.3). third, empirical results on more than one million news articles show a natural fifteen-fifty salience split for news article sentences ( §4.3), providing a useful insight for composing news articles. a series of studies attempt to benefit from the advantages of both paradigms by combining them. @xcite encourage the word-level attention of an abstractive summarization model and the relative sentence-level extraction probability from an extractive summarization model to be consistent. more recent studies show that conducting abstractive summarization with extractive summaries as a part of the input leads to better performance @xcite @xcite . extractive summarization can also work as an effective content selector for abstractive summarization when summarizing long documents @xcite . some studies @xcite @xcite also consider to extract key words or phrases instead of summary worthy sentences as guidance, but their performances are not as good as those using sentences @xcite . our work extends the strict extractive summary guidance to a soft guidance of salience allocation. the proposed guidance is more flexible, reliable and adaptive, leading to better performance. selective attention. selective attention is a psychological concept referring to the differential processing of simultaneous sources of information @xcite . incorporating prior knowledge through selective attention is widely explored in natural language processing, especially in recent nlp models with attention mechanism @xcite al., self attention feed forward add & norm n× input embedding add & norm self attention feed forward add & norm ×n output embedding add & norm cross attention add & norm linear softmax salience probabilities softmax output probabilities linear salience embedding q k ⊕ v 𝝵(x) figure 2: model architecture of season. the proposed modules are highlighted with bold lines. season adds a salience predictor on top of the encoder, maps (the expectation of) salience degrees to corresponding embeddings, and adds these salience embeddings to the key vectors of cross attention. different from prior studies, season maps salience degrees to distinct embeddings and adds them to the encoder outputs as key vector for crossattention. this gives our model the flexibility to decide how much signal to accept from the salience guidance for supervising the abstractive summarization process. this strategy achieves better performance in comparison with previous salienceguided selective attention methods. ## season in this work, we employ a transformer-based encoder-decoder model for abstractive summarization. as shown in fig. 2 , our model season encapsulates salience prediction and text summariza-tion in a single network. we perform multi-task end-to-end training, and inference via one forward pass. during training, the model jointly learns to predict the degree of salience for each sentence and is guided with rouge-based ground-truth salience allocation to generate the abstractive summary. during inference, season predicts the expected salience allocation intermediately with the encoder outputs, and uses this predicted information to guide the decoder to generate the summary. ## experiment in this section, we first describe our experimental setting, including datasets, baselines, evaluation metrics and implementation details ( §4.1). then, we show the model performance on two summarization datasets ( §4.2), and provide an insight on salience threshold selection ( §4.3). ## analysis to gain further insights on the proposed method, we perform additional analyses on cnndm to comprehensively investigate performance by abstractiveness ( §5.1), summary length ( §5.2), human evaluation ( §5.3), and the impact of different model components ( §5.4). a case study is also presented in §5.5. ## conclusion in this paper, we propose season, an abstractive summarization approach guided with salience allocation expectation. in season, the salience guidance is adaptive to documents with different abstractiveness, and the salience-aware crossattention module is flexible to decide how much signal to accept from the salience guidance. automatic and human evaluation further demonstrate the effectiveness and reliability of our proposed method. comparing to the strong baseline model (i.e. bart), our method achieves 2.06/1.41/1.91 rouge-1/2/l performance gain on cnndm, and 0.33/0.32/0.60 performance gain on newsroom. finally, the empirical results on more than one million news articles demonstrate a natural fifteen-fifty salience split for news article sentences providing a useful insight for composing news articles.