A New Approach to Overgenerating and Scoring Abstractive Summaries (2021)

Abstract: We propose a new approach to generate multiple variants of the target summary with diverse content and varying lengths, then score and select admissible ones according to users’ needs. Abstractive summarizers trained on single reference summaries may struggle to produce outputs that achieve multiple desirable properties, i.e., capturing the most important information, being faithful to the original, grammatical and fluent. In this paper, we propose a two-staged strategy to generate a diverse set of candidate summaries from the source text in stage one, then score and select admissible ones in stage two. Importantly, our generator gives a precise control over the length of the summary, which is especially well-suited when space is limited. Our selectors are designed to predict the optimal summary length and put special emphasis on faithfulness to the original text. Both stages can be effectively trained, optimized and evaluated. Our experiments on benchmark summarization datasets suggest that this paradigm can achieve state-of-the-art performance.

## introduction the learning objective of a modern abstractive summarizer is to produce system outputs that resemble reference summaries on a word-to-word basis. it does not promote outputs that possess multiple desirable properties, i.e., capturing the most important information, being faithful to the original text, grammatical and fluent, though some of these properties are exhibited by system abstracts as a natural outcome of a learned summarizer @xcite @xcite @xcite @xcite @xcite . without direct optimization of desired properties, system abstracts often change the meaning of the original document or fail to convey the main concepts @xcite . ## a confidence-driven generator we seek to produce a highly diverse set of alternative summaries from any source input, but standard neural language generators with beam search only produce high-likelihood sequences rather than diverse ones @xcite . to address this limitation, we devise a new generator that is capable of producing summaries of varying lengths. a long summary can cover more important information of the source text, whereas a short summary is easy-to-read. moreover, it produces a summary having the exact given length and with a proper endpoint. this is achieved by shifting away from left-to-right generation but building a summary using a confidence-driven approach. our generator is illustrated in figure 1 . to generate a summary of l tokens, we place a number of [mask] tokens following the source text, which serve as "placeholders" for summary tokens. importantly, our generator simultaneously predicts the most probable tokens for all positions, as opposed to predicting only the most probable next token in an autoregressive setting. we obtain the token that has the highest probability across all positions, and use it to replace the [mask] token of that position. next, the model continues to make predictions for all remaining positions, conditioned on the source text and the summary tokens seen thus far of varying positions. let @xmath0. our confidence-driven generation process defines a new order of summary tokens, @xmath1, o @xmath2, according to which p θ (y|x) is factorized into a product of conditional probabilities p θ (y o j |y @xmath3, x) (eq. ( 1 )), where θ are model parameters to be optimized during training. our learning objective is to minimize the negative data log-likelihood (eq. ( 2 )) to predict missing tokens y * o j conditioned on the source text x and the summary tokens seen thus far y @xmath4. our generator is trained with a denoising objective. it consists of a decoder-only architecture with 12 transformer blocks @xcite . given input the bank of japan appealed to financial markets to remain calm friday following the us decision to order daiwa bank ltd. to close its us operations. @xmath5,5 calls 4 for 6 calm. ## the selectors we present two selectors to respectively assess the overall quality of the summary and predict the optimal summary length. our selectors assume the role of a responsible agent that, when provided with a source text and multiple alternative summaries, can effectively recognize the admissible ones. it has the potential to incorporate human-in-the-loop in future to teach the model to select best summaries. ## conclusion we investigate a new approach to neural abstractive summarization that focuses on producing multiple summary hypotheses with varying lengths and levels of details. our selectors are designed to identify summaries that have the optimal length and the best overall quality. the approach obtains state-ofthe-art results on summarization benchmarks and opens up a potential new avenue for customizing summary selectors to suit users' needs. future work includes extending this research to long documents. our confidence-driven generator and the selectors could potentially be extended to operate on spans of text @xcite ) rather than individual tokens, thus allowing for efficient generation of multiple summary hypotheses and identification of admissible summaries and/or summary segments.