A Cascade Approach to Neural Abstractive Summarization with Content Selection and Fusion (2020)

Abstract: We present an empirical study in favor of a cascade architecture to neural text summarization. Summarization practices vary widely but few other than news summarization can provide a sufficient amount of training data enough to meet the requirement of end-to-end neural abstractive systems which perform content selection and surface realization jointly to generate abstracts. Such systems also pose a challenge to summarization evaluation, as they force content selection to be evaluated along with text generation, yet evaluation of the latter remains an unsolved problem. In this paper, we present empirical results showing that the performance of a cascaded pipeline that separately identifies important content pieces and stitches them together into a coherent text is comparable to or outranks that of end-to-end systems, whereas a pipeline architecture allows for flexible content selection. We finally discuss how we can take advantage of a cascaded pipeline in neural text summarization and shed light on important directions for future research.

## introduction there is a variety of successful summarization applications but few can afford to have a large number of annotated examples that are sufficient to meet the requirement of end-to-end neural abstractive summarization. examples range from summarizing radiology reports @xcite to congressional bills @xcite and meeting conversations @xcite @xcite . the lack of annotated resources suggests that end-toend systems may not be a "one-size-fits-all" solution to neural text summarization. there is an increasing need to develop cascaded architectures to allow for customized content selectors to be combined with general-purpose neural text generators to realize the full potential of neural abstractive summarization. we advocate for explicit content selection as it allows for a rigorous evaluation and visualization of intermediate results of such a module, rather than associating it with text generation. existing neural abstractive systems can perform content selection implicitly using end-to-end models @xcite @xcite , or more explicitly, with an external module to select important sentences or words to aid generation @xcite @xcite @xcite @xcite . however, content selection concerns not only the selection of important segments from a document, but also the cohesiveness of selected segments and the amount of text to be selected in order for a neural text generator to produce a summary. in this paper, we aim to investigate the feasibility of a cascade approach to neural text summarization. we explore a constrained summarization task, where an abstract is created one sentence at a time through a cascaded pipeline. our pipeline architecture chooses one or two sentences from the source document, then highlights their summary-worthy segments and uses those as a basis for composing a summary sentence. when a pair of sentences are selected, it is important to ensure that they are fusible-there exists cohesive devices that tie the two sentences together into a coherent text-to avoid generating nonsensical outputs @xcite . highlighting sentence segments allows us to perform fine-grained content selection that guides the neural text generator to stitch selected segments into a coherent sentence. the contributions of this work are summarized as follows. sent pred highlight non-highlight [cls] a duke thursday sentence selection fine-grained content selection highlight student non-highlight ... ... ... ... a duke thursday student + + + + @xmath0... encoder decoder word embedding highlight embedding ## a cascade approach our cascaded summarization approach focuses on shallow abstraction. it makes use of text transformations such as sentence shortening, paraphrasing and fusion @xcite and is in contrast to deep abstraction, where a full semantic analysis of the document is often required. a shallow approach helps produce abstracts that convey important information while, crucially, remaining faithful to the original. in what follows, we describe our approach to select single sentences and sentence pairs from the document, highlight summary-worthy segments and perform summary generation conditioned on highlights. ## conclusion we present a cascade approach to neural abstractive summarization that separates content selection from surface realization. importantly, our approach makes use of text highlights as intermediate representation; they are derived from one or two sentences using a coarse-to-fine content selection strategy, then passed to a neural text generator to compose a summary sentence. a successful cascade approach is expected to accurately select sentences and highlight an appropriate amount of text, both can be customized for domain-specific tasks.