Learning to Fuse Sentences with Transformers for Summarization (2020)

Abstract: The ability to fuse sentences is highly attractive for summarization systems because it is an essential step to produce succinct abstracts. However, to date, summarizers can fail on fusing sentences. They tend to produce few summary sentences by fusion or generate incorrect fusions that lead the summary to fail to retain the original meaning. In this paper, we explore the ability of Transformers to fuse sentences and propose novel algorithms to enhance their ability to perform sentence fusion by leveraging the knowledge of points of correspondence between sentences. Through extensive experiments, we investigate the effects of different design choices on Transformer’s performance. Our findings highlight the importance of modeling points of correspondence between sentences for effective sentence fusion.

## introduction a renewed emphasis must be placed on sentence fusion in the context of neural abstractive summarization. a majority of the systems are trained end-toend @xcite @xcite @xcite , where an abstractive summarizer is rewarded for generating summaries that contain the same words as human abstracts, measured by automatic metrics such as rouge @xcite . a summarizer, however, is not rewarded for correctly fusing sentences. in fact, when examined more closely, only few sentences in system abstracts are generated by fusion @xcite . for instance, 6% of summary sentences generated by pointer-gen @xcite are through fusion, whereas human abstracts contain 32% fusion sentences. moreover, sentences generated by fusion are prone to errors. they can be ungrammatical, nonsensical, or otherwise ill-formed. there is thus an urgent need to develop neural abstractive summarizers to fuse sentences properly. the importance of sentence fusion has long been recognized by the community before the era of neural text summarization. the pioneering work of @xcite introduces an information fusion algorithm that combines similar elements across related text to generate a succinct summary. later work, such as @xcite @xcite @xcite , builds a dependency or word graph by combining syntactic trees of similar sentences, then employs integer linear programming to decode a summary sentence from the graph. most of these studies have assumed a set of similar sentences as input, where fusion is necessary to reduce repetition. nonetheless, humans do not limit themselves to combine similar sentences. in this paper, we pay particular attention to fuse disparate sentences that contain fundamentally different content but remain related to make fusion sensible @xcite . in figure 1 , we provide an example of a sentence fusion instance. we address the challenge of fusing disparate sentences by enhancing the transformer architecture @xcite with points of correspondence between sentences, which are devices that tie two sentences together into a coherent text. the task of sentence fusion involves choosing content from each sentence and weaving the content pieces together into an output sentence that is linguistically plausible and semantically truthful to the original input. it is distinct from @xcite that connect two sentences with discourse markers. our contributions are as follows. ## method a poc is a pair of text chunks that express the same or similar meanings. in fig. 1 , allan donald vs. the 48-year-old former test paceman, south africa bowling coach vs. part of the coaching team are two pocs. the use of alternative expressions for conveying the same meanings is standard practice in writing, as it increases lexical variety and reduces redundancy. however, existing summarizers cannot make effective use of these expressions to establish correspondence between sentences, often leading to ungrammatical and nonsensical outputs. ## conclusion we address the challenge of information fusion in the context of neural abstractive summarization by making crucial use of points of correspondence between sentences. we enrich transformers with poc information and report model performance on a new test bed for information fusion. our findings suggest that modeling points of correspondence is crucial for effective sentence fusion, and sentence fusion remains a challenging direction of research. future work may explore the use of points of correspondence and sentence fusion in the standard setting of document summarization. performing sentence fusion accurately and succinctly is especially important for summarizing long documents and book chapters @xcite . these domains may contain more entities and events to potentially confuse a summarizer, making our method of explicitly marking these entities beneficial.