Semantic Extractor-Paraphraser based Abstractive Summarization (2020)

Abstract: The anthology of spoken languages today is inundated with textual information, necessitating the development of automatic summarization models. In this manuscript, we propose an extractor-paraphraser based abstractive summarization system that exploits semantic overlap as opposed to its predecessors that focus more on syntactic information overlap. Our model outperforms the state-of-the-art baselines in terms of ROUGE, METEOR and word mover similarity (WMS), establishing the superiority of the proposed system via extensive ablation experiments. We have also challenged the summarization capabilities of the state of the art Pointer Generator Network (PGN), and through thorough experimentation, shown that PGN is more of a paraphraser, contrary to the prevailing notion of a summarizer; illustrating it’s incapability to accumulate information across multiple sentences.

## introduction over the past few years, the internet has become the most convenient and preferred form of information sharing worldwide. the evolution of technology has made it possible for anyone to convey their knowledge, opinions and ideals to the world, resulting in an increasing surge of information hindering users from accessing desired content. this increasing need to obtain key information makes the task of summarization paramount. text is the most widely adopted form of communication, be it for personal messaging 1 or for broadcasting, owing to its ability to convey almost any concept, its general flexibility to suit everyone's needs, and its less storage requirement (opposed to other modes of communication like audio and video). text summarization is a problem at the very core of natural language processing, and has various applications in the spoken languages, including summarization of conversations, and public speeches. some works have been done in the field of reinforcement learning based text summarization @xcite , the most prominent architecture being extractor-abstractor (ext-abs) model @xcite . inspired from this architecture, in this manuscript, we have proposed an extractor-paraphraser system that uses semantic information overlap as the underlying guidance strategy. the model is further enhanced to surpass its limits using reinforcement learning, for which we have proposed a novel semantic overlap based reward function. word mover similarity (wms) @xcite is utilized to evaluate semantic similarity across generated sentences and the true ground truth summary sentences. we assume that paraphrasing is a relatively simpler task than abstractive summarization, with the underlying intuition that paraphrasing is a subproblem within abstractive summarization. to bolster our hypothesis, experiments are conducted on the extractor-abstractor (ext-abs) model @xcite and the pointer generator network (pgn) @xcite , which is used as the basic abstraction unit in the former architecture. the results are rather staggering and reveal that the pgn model also paraphrases input document sentences, albeit implicitly. the major contributions of the paper are as follows: ## related work automatic text summarization has been extensively researched over more than three decades, and has shown a lot of progress and promise over the course of time. various approaches have been explored to tackle both extractive and abstractive summarization. initial research @xcite focused on extractive summarization due to its easier setup. various techniques ranging from integer linear programming @xcite , graph based approaches @xcite , genetic algorithms @xcite , and neural networks @xcite have been adopted to solve the extractive summarization task. the majority of the research in abstractive summarization revolves around deep learning @xcite @xcite . @xcite proposed a generative adversarial network based model to generate document abstracts. a handful of works however also use ilp @xcite and graph-based @xcite techniques to attempt to solve the problem. a lot of domain specific summarization techniques have also been explored, like radiology findings summarization @xcite , across-time sum-marization @xcite , movie review summarization @xcite , book summarization @xcite , and customer review based opinion summarization @xcite . lately, multi-modal summarization @xcite @xcite has also gained popularity . recently, people have also explored reinforcement learning to tackle the problem of automatic text summarization in both extractive @xcite and abstractive domains @xcite . @xcite have proposed an extractorabstractor architecture, separating the relevant data searching part and the paraphrasing part to individual modules. in this work, we have proposed a system inspired from @xcite , stressing on the significance of semantic information over the traditional syntactic overlap. the literature on text summarization is rich, and has an abundance of survey papers @xcite to get an in depth overview of the domain. ## proposed method problem definition: given the training data {x, y} where @xmath0, d 2 , ..., d n } is the set of input documents and @xmath1, y 2 , ..., y n } is the set of corresponding output summaries, the task of automatic summarization is defined as the problem of discovering a function f : @xmath2, such that f (d i ) = y i ; ∀@xmath3, 2, ..., n }. we have proposed an extractor-paraphraser framework, which is inspired from the extractorabstractor (ext-abs) framework introduced by @xcite . the summarization function f (.) is approximated as the composition, f (d i ) = h(g(d i )), where the functions g(.) and h(.) are modeled as the extractor and the paraphraser components of the model, respectively. ## results results of different baselines and the proposed approach are discussed in this section. ## conclusion in this paper, we propose the extractor-paraphraser model, that comprises of the n-to-one paraphraser and a novel word mover similarity based reward function. we show that the semantic overlap based techniques surpass the strong baselines that rely on syntactic information overlap. we also develop experiments to unveil that the state-of-the-art pointer-generator network (pgn) indeed paraphrases input sentences intrinsically, and is unable to merge two sentences into one agglomerate sentence when explicitly conditioned to do so, changing our perception of sequence-to-sequence abstractive summarization models. we show that the existing works are still nowhere close to mimicking a true human summary, portraying the difficulty of true abstraction over its simplified formulation of extracting and then paraphrasing.