Enriching and Controlling Global Semantics for Text Summarization (2021)

Abstract: Recently, Transformer-based models have been proven effective in the abstractive summarization task by creating fluent and informative summaries. Nevertheless, these models still suffer from the short-range dependency problem, causing them to produce summaries that miss the key points of document. In this paper, we attempt to address this issue by introducing a neural topic model empowered with normalizing flow to capture the global semantics of the document, which are then integrated into the summarization model. In addition, to avoid the overwhelming effect of global semantics on contextualized representation, we introduce a mechanism to control the amount of global semantics supplied to the text generation module. Our method outperforms state-of-the-art summarization models on five common text summarization datasets, namely CNN/DailyMail, XSum, Reddit TIFU, arXiv, and PubMed.

## introduction automatic text summarization corresponds to text understanding and text generation processes. in general, there are two main approaches to perform this task. extractive systems @xcite @xcite highlight salient words or sentences from the source text and form the final summary by concatenating them. on the other hand, abstractive methods @xcite @xcite switch among generating new words, choosing phrases from the source document, and rephrasing them. abstractive summarization, which is the focus of this paper, is usually more advanced and closer to human-like interpretation. recently, abstractive summarization studies @xcite @xcite are dominated by transformer-based document: while richie benaud rose from the suburbs to captain australia, he will be remembered forever for his mastery of commentating. the champion leg spinner turned cricket commentating into an art form, earning him the title of 'the voice of cricket.' his commentary was understated, measured and often extremely funny, and were perfectly timed. scroll down for video. 84-year-old cricket commentator richie benaud has passed away after a battle with skin cancer . his sayings from the hundreds of test and one day cricket matches he commentated on across the world were often what fans remembered from important moments. his signature one liners soon dropped to a simple word. 'marvellous...' will forever be linked to the cricket legend. on commentating, benaud said: 'my mantra is -put your brain into gear and if you can add to what's on the screen then do it, otherwise shut up.' he once described the scene on the field: 'from our broadcast box you can't see any grass at all, it is simply a carpet of humanity.' on captaincy, and he was one of the best test captains australia ever had, benaud was modest: 'the hallmark of a great captain is the ability to win the toss, at the right time.' the former leg spinner turned cricket commentating into an art form, giving him the title 'the voice of cricket'. but he cautioned that description with: 'captaincy is 90 per cent luck and 10 per cent skill. but don't try it without that 10 per cent.' [...] gold summary: cricket commentator richie benaud has passed away after cancer battle . the 84-year-old will be remembered for his mastery of commentating . the former leg spinner earned himself the title of the 'voice of cricket'. his trademark line was 'marvellous'. pegasus: the champion leg spinner turned cricket commentating into an art form, earning him the title of 'the voice of cricket'. his commentary was understated, measured and often extremely funny, and were perfectly timed. our model: 84-year-old cricket commentator richie benaud has passed away after a battle with skin cancer. the champion leg spinner earned the title of 'the voice of cricket'. his commentary was understated, measured and often extremely funny. his trademark word, 'marvellous...' will forever be linked to the cricket legend. to avoid missing key points in summarizing, one solution is to furnish the models with global semantics by using probabilistic topic models such as lda @xcite , poisson factor analysis @xcite , or inner hidden states @xcite . nevertheless, traditional topic models were shown to be inefficient in scalability for large-scale datasets @xcite and have limited capability of describing documents @xcite . to overcome the above problems, we propose a novel method that integrates neural topic model into summarization architecture. specifically, we aim to utilize the posterior distribution learned from the neural topic model as an approximation of global semantics of the document and from that, provide a signal for summarization model to have a better understanding of overall document. however, there is one critical question: how can we match the neural topic model's posterior distribution with the true posterior as it has been proven in improving the performance of variational inference @xcite ? to this end, we propose a method to adapt normalizing flow in the neural topic model to have a better approximation of true distribution and integrate it into the summarization model. integrating flow mechanism to better approximate the true posterior has been proven to improve performance for variational inference @xcite as well as for downstream tasks such as image synthesis @xcite , etc. however, to the best of our knowledge, there is no study to investigate the benefit of flow mechanism for the abstractive summarization task. on the other hand, even though rich global semantics is beneficial, there are recent studies showing that the redundant amount of global semantics may cause harm to hidden representation since it introduces detrimental noise to the model @xcite . therefore, we propose a novel contextualized gating mechanism to control the flow of global semantics and maintain important information of the hidden states in the main summarization model. the contributions of our paper can be summerized as follows: ## methodology the overall architecture of our approach is given in figure 2 . it comprises of a topic-oriented encoder, a topic-oriented decoder, and a flow-based neural topic model. formally, given a document as input, we process it into a sequence of tokens @xmath0, and the bagof-word (bow) representation x bow . x is taken as the input for the text summarization module, while x bow serves as the input for the neural topic model. ## conclusion in this paper, we propose a method to utilize global semantics for text summarization task. in particular, we aim to fit the global semantics to expressively describe the documents. moreover, we find that maintaining the information in the original contextualized representations is also beneficial for the summarization performance. we outperform other state-of-the-art models on five benchmark datasets.