Friendly Topic Assistant for Transformer Based Abstractive Summarization (2020)

Abstract: Abstractive document summarization is a comprehensive task including document understanding and summary generation, in which area Transformer-based models have achieved the state-of-the-art performance. Compared with Transformers, topic models are better at learning explicit document semantics, and hence could be integrated into Transformers to further boost their performance. To this end, we rearrange and explore the semantics learned by a topic model, and then propose a topic assistant (TA) including three modules. TA is compatible with various Transformer-based models and user-friendly since i) TA is a plug-and-play model that does not break any structure of the original Transformer network, making users easily fine-tune Transformer+TA based on a well pre-trained model; ii) TA only introduces a small number of extra parameters. Experimental results on three datasets demonstrate that TA is able to improve the performance of several Transformer-based models.

## introduction automatic summarization, requiring both document understanding and text generation, is a comprehensive task in natural language processing (nlp). extractive approaches @xcite @xcite identify and then concatenate the most representative sentences as a summary. by contrast, abstractive summarization @xcite is more challenging, aiming to generate a summary via rephrasing and introducing new concepts/words. our work focuses on abstractive summarization, for which sequence-to-sequence (s2s) models are widely studied. recently, equipped with the attention mechanism @xcite , some transformerbased language models @xcite @xcite @xcite are built with an encoder-decoder structure. these models benefit from pre-training on large-scale corpus, and then are fine-tuned to adapt to the summarization task. as a result, the encoder with bidirectional self-attention (sa) extracts document-token features, the decoder with left-to-right sa generates the summary, and the cross attention (ca) bridges the document and summary tokens. though achieving appealing performances, these transformer-based models are better at exploring the relationships among local tokens than the document global semantics. further, due to the limited position index during pre-training, most transformer-based models have a maximum capacity of input tokens. thus, they often truncate the length of a document to satisfy the length limitation of the encoder, which may lose some important semantics, especially for long documents. global semantics are important to summarization @xcite @xcite , since one need to comprehend the entire content before generate summaries. compared with language models, topic models tell global semantics more explicitly. basically, topic models, such as lda @xcite and pfa @xcite , represent each document as a bagof-word (bow) vector and then factor the count vector as a product of topics and topic proportions, as shown in fig. 1 . topics are global variables, describing the distributions over all tokens in the vocabulary. topic proportions are local (documentspecific) features, describing the weights of corresponding topics in each document. therefore, topic models explore the word co-occurrence patterns, i.e., semantics. however, no transformer-based model considers these explicit semantics. in this paper, we rearrange and further explore the semantics of the topic model and develop a friendly topic assistant (ta) for transformer-based abstractive summarization models. by introducing only a small number of parameters into the finetuning stage, ta is a flexible plug-and-play model, consisting of three modules: to prevent the topic features affected by the summary-token features via attention, we perform masked attention in the decoder. ta does not break any structure of the original transformer network, and hence is able to be jointly learned with a pre-trained model during the fine-tuning stage. besides, sia, tema, and drm are cooperated with some basic transformer modules, such as embedding and multi-head attention. therefore, we can plug an arbitrary combination of these three modules into various transformerbased models. ## background ta is a friendly plug-and-play model that is compatible with many transformer-based summarization models. to illustrate ta without loss of generality, we choose the bertsum @xcite as an example transformer-based model, and pfa @xcite as an example topic model. ## topic assistant for transformer given a corpus, we train a pfa based on documents. then, we use the extracted topics and topic proportions to build three plug-and-play modules to help the transformer fine-tuning, including semanticinformed attention, topic embedding with masked attention, and document-related modulation. ## conclusion and future work in this paper, we explore and rearrange semantics of a topic model and then propose a friendly plug-and-play ta for transformer-based abstractive summarization models. by introducing a small number of parameters, ta is able to further improve the performance of these models, especially under a long-document scenario. in the future, we will study the effectiveness of ta on other nlp tasks, such as the document-level translation, and investigate whether ta is useful for transformer pre-training. (204, 045/11, 332/11, 334) and follow the pre-processing in @xcite . to satisfy the maximum capacity of the encoder in the base model, such as 512 for bertsum, we use truncated document as the encoder input.