Do We Really Need That Many Parameters In Transformer For Extractive Summarization? Discourse Can Help ! (2020)

Abstract: The multi-head self-attention of popular transformer models is widely used within Natural Language Processing (NLP), including for the task of extractive summarization. With the goal of analyzing and pruning the parameter-heavy self-attention mechanism, there are multiple approaches proposing more parameter-light self-attention alternatives. In this paper, we present a novel parameter-lean self-attention mechanism using discourse priors. Our new tree self-attention is based on document-level discourse information, extending the recently proposed “Synthesizer” framework with another lightweight alternative. We show empirical results that our tree self-attention approach achieves competitive ROUGE-scores on the task of extractive summarization. When compared to the original single-head transformer model, the tree attention approach reaches similar performance on both, EDU and sentence level, despite the significant reduction of parameters in the attention component. We further significantly outperform the 8-head transformer model on sentence level when applying a more balanced hyper-parameter setting, requiring an order of magnitude less parameters.

## introduction the task of extractive summarization aims to generate summaries for multi-sentential documents by selecting a subset of text units in the source document that most accurately cover the authors communicative goal (as shown in red in figure 1 ). as such, extractive summarization has been a long standing research question with direct practical implications. the main objective for the task is to determine whether a given text unit in the document is important, generally implied by multiple factors, such as position, stance, semantic meaning and discourse. @xcite already showed early on that discourse information, as defined in the rhetorical structure theory (rst) @xcite , are a good indicator of the importance of a text unit in the given context. the rst framework, one of the most elaborate and widely used theories of discourse, represents a coherent document (a discourse) as a constituency tree. the leaves are thereby called elementary discourse units (edus), clause-like sentence fragments corresponding to minimal units of content (i.e. propositions). internal tree nodes, comprising document sub-trees, represent hierarchically compound text spans (or constituents). an additional nuclearity attribute is assigned to each child, representing the importance of the subtree in the local constituent, i.e. the 'nucleus' child plays a more important role than the 'satellite' child in the parent's relation. alternatively, if both children are equally important, both are represented as nuclei. while other popular theories of discourse exist (most notably pdtb @xcite ), rst along with its human-annotated rst-dt treebank @xcite have been leveraged in the past to improve extractive summarizations, with either unsupervised @xcite , or supervised @xcite methods. in this paper, we explore a novel, equally important application for discourse information in extractive summarization, namely to reduce the number of parameters. instead of exploiting discourse trees as an additional source of information on top of neural models, we use the information as a prior to reduce the number of parameters of existing neural models. this is critical not only to reduce the risk of over-fitting but also to create smaller models that are easier to interpret and deploy. not surprisingly, reducing the number of parameters has become increasingly important in the last years, due to the deep-learning revolution. generally speaking, the objective of reducing neural network parameters involves addressing two central questions: (1) what do these models really learn? such that better priors can be provided and less parameters are required and (2) are all the model parameters necessary? to identify which parameters can be safely removed. recently, researchers have explored these questions especially in the context of transformer models. with respect to what is learned in such models, several experiments reveal that the information captured by the multi-head self-attention in the popular bert model (i.e., the learned attention weights) generally align well with syntactic and semantic relations within sentences @xcite . regarding the second question, building on previous work exploring how to prune large neural models while keeping the performance comparable to the original model @xcite , very recently @xcite has proposed the "synthesizer" framework, comparing the performance when replacing the dot-product selfattention in the original transformer model with other, less parameterized, attention types. inspired by these two lines of research on transformer-based models, namely the identification of a close connection between learned attention weights and linguistic structures, and the potential for safely reducing attention parameters, we propose a document-level discourse-based attention method for extractive summarization. with this new, discourse-inspired approach, we reduce the size of the attention module, the core component of the transformer model, while keeping the modelperformance competitive to comparable, fully parameterized models on both edu and sentence level. ## synthesizer-based self-attention evaluation framework aiming to answer the two guiding questions stated in section 1, @xcite propose a suite of alternative self-attention approaches besides the standard dot-product self-attention, as used in the original transformer model. in their "synthesizer" framework, they show that parameter-reduced selfattention mechanisms can achieve competitive performance across multiple tasks, including abstractive summarization. while the experiments in the original "synthesizer" framework are on token level, employing an sequence-to-sequence architecture, we adapt the framework to explore different attention mechanisms on edu-/sentence-level for the extractive summarization task. to evaluate the effect of different attention types in our scenario, we apply the general system shown in figure 2 , using the pretrained bert model as our unit encoder. each unit is thereby represented as the hidden state of the first token in the last bert layer. subsequently, we feed the bert representations into the "synthesizer" documentencoder @xcite with different attention types and employ a multi-layer perceptron (mlp) with sigmoid activation to retrieve a confidence score for each unit, indicating its predicted likelihood to be part of the extractive summary. the "synthesizer" document encoder is essentially a transformer encoder with alternative attention modules, other than the dot-product selfattention. as commonly done, we employ multiple self-attention heads, previously shown to improve the performance of similar models @xcite . for each attention head, the input is defined as @xmath0.e. the number of units), and d represents the hidden dimension of the model. the self-attention matrix is accordingly defined as @xmath1, where a ij is the attention-value that unit i pays to unit j. we further force the sum of the incoming attentions to each unit (as commonly done) to add up to 1, i.e. j a @xmath2. the pafigure 3: comparison of attention methods. (a),(b) and (c) taken from (tay et al., 2020), (d) proposed in this paper. here, we evaluate the three self-attention methodologies proposed by @xcite as our baselines: dot product: as used in the original transformer model, this self-attention calculates a key, a value and a query representation for each textual unit. the attention value is learned as the relationship between the key-and the query-vector defined as dense: instead of using the relationship between units, encoded as keys and values, the dense self-attention @xmath3, where dense(•) is a two-layer fully connected layer mapping from r l×d to r l×l , which can be represented as dense random: a random attention matrix is generated for each attention-head, shared across all data points, i.e. @xmath4. r can thereby be either updated (referred as learned random in sec. 5) or fixed (fixed random) during training. ## conclusion and future work we extend and adapt the "synthesizer" framework for extractive summarization by proposing a new tree self-attention method, based on rst-style consitituency and dependency trees. in our experiments, we show that the performance of the tree self-attention is significantly better than other fixed attention models, while being competitive to the single-head standard dot product self-attention in the transformer model on both, the edu-level and sentence-level extractive summarization task. furthermore, our tree attention is better than the 8-head dot product in the balanced setting. besides these general results, we further investigate low-resource scenarios, where our parameter-light approaches are assumed to be especially useful. however, contrary to this expectation, they do not seem to be more stable and robust than other solutions. in addition, we also find that the multi-head dot product model is not always significantly better than the single-head approach. this, combined with the previous finding, suggest that more research is needed on the balance between the number of parameter and the performance of the summarization model. in the future, we plan to explore ways to also incorporate rhetorical relations into self-attention, in addition to discourse structure and nuclearity. further, we want to replace the hard-coded weight trade-off between nucleus and satellite in the c-tree w/nuc approach, using instead the confidence score from the discourse parser as the weight. finally, since the current two-level encoder performs generally worse than a single token-based encoder (e.g. bertsum @xcite )), we intend to explore tree self-attention in combination with the bertsum model.