HETFORMER : Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization (2021)

Abstract: To capture the semantic graph structure from raw text, most existing summarization approaches are built on GNNs with a pre-trained model. However, these methods suffer from cumbersome procedures and inefficient computations for long-text documents. To mitigate these issues, this paper proposes HetFormer, a Transformer-based pre-trained model with multi-granularity sparse attentions for long-text extractive summarization. Specifically, we model different types of semantic nodes in raw text as a potential heterogeneous graph and directly learn heterogeneous relationships (edges) among nodes by Transformer. Extensive experiments on both single- and multi-document summarization tasks show that HetFormer achieves state-of-the-art performance in Rouge F1 while using less memory and fewer parameters.

## introduction recent years have seen a resounding success in the use of graph neural networks (gnns) on document summarization tasks @xcite , due to their ability to capture inter-sentence relationships in complex document. since gnn requires node features and graph structure as input, various methods, including extraction and abstraction @xcite @xcite , have been proposed for learning desirable node representations from raw text. particularly, they have shown that transformer-based pre-trained models such as bert @xcite and roberta @xcite offer an effective way to initialize and fine tune the node representations as the input of gnn. despite great success in combining transformerbased pre-trained models with gnns, all existing approaches have their limitations. the first limitation lies in the adaptation capability to long-text input. most pre-trained methods truncate longer documents into a small fixed-length sequence (e.g., @xmath0, as its attention mechanism requires a quadratic cost w.r.t. sequence length. this would lead to serious information loss @xcite . the second limitation is that they use pre-trained models as a multilayer feature extractor to learn better node features and build multi-layer gnns on top of extracted features, which have cumbersome networks and tremendous parameters @xcite . recently there have been several works focusing on reducing the computational overhead of fullyconnected attention in transformers. especially, etc @xcite and longformer @xcite proposed to use local-global sparse attention in pre-trained models to limit each token to attend to a subset of the other tokens @xcite , which achieves a linear computational cost of the sequence length. although these methods have considered using local and global attentions to preserve hierarchical structure information contained in raw text data, their abilities are still not enough to capture multi-level granularities of semantics in complex text summarization scenarios. in this work, we propose hetformer, a heterogeneous transformer-based pre-trained model for long-text extractive summarization using multi-granularity sparse attentions. specifically, we treat tokens, entities, sentences as different types of nodes and the multiple sparse masks as different types of edges to represent the relations (e.g., token-to-token, token-to-sentence), which can preserve the graph structure of the document even with the raw textual input. moreover, our approach will eschew gnn and instead rely entirely on a sparse attention mechanism to draw heterogeneous graph structural dependencies between input tokens. the main contributions of the paper are summarized as follows: 1) we propose a new structured pre-trained method to capture the heterogeneous structure of documents using sparse attention; 2) we extend the pre-trained method to longer text cls cls cls cls sen1 w1 w2 wn w3 w1 w2 w3 w4 wm sen2 w1 w2 w4 w3 w1 w2 w3 w4 wm w4 wn (a) token-to-token attention wm (b) sen-to-sen & sen↔token attentions cls cls w1 w2 w4 w3 w1 w2 w3 w4 wm wn cls cls sen1 w1 w2 wn w3 w1 w2 w3 w4 sen2 w4 ent ent ent ent cls cls cls cls sen1 ent w2 w4 ent ent w2 w3 ent wm sen2 w2 wn w2 w3 wm wn w4 (d) integrated attention ent ent ent cls cls cls cls sen1 ent w2 w4 ent ent w2 w3 ent wm sen2 w2 wn w2 w3 wm wn w4 ent (c) entity-to-entity attention extractive summarization instead of truncating the document to small inputs; 3) we empirically demonstrate that our approach achieves state-of-the-art performance on both single-and multi-document extractive summarization tasks. ## hetformer on summarization hetformer aims to learn a heterogeneous transformer in pre-trained model for text summarization. to be specific, we model different types of semantic nodes in raw text as a potential heterogeneous graph, and explore multi-granularity sparse attention patterns in transformer to directly capture heterogeneous relationships among nodes. the node representations will be interactively updated in a fine-tuned manner, and finally, the sentence node representations are used to predict the labels for extractive text summarization.