HEGEL : Hypergraph Transformer for Long Document Summarization (2022)

Abstract: Extractive summarization for long documents is challenging due to the extended structured input context. The long-distance sentence dependency hinders cross-sentence relations modeling, the critical step of extractive summarization. This paper proposes HEGEL, a hypergraph neural network for long document summarization by capturing high-order cross-sentence relations. HEGEL updates and learns effective sentence representations with hypergraph transformer layers and fuses different types of sentence dependencies, including latent topics, keywords coreference, and section structure. We validate HEGEL by conducting extensive experiments on two benchmark datasets, and experimental results demonstrate the effectiveness and efficiency of HEGEL.

## introduction extractive summarization aims to generate a shorter version of a document while preserving the most salient information by directly extracting relevant sentences from the original document. with recent advances in neural networks and large pretrained language models @xcite , researchers have achieved promising results in news summarization (around 650 words/document) @xcite @xcite @xcite . however, these models struggle when applied to long documents like scientific papers. the input length of a scientific paper can range from 2000 to 7, 000 words, and the expected summary (abstract) is more than 200 words compared to 40 words in news headlines. scientific paper extractive summarization is highly challenging due to the long structured input. the extended context hinders sequential models like rnn from capturing sentence-level longdistance dependency and cross-sentence relations, which are essential for extractive summarization. in addition, the quadratic computation complexity of attention with respect to the input tokens length makes transformer @xcite based models not applicable. moreover, long documents typically cover diverse topics and have richer structural information than short news, which is difficult for sequential models to capture. as a result, researchers have turned to graph neural network (gnn) approaches to model crosssentence relations. they generally represent a document with a sentence-level graph and turn extractive summarization into a node classification problem. these work construct graph from document in different manners, such as inter-sentence cosine similarity graph in @xcite , rhetorical structure theory (rst) tree relation graph in @xcite , approximate discourse graph in @xcite , topicsentence graph in @xcite and worddocument heterogeneous graph in @xcite . however, the usability of these approaches is limited by the following two aspects: (1) these methods only model the pairwise interaction between sentences, while sentence interactions could be triadic, tetradic, or of a higher-order in natural language @xcite . how to capture high-order cross-sentence relations for extractive summarization is still an open question. (2) these graph-based approaches rely on either semantic or discourses structure cross-sentence relation but are incapable of fusing sentence interactions from different perspectives. sentences within a document could have various types of interactions, such as embedding similarity, keywords coreference, topical modeling from the semantic perspective, and section or rhetorical structure from the discourse perspective. capturing multi-type crosssentence relations could benefit sentence representation learning and sentence salience modeling. figure 1 is an illustration showing different types of sentence interactions provide different connectivity for document graph construction, which covers both local and global context information. to address the above issues, we propose hegel (hypergraph transformer for extractive long document summarization), a graph-based model designed for summarizing long documents with rich discourse information. to better model high-order cross-sentence relations, we represent a document as a hypergraph, a generalization of graph structure, in which an edge can join any number of vertices. we then introduce three types of hyperedges that model sentence relations from different perspectives, including section structure, latent topic, and keywords coreference, respectively. we also propose hypergraph transformer layers to update and learn effective sentence embeddings on hypergraphs. we validate hegel by conducting extensive experiments and analyses on two benchmark datasets, and experimental results demonstrate the effectiveness and efficiency of hegel. we highlight our contributions as follows: ## method in this section, we introduce hegel in great detail. we first present how to construct a hypergraph for a given long document. after encoding sentences into contextualized representations, we extract their section, latent topic, and keyword coreference relations and fuse them into a hypergraph. then, our hypergraph transformer layer will update and learn sentence representations according to the hypergraph. finally, hegel will score the salience of sentences based on the updated sentence representations to determine if the sentence should be included in the summary. the overall architecture of our model is shown in figure 2(a) . ## experiment this section presents experimental details on two benchmarked academic paper summarization datasets. we compare our proposed model with state-of-the-art baselines and conduct detailed analysis to validate the effectiveness of hegel. ## conclusion this paper presents hegel for long document summarization. hegel represents a document as a hypergraph to address the long dependency issue and captures higher-order cross-sentence relations through multi-type hyperedges. the strong performance of hegel demonstrates the importance of modeling high-order sentence interactions and fusing semantic and structural information for future research in long document extractive summarization.