Hierarchical Attention Graph for Scientific Document Summarization in Global and Local Level (2024)

Abstract: Scientific document summarization has been a challenging task due to the long structure of the input text. The long input hinders the simultaneous effective modeling of both global high-order relations between sentences and local intra-sentence relations which is the most critical step in extractive summarization. However, existing methods mostly focus on one type of relation, neglecting the simultaneous effective modeling of both relations, which can lead to insufficient learning of semantic representations. In this paper, we propose HAESum, a novel approach utilizing graph neural networks to locally and globally model documents based on their hierarchical discourse structure. First, intra-sentence relations are learned using a local heterogeneous graph. Subsequently, a novel hypergraph self-attention layer is introduced to further enhance the characterization of high-order inter-sentence relations. We validate our approach on two benchmark datasets, and the experimental results demonstrate the effectiveness of HAESum and the importance of considering hierarchical structures in modeling long scientific documents.

## introduction extractive summarization aims to select a set of sentences from the input document that best represents the information of the whole document. with the advancement of pre-trained models and neural networks over the years, researchers have achieved promising results in news summarization @xcite . however, when applying these methods to long scientific documents, they encounter challenges due to the relatively lengthy inputs. the considerable length of the text hinders sequential models from capturing both long-range dependencies across sentences and intra-sentence relations simultaneously @xcite .moreover, the extended context exceeds the input limits of the transformer-based model @xcite due to the quadratic computational complexity of self-attention. recently, the application of large language models (llm) such as chatgpt to text summarization tasks has gained significant interest and attracted widespread attention. a recent study by @xcite evaluated the performance of chat-gpt on extractive summarization and further enhanced its performance through in-context learning and chain-of-thought. another study @xcite conducted experiments on abstractive summarization using various llms on a variety of datasets that included long inputs. while the use of llms in text summarization tasks has demonstrated exciting potential, there are still several limitations that have not been addressed. the most important of these is the phenomenon of lost-in-themiddle @xcite , where llms ignore information in the middle and pay more attention to the context at the beginning and end. this bias raises concerns especially in sum-marization tasks where important text may be scattered throughout the document @xcite . additionally, as the input length increases, even on explicitly long-context models, the model's performance gradually declines @xcite . as a result, researchers have turned to graph neural networks to model long-distance relations. they represent a document as a graph and update node representations in the graph using message passing. these works use different methods to construct a graph from documents, such as using sentence similarity as edge weights to model cross-sentence relations @xcite . another popular approach is to construct a word-document heterogeneous graph @xcite , using words as intermediate connecting sentences. @xcite further added passage nodes to the heterogeneous graph to enhance the semantic information. @xcite proposed a hypergraph transformer to capture high-order cross-sentence relations. despite the impressive success of these approaches, we observe that the current work still lacks a comprehensive consideration on relational modeling. more specifically, two limitations are mentioned: (1) most of the existing approaches focus on modeling intra-sentence relations but often overlook cross-sentence high-order relations. inter-sentence connections may not only be pairwise but could also involve triplets or higher-order relations @xcite . in the hierarchical discourse structure of scientific documents, sentences within the same section often express the same main idea. it is difficult to fully understand the content of a document by merely considering intra-sentence and cross-sentence relations in pairwise. (2) these approaches rely on updating relations at different levels simultaneously but ignore the hierarchical structure of scientific documents. sentences are composed of words and, in turn, contribute to forming sections. by understanding the meaning of individual tokens, we get the meaning of the sentence and thus the content of the section. therefore, bottom-to-top structured modeling is crucial to understand the content of the document. to address the above challenges, we propose haesum (hierarchical attention graph for extractive document summarization), a method that leverages a graph neural network model to fully explore hierarchical structural information in scientific documents. haesum first constructs a local heterogeneous graph of word-sentence and updates sentence representations at the intra-sentence level. the local sentence representations are then fed into a novel hypergraph self-attention layer to further update and learn the cross-sentence sentence representations through a self-attention mechanism that fully captures the relations between nodes and edges. figure 1 is an illustration showing the modeling of local and global context information from a hierarchical point of view, and the resulting representations contain both local and global hierarchical information. we validate haesum with extensive experiments on two benchmark datasets and the experimental results demonstrate the effectiveness of our proposed method. in particular, we highlight our main contributions as follows: ## method given an arbitrary document @xmath0, s 2 , ..., s n } consisting of n sentences, each sentence consists of m words s @xmath1, w i2 , ..., w im }. the goal of extractive summarization is to predict labels y @xmath2, 1} for all sentences, where y @xmath3. the overall structure of haesum is shown in figure 2 . ## conclusion this paper presents haesum for scientific document summarization. haesum employs a graphbased model to comprehensively learn local intrasentence and high-order inter-sentence relations, utilizing the hierarchical discourse structure of scientific documents for modeling. the impressive performance of haesum demonstrates the importance of simultaneously considering multiple perspectives of semantics and hierarchical structural information in modeling scientific documents.