H eter G raph L ong S um: Heterogeneous Graph Neural Network with Passage Aggregation for Extractive Long Document Summarization (2022)

Abstract: Graph Neural Network (GNN)-based models have proven effective in various Natural Language Processing (NLP) tasks in recent years. Specifically, in the case of the Extractive Document Summarization (EDS) task, modeling documents under graph structure is able to analyze the complex relations between semantic units (e.g., word-to-word, word-to-sentence, sentence-to-sentence) and enrich sentence representations via valuable information from their neighbors. However, long-form document summarization using graph-based methods is still an open research issue. The main challenge is to represent long documents in a graph structure in an effective way. In this regard, this paper proposes a new heterogeneous graph neural network (HeterGNN) model to improve the performance of long document summarization (HeterGraphLongSum). Specifically, the main idea is to add the passage nodes into the heterogeneous graph structure of word and sentence nodes for enriching the final representation of sentences. In this regard, HeterGraphLongSum is designed with three types of semantic units such as word, sentence, and passage. Experiments on two benchmark datasets for long documents such as Pubmed and Arxiv indicate promising results of the proposed model for the extractive long document summarization problem. Especially, HeterGraphLongSum is able to achieve state-of-the-art performance without relying on any pre-trained language models (e.g., BERT). The source code is available for further exploitation on the Github.

## introduction document summarization is one of the central problems in nlp, which aims to rewrite a single document or multi documents under a shorter version with preserving the main information. there are two major approaches such as extractive and abstractive summarization. abstractive models are more sophisticated abilities that require wellcomprehensive reading text and generating highquality text. specifically, most of the existing architectures have been built based on sequence-tosequence (seq2seq) techniques in different ways such as recurrent neural network (rnn) @xcite , pointer-generator-network @xcite , or transformer-based models @xcite . furthermore, the external information, for instance, pretrained model bertsum @xcite and topic modeling @xcite ) are incorporated to improve performances. nevertheless, this approach requires a complicated neural network that consists of millions of learnable parameters, which is the cause of raising significant costs in both terms of computation time perplexity and resources. therefore, extractive models still gain much attention. particularly, extractive document summarization (eds) takes a document in the form list of sentences and chooses several best candidates from the original document, then combine them to create the summarization. recent models trend to turn eds into the sequential binary-labeling task @xcite @xcite . graph neural network (gnn) has recently been exploited as an emerging line of deep learning architectures, which has powered various domains, including nlp tasks @xcite . specifically, gnn models are able to model complex structural data containing semantic units (node) with relationships (edge) between them @xcite . for the eds task, each document is represented as a graph structure in which the nodes are the semantic units of the document such as words and sentences. sequentially, developing edges among sentence nodes are capable to model the cross-sentence relations, which is able to handle the limitation of traditional seq2seq-based methods in terms of long-distance dependency among sentences @xcite . in particular, crosssentence edges can be constructed explicitly between sentences (sentence-to-sentence) @xcite @xcite or through intermediate bridge via common words (sentence-word-sentence) @xcite or latent topics (sentence-topic-sentence) @xcite . although the aforementioned approaches have achieved remarkable results in the eds problem, most of the architectures are proposed for short documents (i.e., new articles). long-form document is still a remaining challenge in this research field due to two main reasons: i) most traditional seq2seq methods truncate longer documents into small fixed-length sequences (i.e., passages) @xcite , which leads to information loss problem, especially for the extractive summarization; ii) using gnn-based methods is able to mitigate the information loss by enabling cross-sentence relations, however, representing an effective way for long-text documents into graph structure is still an open research issue. specifically, since the vocabulary size is limited, when the length of the document is increased, more sentences become neighbors with each other (via common words) which is the cause of the similar embedding between sentences. therefore, a graph structure, which includes only word nodes and sentence nodes, might not be an effective way to represent the long documents for the eds problem. in order to alleviate the aforementioned challenges, this paper presents a new graph-based architecture, which contains three semantic units such as word, sentence, and passage. in particular, the passage nodes are adopted for learning the crossrelations between sentences in different passages. furthermore, the passage node can be regarded as the local structure of a group of sentence nodes in which the edges between passages and sentences have the possibility to reduce the harm of similar representations of sentences when expanding graph structure with long documents. figure 1 illustrates the model architecture of hetergraph-longsum. specifically, the main contributions of this paper are threefold as follows: ## hetergraphlongsum model hetergraphlongsum aims to learn a heterogeneous graph structure for long text summarization. specifically, we model an input document with three types of nodes such as word, sentence, and passage nodes, as a heterogeneous graph, and using graph attention network (gat) @xcite for capturing information relations among nodes. ## conclusion this paper presents a new gnn-based model for extractive long document summarization. specifically, gnn has been introduced as a promising approach for exploiting the complex relation of elements (e.g., word and sentence) from an input document. however, representing long documents as graph structure is still a remaining challenge. specifically, lacking cross-relation information between sentences (e.g., long-distance of position in the document) and the increment of nodes might influence the performance. in this regard, this paper proposes a heterogeneous graph including three types of nodes such as word, sentence, and passage, which are simultaneously learned for enabling the cross-relation between sentences. the evaluation of two standard long documents datasets such as arxiv and pubmed shows that the proposed model outperforms state-of-the-art models in this research field without relying on pre-trained language models (e.g., bert).