Multi Graph Neural Network for Extractive Long Document Summarization (2022)

Abstract: Heterogeneous Graph Neural Networks (HeterGNN) have been recently introduced as an emergent approach for extracting document summarization (EDS) by exploiting the cross-relations between words and sentences. However, applying HeterGNN for long documents is still an open research issue. One of the main majors is the lacking of inter-sentence connections. In this regard, this paper exploits how to apply HeterGNN for long documents by building a graph on sentence-level nodes (homogeneous graph) and combine with HeterGNN for capturing the semantic information in terms of both inter and intra-sentence connections. Experiments on two benchmark datasets of long documents such as PubMed and ArXiv show that our method is able to achieve state-of-the-art results in this research field.

## introduction extractive document summarization aims to automatically extract a set of sentences, which represents information for the whole document, by ranking the importance of sentence features. recent works focus on gnn, a deep learning-based approach that operates on graph domain @xcite , to achieve remarkable results in this research field. specifically, gnn-based models are able to encode the complicated pairwise relationships between entity tokens for better informative representations @xcite . @xcite uses information of topic-aware to change the representation of words to a new representation. then, a gnn-based model is presented for capturing relationships efficiently via graph-structured document representation between sentences. sequentially, hetergnn, a special kind of gnn @xcite , has been proposed as a promising approach to enrich the relationships between words and sentences. @xcite introduced hsg, * corresponding author a heterogeneous graph-based neural network for extractive summarization by using more fine-grained semantic units in the summarization graph to extract the complex relationships between words and sentences. accordingly, the model has achieved the top performance in cnn/dailymail and nyt50 datasets in terms of the non-bert-based approach. in order to utilize the capability of bert-based models @xcite , @xcite proposed a hierarchical attentive heterogeneous graph (hahsum) to improve the redundant phrases problem between extracted sentences of the summarization, which has achieved promising results on news article datasets such as cnn/dailymail, nyt, and newsroom. nevertheless, the model requires external analysis for modeling long-range dependencies. intuitively, transformer-based language models are not able to process long pieces of text. several works have provided promising results @xcite , however, the input length limitation and encoding of long texts are still open challenges in this research field @xcite . in this study, we take an investigation on improving the performance of the eds problem for long documents in which the core idea is to exploit the complex relationship in terms of both inter and intra-sentence connections using graph-based methods. specifically, hetergnn-based models are able to enrich the cross-sentence relations by adding a word node as the intermediary to connect sentences. however, the inter-sentence connections are not considered. specifically, only sentences with common words can have a connection, which might influence the performance, especially in terms of long-form document representation. therefore, we present a novel method for enriching the inter-sentences relations by proposing a homogeneous graph neural network (homognn) and incorporating the hetergnn for final sentence representations. in particular, inspired by recent stateof-the-art models for long-form document representhe output of the two phases is concatenated and put into an mlp layer in order to classify labels for each sentence. ## methodology given an arbitrary document @xmath0, .., s n } consisting n sentences, the objective of eds problem is to predict a sequence of a set of binary label {y 1 , .., y n }. specifically, y @xmath1, 1] represents the j th sentence, which should be included in the summary. our proposed model for the eds problem includes two learning layers, which execute simultaneously, such as the homogeneous graph layer and the heterogeneous graph layer. ## quality analysis ablation study. in our model, we enrich the complex relationships by exploring both heterogeneous graph and homogeneous graph operations for sentence connection. in order to explore the effectiveness of each component, we design different variants of the proposed model as follows: by increasing the maximum length of sentences, the performances are improved. in particular, the results indicated that tuning the max length of sentence value is able to enhance the performance. specifically, we take this issue into account for the future work of this study by executing our model with a longer maximum size of documents. ## conclusion this paper presents a novel graph-based method for the eds task, which focuses on exploiting the complex relationship in terms of both inter and intra-sentence relations of the long-form documents. specifically, two types of graph structures are developed for enriching sentence representations. the experiments on two benchmark datasets show promising results of the proposed method. regarding future work, segmentation methods are taken into account for dividing long documents into paragraphs. specifically, analyzing the complex relationships between paragraphs and integrating them into graphs as an additional node is able to enrich the information for the representation.