Contrastive Hierarchical Discourse Graph for Scientific Document Summarization (2023)

Abstract: The extended structural context has made scientific paper summarization a challenging task. This paper proposes CHANGES, a contrastive hierarchical graph neural network for extractive scientific paper summarization. CHANGES represents a scientific paper with a hierarchical discourse graph and learns effective sentence representations with dedicated designed hierarchical graph information aggregation. We also propose a graph contrastive learning module to learn global theme-aware sentence representations. Extensive experiments on the PubMed and arXiv benchmark datasets prove the effectiveness of CHANGES and the importance of capturing hierarchical structure information in modeling scientific papers.

## introduction extractive document summarization aims to extract the most salient sentences from the original document and form the summary as an aggregate of these sentences. compared to abstractive summarization approaches that suffer from hallucination generation problems @xcite , summaries generated in an extractive manner are more fluent, faithful, and grammatically accurate, but may lack coherence across sentences. recent advances in deep neural networks and pre-trained language models @xcite have led to significant progress in single document summarization @xcite @xcite . however, these methods mainly focus on short documents like news articles in cnn/dailymail @xcite and new york times @xcite , and struggle when dealing with relatively long documents such as scientific papers. the challenges of lengthy scientific paper summarization lie in several aspects. first, the extended input context hinders cross-sentence relation modeling, the critical step of extractive summarization @xcite . thus, sequential models like rnn are incapable of capturing the long-distance dependency between sentences, and hard to differentiate salient sentences from others. furthermore, scientific papers tend to cover diverse topics and contain rich hierarchical discourse structure information. the internal hierarchy structure, like sections, paragraphs, sentences, and words, is too complex for sequential models to capture. scientific papers generally follow a standard discourse structure of problem definition, methodology, experiments and analysis, and conclusions @xcite . moreover, the lengthy input context also makes the widely adopted self-attention transformer-based models @xcite inapplicable. the input length of a scientific paper can range from 2000 to 7, 000 words, which exceeds the input limit of the transformer due to the quadratic computation complexity of self-attention. thus, sparse transformer models like bigbird @xcite and longformer @xcite are proposed. recently, researchers have also turned to graph neural networks (gnn) as an alternative approach. graph neural networks have been demonstrated to be effective at tasks with rich relational structure and can preserve global structure information @xcite @xcite . by representing a document as a graph, gnns update and learn sentence representations by message passing, and turn extractive summarization into a node classification problem. among all attempts, one popular way is to construct crosssentence similarity graphs @xcite , which uses sentence representation cosine similarity as edge weights to model cross-sentence semantic relations. @xcite proposed using rhetorical structure theory (rst) trees and coreference mentions to capture cross-sentence discourse relations. @xcite proposed constructing a word-document het-erogeneous graph by using words as the intermediary between sentences. despite their success, how to construct an effective graph to capture the hierarchical structure for academic papers remains an open question. to address the above challenges, we propose changes (contrastive hierarchical graph neural network for extractive summarization), a hierarchical graph neural network model to fully exploit the section structure of scientific papers. changes first constructs a sentence-section hierarchical graph for a scientific paper, and then learns hierarchical sentence representations by dedicated designed information aggregation with iterative intra-section and inter-section message passing. inspired by recent advances in contrastive learning @xcite , we also propose a graph contrastive learning module to learn global theme-aware sentence representations and provide fine-grained discriminative information. the local sentence and global section representations are then fused for salient sentence prediction. we validate changes with extensive experiments and analyses on two scientific paper summarization datasets. experimental results demonstrate the effectiveness of our proposed method. our main contributions are as follows: ## method given a document @xmath0, s 2 , ..., s n } with n sentences and m sections, we first represent it as a hierarchical graph and formulate extractive summarization as a node labeling task. the objective is to predict labels y @xmath1, 1) for all sentences, where y @xmath2, respectively. the overall model architecture of changes is shown in figure 1 . changes consists of two modules: a graph contrastive learning module to learn global theme-aware sentence representations and a hierarchical graph layer module to learn hierarchical graph node representations with iterative message passing. the learned sentence node and section node representations will be used as indicators for salient sentence selection. ## conclusion in this paper, we propose changes, a contrastive hierarchical graph-based model for scientific paper extractive summarization. changes first leans global theme-aware sentence representations by graph contrastive learning module. moreover, changes incorporates the sentence-section hierarchical structure by separating intra-section and inter-section message passing and aggregating both global and local information for effective sentence embedding. automatic evaluation on the pubmed and arxiv benchmark datasets proves the effectiveness of changes and the importance of capturing both semantic and discourse structure information in modeling scientific papers. in spite of the strong zero-shot performance of large language models like chatgpt on various downstream tasks, long document modeling is still a challenging problem in the llm era. transformer-based gpt-like systems still suffer from the attention computing complexity problem and will benefit from effective and efficient modeling of long documents.