SAPG raph: Structure-aware Extractive Summarization for Scientific Papers with Heterogeneous Graph (2022)

Abstract: Scientific paper summarization is always challenging in Natural Language Processing (NLP) since it is hard to collect summaries from such long and complicated text. We observe that previous works tend to extract summaries from the head of the paper, resulting in information incompleteness. In this work, we present SAPGraph to utilize paper structure for solving this problem. SAPGraph is a scientific paper extractive summarization framework based on a structure-aware heterogeneous graph, which models the document into a graph with three kinds of nodes and edges based on structure information of facets and knowledge. Additionally, we provide a large-scale dataset of COVID-19-related papers, CORD-SUM. Experiments on CORD-SUM and ArXiv datasets show that SAPGraph generates more comprehensive and valuable summaries compared to previous works.

## introduction in recent years, scientific papers represented by covid-19-related papers have shown an expanding growth in a short period, which produces information overload and makes it difficult for researchers to follow. automatic summarization can help researchers quickly focus on valuable information in the article and be updated about the latest research progress. the goal of automatic summarization is to condense a long text into a concise summary while retaining essential information. it evolves mainly in two directions: abstractive and extractive methods. abstractive summarization generates summaries which are rewritten and refined @xcite , while the extractive one selects text segments as summaries @xcite @xcite , which bacille calmette-guérin (bcg) is an attenuated strain of mycobacterium bovis currently used as a vaccine against tuberculosis. global distribution and propagation of bcg has contributed to the in vitro evolution of the vaccine strain and is thought to partially account for the different outcomes of bcg vaccine trials. previous efforts by several molecular techniques effectively identified large sequence polymorphisms among bcg daughter strains, but lacked the resolution to identify smaller changes. in this study, we have used a nimblegen tiling array for whole genome comparison of 13 bcg strains. using this approach, in tandem with dna resequencing, we have identified six novel large sequence polymorphisms including four deletions and two duplications in specific bcg strains. moreover, we have uncovered various polymorphisms in the phop-phor locus. importantly, these polymorphisms affect genes encoding established virulence factors including cell wall complex lipids, esx secretion systems, and the phop-phor two-component system. our study demonstrates that major virulence factors are different among bcg strains, which provide molecular mechanisms for important vaccine phenotypes including adverse effect profile, tuberculin reactivity and protective efficacy. these findings have important implications for the development of a new generation of vaccines. is easier to be applied practically and keep grammar correct. in this work, we study the extractive summarization of scientific papers, which are much longer than news articles (see for scientific paper summarization, it is difficult to generate summaries from professional texts like covid-19-related papers, due to their long texts with complicated structures. to deal with the long text, classical deep learning methods simply truncate documents and may therefore discard useful information. other methods propose a better data structure, such as graph-based models @xcite @xcite or sliding window in sequence models @xcite . some scientific paper summarization studies have noticed the importance of writing structure in papers, to better deal with long text @xcite . these works consider the paper structure and try to manually pick sections as input @xcite , or they consider hierarchical features of a document @xcite . among the extractive methods, we notice that these works are still insufficient at dealing with papers and are prone to obtain summaries with head distribution problems, which means that systems tend to extract summaries from the beginning of the document (see figure 5 ). the reasons might be that sequence-based extractive summarization models are weak at establishing potential associations of distant sentences, despite the sliding window mechanism. and furthermore, the structure of long papers is not well-utilized because long documents always possess several facets with certain logical relations, as in figure 1 . hence, the extracted summaries are incomplete and cannot cover all the critical information that researchers need. to improve this problem, we propose a structure-aware paper heterogeneous graph network (sapgraph) for scientific paper summarization. inspired by @xcite and @xcite , facet structure is deeply considered in sap-graph. and the domain knowledge is also crucial for papers, which can be seen as a latent structure. based on these structures, sapgraph models an entire paper as a heterogeneous graph with three node types: section, sentence, and entity, and is trained with the graph neural network (gnn) @xcite . such a design can effectively aggregate information from different facets and improve the diversity and coverage of summaries. also, we provide cord-sum, a summarization dataset based on covid-19 open research dataset (cord-19) @xcite foot_0 . we compare sapgraph with strong extractive summarization models, and our experiments show that sapgraph outperforms previous works in terms of rouge @xcite and bertscore @xcite on cord-sum and arxiv @xcite . in our metrics, rouge-n and rouge-l can measure the similarity between system summaries and reference summaries by the n-gram co-occurrences and the longest common subsequence, and bertscore computes this similarity based on cosine similarities between their tokens' embeddings. ablation studies show our evaluation on different graph structures, suggesting that sapgraph can surpass other types of graph construction. our contributions are highlighted as follows: firstly, we provide cord-sum, a summarization dataset compiled of scientific papers about covid-19, and their summaries. the dataset and construction code are publicly available for researchers to process the updated cord-19 dataset. secondly, we propose sapgraph, a multi-layer heterogeneous graph for structure-aware paper summarization. sapgraph effectively models an entire paper with much fewer structural nodes and edges than state-of-the-art graphs. the final point is that results on the dataset of cord-sum and arxiv prove the effectiveness of our work. and our experiments show that sapgraph successfully utilizes the explicit structure of facets and the implicit structure of knowledge to alleviate the head distribution problem in scientific paper summarization. ## related work the study of extractive summarization of scientific papers has always been a hotspot. just as regular extractive summarization, systems for scientific papers aim to pick informative texts from the source document to form a summary, except that these documents are longer, more professional, and have a clear hierarchical structure. with the development of sequence neural networks, more rnn and transformer-based models are used for scientific paper summarization. sequence models like hierarchical rnn are used to build attention between different layers of the paper on arxiv and pubmed @xcite . global and local contexts are also considered when extracting sentences @xcite . dancer @xcite ) selects sections and makes multiple source-target pairs to generate summaries respectively. @xcite generate a summary from four aspects of emerald dataset, including purpose, method, findings, and value. @xcite use an extract-then-abstract model and pick out the introduction section as one input. for sequence-based methods, papers are too long to process directly. unlike vanilla sequence models accompanied by truncation of long text, scitldr @xcite performs extreme summarization from concatenated introduction and conclusion, which is more reasonable than treating every section equally. but other than shortening the text, sliding window @xcite @xcite is commonly used. for instance, longformer @xcite relieves the computational pressure caused by the attention mechanism with sliding window attention, and can be used on long text summarization as bert does @xcite . other pretrained language models such as scibert @xcite and biobert @xcite , which are pretrained on scientific literature or medical papers, are more adaptable to scientific document processing tasks. although some of the above works value the function of facet structure, the majority of them rely on manual selection, which lacks universality and may also result in the loss of supporting information. in contrast, graph-based models are more flexible and can build connections between long-span texts. early works like lexrank @xcite and textrank @xcite predict sentence centrality of a document graph. recently, more well-designed graph-based methods consider the structure information, such as pacsum @xcite , hipo-rank @xcite , far @xcite , etc. to rank sentences, they fuse together such information as hierarchical structure, sentence position, and sentence similarity. @xcite can learn nodes representation with neural networks. heterogeneous graph methods @xcite @xcite can consider more diverse information with multitype nodes and edges. in graph-based works, het-ersumgraph (hsg) @xcite ) is comparable to our sapgraph, but sapgraph takes into account the structure of facets and knowledge in the paper, making it a better graph prior to paper summarizing. ## approach here we describe three main stages of sapgraph: the facet alignment between summaries and source documents, the graph construction, and the learning method applied to the constructed graph. figure 2 shows the overall framework of sapgraph. ## conclusion in this paper, we propose sapgraph, a structureaware heterogeneous graph model for scientific paper extractive summarization. sapgraph can generate more comprehensive summaries while operating on much smaller graphs, with the well-designed graph construction considering the explicit structure of facets and implicit structure of knowledge. along with sapgraph, we propose cord-sum, a large structure-rich medical-domain scientific paper summarization dataset. detailed experiments and case studies prove the effectiveness of sap-graph on alleviating the head distribution problem. sapgraph can generate more comprehensive summaries on cord-sum and arxiv datasets than previous works. in the future, we will explore how to automatically learn graph structure and find a more effective way to integrate pretrained models and sapgraph.