Scientific Paper Extractive Summarization Enhanced by Citation Graphs (2022)

Abstract: In a citation graph, adjacent paper nodes share related scientific terms and topics. The graph thus conveys unique structure information of document-level relatedness that can be utilized in the paper summarization task, for exploring beyond the intra-document information.In this work, we focus on leveraging citation graphs to improve scientific paper extractive summarization under different settings.We first propose a Multi-granularity Unsupervised Summarization model (MUS) as a simple and low-cost solution to the task.MUS finetunes a pre-trained encoder model on the citation graph by link prediction tasks.Then, the abstract sentences are extracted from the corresponding paper considering multi-granularity information.Preliminary results demonstrate that citation graph is helpful even in a simple unsupervised framework.Motivated by this, we next propose a Graph-based Supervised Summarizationmodel (GSS) to achieve more accurate results on the task when large-scale labeled data are available.Apart from employing the link prediction as an auxiliary task, GSS introduces a gated sentence encoder and a graph information fusion module to take advantage of the graph information to polish the sentence representation.Experiments on a public benchmark dataset show that MUS and GSS bring substantial improvements over the prior state-of-the-art model.

## introduction text summarization is to automatically glean the most important concepts from an article, removing secondary or redundant concepts. among various summarization tasks such as news summarization @xcite , dialog summarization @xcite , and timeline summarization @xcite , scientific paper summarization remains a challenging task, since scientific papers are usually longer, and full of complex concepts target paper abstract: ...we propose a multi-granularity interaction network for extractive and abstractive multi-document summarization... we employ attention mechanisms to interact between different granularity of semantic representations... our proposed model outperforms all strong baseline methods... paper 1 abstract: ...we propose a unified model combining the strength of extractive and abstractive summarization... paper 2 abstract: ...we propose hibert (hierachical bidirectional encoder representations from transformers)... paper 3 abstract: ... our empirical analysis shows state-of-the-art performance on several multi document datasets. paper 4 abstract: ...the hidden markov model (hmm) for part-of-speech (pos) tagging is typically based on tag trigrams... paper 5 abstract: ...paraphrase recognition is used in a number of applications such as tutoring systems... ... and domain-specific items in specific fields @xcite . existing works on extractive scientific paper summarization mainly focus on utilizing intradocument relationships. for example, @xcite divided papers into sections and incorporated the global and local context for extractive summarization. @xcite proposed an unsupervised graph-based ranking model, which assumes a hierarchical graph representation of the scientific papers. however, the effectiveness of the citation graph in extractive summarization tasks is left to be explored. in the citation graph domain, nodes are inherently linked and dependent on each other. correspondingly, we assume that a good scientific paper abstract should be able to capture this structural information. figure 1 demonstrates this intuition. a good paper abstract of the query paper is more rel-evant to the directly cited paper 1, 2, and indirectly cited paper 3 by sharing domain-specific terms, and not relevant to paper 4 and 5 that are far from it in the citation graph. based on this observation, we first propose a multi-granularity unsupervised summarization (mus) as a light solution to the task without the requirement of rich supervision information. mus first finetunes a pre-trained encoder model on the citation graph to obtain better sentence and document representations by solving a link prediction task. then, mus transforms the citation graph into a multi-granularity graph. sentences are then extracted considering the multi-granularity positionaware centrality. experimental results demonstrate that citation graph information can enhance the summarization even by this simple unsupervised framework. motivated by this, we further propose a graphbased supervised summarization model (gss) to achieve more accurate results for the scenario where large-scale labeled data is feasible. firstly, a graph neural network encoder based on a pretrained language model is employed to obtain sentence representations for the target document and document representations for related papers in the citation graph. secondly, we propose a gated sentence encoder that polishes the sentence representations based on their relatedness to the document gist. then, a graph information fusion module is utilized to incorporate the information from reference papers to the polished sentence representations. finally, a multi-task framework is applied to the model, which jointly assigns selection weights to extract abstracts and predicts whether there exists an edge between two nodes. under the supervision setting, the graph information gives stronger guidance in two ways. in one way, we employ the link prediction task on the graph to obtain better document representation, and in the other way, the document representation is used to polish sentence representations. results show that our model significantly surpasses the prior state-of-the-art model by on public benchmark dataset. our contributions can be summarized as follows: ## related work extractive summarization aims to generate a summary by integrating the salient sentences in the document. traditional extractive summarization methods are mostly unsupervised, which directly select sentences based on explicit features or graphbased methods. unsupervised methods can save a lot of manual work and expenses since it does not demand labeled data. more recently, benefiting from the success of neural sequence models and large-scale datasets, neural network-based supervised summarization models have been proposed @xcite . more recetly, @xcite and @xcite showcased how pre-trained models can be applied in extractive text summarization tasks. these models often achieve better performance since labeled data is provided. research on summarizing scientific articles has been studied for decades @xcite . in the unsupervised domain, @xcite proposed to extract citationcontexts from citing articles, which does not help draft an abstract when the paper has not been cited yet. @xcite came up with an unsupervised graph-based ranking model for extractive summarization. as for supervised methods, @xcite performed an extractive step before generating a summary, which is then used as guidance for abstractive summarization. @xcite incorporated both the global context and local context to summarize papers. early approaches for extractive summarization, such as lexrank and textrank @xcite , have taken advantage of graph structures with inter-sentence cosine similarity. as for the neural-based approaches, @xcite designed a graph attention-based transformer encoder to generate a summary with the help of knowledge graphs extracted from scientific texts. graph modeling is also explored in abstractive summarization. for example, @xcite proposed a citation graph-based summarization model which incorporates inter-document information of the source paper and its references. introduction related work sentence 1 .... sentence 2 .... sentence 3 .... __________________________ __________________________ __________________________ __________________________ __________________________ paper 1 __________________________ __________________________ __________________________ __________________________ __________________________ __________________________ paper 2 __________________________ __________________________ __________________________ __________________________ __________________________ __________________________ __________________________ __________________________ paper 4 paper 3 example of a multi-granularity graph for unsupervised summarization on a toy scholar dataset that contains four papers (paper 1,2, and 4 is cited by paper 4). we select sentence 2 in paper 3 as the centered node to denote the multiple granularities (sentence-to-sentence, sentence-to-section, and sentence-to-paper). figure 2 : example of a multi-granularity graph for unsupervised summarization (paper 1,2, and 4 are cited by 3). we select sentence 2 in paper 3 as the centered node to denote the multiple granularities (sentence-sentence, sentence-section, and sentence-document). ## problem formulation we define a citation graph on the whole dataset, which contains scientific papers and citation relationships. each node represents a scientific paper in the dataset, and each edge indicates the citation relationship between two papers. each paper in the graph consists of a list of sentences @xmath0, ..., s ts }, where t s is the number of sentences in the document. our model aims to generate a score vector @xmath1, ..., y ts } for each sentence, where each score y @xmath2, 1] denotes the sentence's extracting probability. for the unsupervised model, the model outputs the y score without supervision. for the supervised model, we annotate the given sentences by a gold label vector y ′ . during the training process, the cross-entropy loss is measured and minimized between y and y ′ . ## multi-granularity unsupervised summarization model in this section, we propose our multi-granularity unsupervised summarization model (mus). mus combines a finetuned pre-trained encoder with a multi-granularity model on the citation graph. ## graph-based supervised summarization model we then introduce a graph-based supervised summarization model (gss), which uses the graph structure to extract abstracts under supervision, as shown in figure 3 . ## conclusion researchers usually write an abstract of a paper by referring to some examples, especially from a large number of papers on the same topic. hence, in this paper, we propose to leverage citation graphs to enhance scientific paper extractive summarization. concretely, we propose a multi-granularity unsupervised summarization model and a graph-based supervised summarization model, both of which outperform the state-of-the-art baselines by a large margin. in the future, we will explore the effect of other graph structure information on the summarization task such as pairwise distance information.