Multi-Document Scientific Summarization from a Knowledge Graph-Centric View (2022)

Abstract: Multi-Document Scientific Summarization (MDSS) aims to produce coherent and concise summaries for clusters of topic-relevant scientific papers. This task requires precise understanding of paper content and accurate modeling of cross-paper relationships. Knowledge graphs convey compact and interpretable structured information for documents, which makes them ideal for content modeling and relationship modeling. In this paper, we present KGSum , an MDSS model centred on knowledge graphs during both the encoding and decoding process. Specifically, in the encoding process, two graph-based modules are proposed to incorporate knowledge graph information into paper encoding, while in the decoding process, we propose a two-stage decoder by first generating knowledge graph information of summary in the form of descriptive sentences, followed by generating the final summary. Empirical results show that the proposed architecture brings substantial improvements over baselines on the Multi-Xscience dataset.

## introduction nowadays, the exponential increasing publication rate of scientific papers makes it difficult and time-consuming for researchers to keep track of the latest advances. multi-document scientific summarization (mdss) is therefore introduced to alleviate this information overload problem by generating succinct and comprehensive summary from clusters of topic-relevant scientific papers @xcite . in mdss, paper content modeling and crosspaper relationship modeling are two main issues. to address the two aforementioned issues, we consider leveraging salient text units, namely entities, and their relations for mdss. scientific papers contain multiple domain-specific entities and relations between them. these entities and relations succinctly capture important information about the main content of papers. knowledge graphs based on these scientific entities and relations can be inherently used for content modeling of scientific papers. take figure 1 as an example. the knowledge graph at the top left illustrates the main content of paper 1, which can be formulated as: paper 1 uses memory augmented networks method to solve the life-long one-shot learning task, the evaluation is based on image classification benchmark datasets. furthermore, knowledge graphs can effectively capture cross-paper relationships through entity interactions and information aggregation. in figure 1 , paper 1, 2 and 3 adopt the same method memory networks to solve different tasks. this relationship is demonstrated in the graph of gold summary by sharing the method node memory networks. in this paper, we develop a transformerbased @xcite abstractive mdss model, which can leverage knowledge graphs to task method otherscien-tificterm image classification benchmark datasets e v a lu a t e -f o r few-shot video classification dataset ev alu at e-f or image classification benchmark datasets eva lua te-f or dataset more recently, memory networks have been employed for one-shot learning @paper1, few-shot learning @paper2 , and semi-supervised learning @paper3. @paper1 designed a memory augmented network that could do life-long one-shot learning. cmn @paper2 encoded videos into fixed-size features via a multi-saliency embedding algorithm. ma-dnn @paper3 leveraged the assimilationaccommodation interaction in memory networks for semi-supervised learning. guide paper representation and summary generation. specifically, in the encoding part, we fuse the knowledge graphs of multiple input papers into a unified graph and design a graph updater to capture cross-paper relationships and global information. besides, we build another graph based on the interaction between entities and sentences, and then apply an entity-sentence updater to enable information flow between nodes and update sentence representations. in the decoding part, knowledge graphs are utilized to guide the summary generation process via two approaches. the first is to incorporate the graph structure into the decoder by graph attention, and the second is inspired by deliberation mechanism @xcite . concretely, we introduce a two-stage decoder to make better use of the guidance information of knowledge graphs. the first-stage decoder concentrates on generating the knowledge graph of gold summary, while the second-stage decoder generates the summary based on the output of the first stage and the input papers. since the knowledge graph of gold summary is in the form of graph structure, we translate the graph into equivalent descriptive sentences containing corresponding entities and relations, called kgtext. kgtext serves as an information-intact alternative to the knowledge graph of gold summary and is generated in the firststage decoder, which we call the kgtext generator. we test the effectiveness of our proposed model on multi-xscience @xcite , a large-scale dataset for mdss. experimental results show that our proposed knowledge graph-centric model achieves considerable improvement compared with the baselines, indicating that knowledge graphs can exert a positive impact on paper representation and summary generation. the main contribution is threefold: (i) we leverage knowledge graphs to model content of scientific papers and cross-paper relationships, and propose a novel knowledge graph-centric model for mdss. (ii) we propose a two-stage decoder that introduces kgtext as intermediate output when decoding, which plays an important guiding role in the final summary generation. (iii) automatic and human evaluation results on the multi-xscience dataset show the superiority of our model. ## related work early mdss works are mainly based on artificially constructed small-scale datasets, using unsupervised extractive methods to extract sentences from multiple papers. @xcite take citation texts, paper abstracts and full paper texts as input for survey generation. they conduct the experiment with just two instances. @xcite construct 15 instances and combine a content model with a discourse model to generate coherent scientific summarizations. hoang and kan (2010) construct 20 instances, each with an annotated topic hierarchy tree, to generate summarization for multiple scientific papers. similar works also exist in @xcite @xcite . these unsupervised works are crude in both content modeling and relationship modeling and fail to generate high-quality summaries. some subsequent efforts apply deep learningbased methods to mdss using large-scale datasets @xcite @xcite . @xcite build a dataset with 8080 instances and construct a heterogeneous bibliography graph, and then utilize a cnn and rnnbased model for extractive summarization. @xcite collect 390,000 instances, and use a hierarchical encoder and a two-step decoder to generate summary in an abstractive manner for the first time. @xcite collect two large-scale datasets with 136,655 and 80,927 instances, respectively. they apply a transformer-based model to capture the relevance between papers for abstractive summarization. however, all the above works neglect salient semantic units to capture semantic information and relationships between papers. in this paper, based on mutli-xscience @xcite , we use knowledge graph information to model content and relationships between papers. ## conclusion in this work, we propose a knowledge graph-centric transformer-based model for mdss. our model is able to incorporate knowledge graph information into the paper encoding process with a graph updater and an entity-sentence updater, and introduce a two-stage decoder including a kgtext generator and a summary generator to guide the summary decoding with knowledge graph information. experiments show that the proposed model significantly outperforms all strong baselines and achieves the best result on the multi-xscience dataset. in the future, we will explore other more intuitive and effective methods to incorporate graph information in both the encoding and decoding phase of summary generation. 2020) is a heterogeneous graph-based extractive model with semantic nodes of different granularity. @xcite expands the pointergenerator network @xcite into a hierarchical network and integrates an mmr module. hiersumm @xcite ) is a transformer based model with an attention mechanism to share information cross-document for abstractive multi-document summarization. mgsum (jin et al., 2020) is a multi-granularity interaction network for abstractive multi-document summarization. we also consider evaluating on single document summarization models by concatenating multiple papers into a long sequence. graphsum @xcite ) is a neural multi-document summarization model that leverages well-known graphs to produce abstractive summaries. we use tf-idf graph as the input graph. pegasus @xcite is a sequence-to-sequence model with gapsentences generation as a pre-training objective tailored for abstractive summarization. pointer-generator @xcite is an rnn based model with an attention mechanism and allows the system to copy words from the source via pointing for abstractive summarization. bertabs (liu and lapata, 2019b) uses a pretrained bert @xcite as the encoder for abstractive summarization. we also report the performance of bertabs with an encoder (scibertabs) pretrained on scientific articles. bart @xcite is a pretrained text generation model.