Multi Perspective Scientific Document Summarization With Graph Attention Networks ( GATS ) (2022)

Abstract: It is well recognized that creating summaries of scientific texts can be difficult. For each given document, the majority of summarizing research believes there is only one best gold summary. Having just one gold summary limits our capacity to assess the effectiveness of summarizing algorithms because creating summaries is an art. Likewise, because it takes subject-matter experts a lot of time to read and comprehend lengthy scientific publications, annotating several gold summaries for scientific documents can be very expensive. The shared task known as the Multi perspective Scientific Document Summarization (Mup) is an exploration of various methods to produce multi perspective scientific summaries. Utilizing Graph Attention Networks (GATs), we take an extractive text summarization approach to the issue as a kind of sentence ranking task. Although the results produced by the suggested model are not particularly impressive, comparing them to the state-of-the-arts demonstrates the model’s potential for improvement.

## introduction a summary is a clear and accurate representation of the input text that distills the main ideas from the source. it is important to maintain the text's interword and inter-sentence reliance. a novel method for ascertaining an article's main objective is text summarization. the article summary assists users in rapidly determining whether a paper is pertinent to their study areas and focusing on them. regardless of the type of documents that need to be summarized, there are two methods for automatic text summarization: extractive and abstractive. while abstractive summarization attempts to recreate the key content in a fresh way after interpreting and analyzing the text with more sophisticated techniques, extractive summarization is based on identifying important sections of the text and producing a subset of the sentences from the original text @xcite @xcite . for news articles, automatic summary has recently produced impressive results; nevertheless, summarizing scholarly publications has gotten less attention @xcite @xcite . published papers differ from other sorts of material, including news, in a few key respects. they are typically longer and feature more complex subjects and technical jargon. scientific publications are also citeable and contain citations. additionally, these documents usually contain tables, charts, and figures, which complicates the summary process. last but not least, another characteristic of scientific publications is that they may have unintended effects after being published. the majority of the current research on scientific document summarization assumes only one optimal gold summary. because creating summaries is a subjective process, having only one perfect summary makes it difficult to assess how well summarization systems are working. on the other hand, annotating several gold summaries for scientific publications can be quite expensive because it calls for specific topic experts to read and comprehend lengthy scientific documents. as the first collaborative activity, multi perspective scientific document summarization aims to investigate techniques for producing multiperspective summaries. in this attempt, we proposed a model using graph attention networks (gats) with data preparation based on transformers to deal with the issue. the remainder of this paper is organized as follows. recent related work is presented in the next section. sections 3 and 4 are dedicated to the model explanation and the experiments' results, respectively, and finally, the paper is ended by section 5 as a conclusion. ## related work although scientific document summarization has been studied for a long time, there are still many outstanding questions about how to do it effectively @xcite ; @xcite ; @xcite . @xcite has reported on the state-of-the-arts' results in abstractive and extractive summarization in the general text domain (news). the authors used pretrained encoders to build their summarizers and provided a two-stage technique in which the encoder is fine-tuned twice, once for extractive summarizing and once for abstractive summarizing. no official model that can provide a reasonable level of data independence has been mentioned @xcite . by choosing important passages from a text and replicating them word for word, extractive summarization creates a subset of the original text's phrases. on the other hand, an abstractive summarizer recreates crucial content in a new way after reading and analyzing the text using sophisticated natural language algorithms to create a new shorter text that offers the most important information from the original one el-kassas et al. (2021); @xcite ; @xcite . from another perspective, scientific paper summarization may be classified into two types: summarization based on content or citation sefid and giles (2022); @xcite . the summarizer just accepts the content of a document as input in content-based summarizing. in citationbased summarization, along with the original paper's content, external knowledge in the form of citations is also leveraged. the community has given those citations for the paper at hand. the majority of current studies in this domain are of the second category. nevertheless, being cited by other research works is required here, which means that newly published papers may not be accurately summarized in their initial days of publication. @xcite proposed one of the first models for the scientific text summarization task. they suggest a clustering method where communities are generated in the lexical network of the citation summary and sentences are retrieved from various clusters. they claimed that , for this particular issue, their method outperforms lexrank, one of the most widely used multi-document summarizing algorithms. scisummnet, @xcite , is a large annotated corpus for scientific paper summarization considering the papers' ci-tations. this dataset is suitable for data-driven approaches due to its size. besides the corpus, the author presented a graph convolutional network for the paper summarization. @xcite used the citation graph to improve the work of summarizing scientific papers. in order to produce the final abstract, summarization algorithms specifically can find relevant data from the relevant research community from the citation graph, in addition to using the document information from the original publication. additionally, they created a novel citation graph-based model that takes into account both the features of an article and its references. scisummpip @xcite , is another unsupervised paper summarization pipeline which uses a transformer-based language model for contextual text representation and pagerank for sentence selection. @xcite , proposed a model to generate summaries for long papers. they used high source compression in their system for creating summaries, which entails complicated domainspecific language that needs to be understood by experts. scibertsum @xcite ,is another model designed for summarizing lengthy texts, such as scientific publications with more than 500 sentences. by 1) incorporating a section embedding layer to include section information in the sentence vector and 2) employing a sparse attention mechanism, which allows each sentence to pay attention to nearby sentences locally while only a small number of sentences pay attention to all other sentences globally, scibertsum extends bert-sum to long documents. another scientific document summarization system is proposed by @xcite . they introduced a new approach for summarizing scientific documents that makes use of multi-objective differential evolution. making use of citation contextualization, different important sentences are first retrieved. the idea of multiobjective clustering is used to further group these sentences. ibrahim altmami and el bachir menai (2022), summarized almost all the recent works in the domain of scientific article summarization. they categorized the work based on different factors and reported the achieved results in terms of popular evaluation metrics. ## proposed model we adopted an extractive summary technique for the mup shared task. for this, we generate a graph for each article after selecting the three sentences that are closest to the each summary sentences based on the cosine similarity between the their sentence embeddings. the oracle rank is defined as the normalized average cosine similarity score between each sentence and the provided summaries' embeddings. figure 1 demonstrates the suggested model. ## experiments the datasets made available by the task organizers were used for all of the experiments. we made use of two-layers gats implemented with the pytorch-geometrics library with 10 epochs and a learning rate of 0.0001 to train the suggested model. applying the trained model on new data, the highly ranked sentences are selected as ## conclusion in this study, we used graph attention networks to perform the multi-perspective scientific documents summary problem while adhering to the extractive summarization methodology. we first chose the three sentences from the input article that most closely resembled each summary sentence to produce the graphs for our node ranking task. then, using each selected sentence as a node, the sentence embedding produced by the pretrained transformerbased language model is taken as the node features, and the dot product between the pairs of nodes is taken as the corresponding edge feature. because of the discrepancy between the results published by other teams and the ones obtained by the suggested model , it can be inferred that preprocessing techniques and the use of external knowledge may improve the results.