Generating Extractive and Abstractive Summaries in Parallel from Scientific Articles Incorporating Citing Statements (2023)

Abstract: Summarization of scientific articles often overlooks insights from citing papers, focusing solely on the document’s content. To incorporate citation contexts, we develop a model to summarize a scientific document using the information in the source and citing documents. It concurrently generates abstractive and extractive summaries, each enhancing the other. The extractive summarizer utilizes a blend of heterogeneous graph-based neural networks and graph attention networks, while the abstractive summarizer employs an autoregressive decoder. These modules exchange control signals through the loss function, ensuring the creation of high-quality summaries in both styles.

## introduction text summarization automates condensing documents while preserving key information. most neural summarization models, like those by @xcite ; @xcite , are designed for shorter texts, e.g., the cnn/daily mail dataset @xcite . however, applying these models to longer documents, such as scientific research papers, remains limited. in scientific document summarization, it is common to focus solely on abstracts, introductions, and conclusions, as demonstrated in @xcite 's work. summarizing scientific publications presents unique challenges due to their length, complex concepts, technical jargon, structured organization, and citations. these complexities make it a more daunting task compared to summarizing other types of documents. additionally, the long-term impact of a scientific article may not be fully evident when it is first published, as its significance can evolve over time. while an abstract provides an initial overview from the authors' perspective, it may not capture the full extent of the paper's influence on the research community and its evolving impact @xcite . as an example, we can consider the abstract from @xcite : we present an approach to pronoun resolution based on syntactic paths. . . . we learn the likelihood of coreference between a pronoun and a candidate noun based on the path in the parse tree between the two entities. . . . highly coreferent paths also allow mining of precise probabilistic gender/number information. we combine statistical knowledge with well known features in a support vector machine pronoun resolution classifier. significant gains in performance are observed on several datasets. this abstract gives insight into the methods the authors used. but the citations emphasize the corpus it presents. for example: we use the approach of @xcite , both because it achieves state-ofthe-art gender classification performance, and because a database of the obtained noun genders is available online. @xcite for the gender task that we study in our experiments, we acquire class instances by filtering the dataset of nouns and their genders created by @xcite . @xcite jaidka et al. @xcite have identified this missing aspect in scientific document summarization and addressed it by introducing a shared task. this task aims to create summaries that take into account not only the information in the body of the documents but also the research community's overview of the documents over time. the work described here continues in this direction. with the advancement of neural networks, there have been a few prominent research works in recent years for generating extractive @xcite and abstractive @xcite summaries from scientific documents @xcite . extractive summarization recognizes key sentences from the source document as the summary but lack the flow of information, whereas the abstractive summarization technique generates new phrases using language models while preserving the semantics of the input document but may miss some important aspects of the text. this is a motivation for designing a model to generate both summaries in parallel and help the counterpart to achieve a performance boost with additional guidance. a key step in extracting brief synopsis sentences from a manuscript is to map the cross-sentence correlations. a lot of recent prominent works @xcite have tried to do so using recurrent neural networks (rnns). however, because of using rnns, these models fail to capture longdistance sentence-level dependencies. another approach to preserve sentence-level dependencies from long documents is using graph-based neural networks. a few recent works (e.g., @xcite ) have utilized discourse information in the article along with inter-sentence correlations for constructing graphs and summarizing document. another approach is to construct a sentence-level fully connected graph. zhong et al. @xcite and @xcite used transformer @xcite encoders to determine how sentences interact with each other. wang et al. @xcite introduced an heterogeneous graph neural network for extractive summarization which used additional semantic units (words) as intermediate nodes to construct relationships between sentences. abstractive summarizers focus heavily on form, with the goal of producing a generalized summary, which tends to necessitate complex languagegenerating models. these models are typically based on sequence-to-sequence (seq2seq) architectures, in which a source document is seen as one sequence whereas its summary as another. the majority of previous research on neural abstractive summarization depended on large-scale, high-quality datasets of supervised document-summarization pairings @xcite . recently, state-ofthe-art solutions on abstractive summarization are built upon the transformer @xcite and bert @xcite models. these attention-based abstractive models are being used in different fields like clinical note summariza-tion @xcite , scientific document summarization @xcite , and layabstract generation @xcite . in this paper, addressing the above-mentioned issues, we have built a standalone summarization model which can generate both extractive and abstractive summaries from scientific documents incorporating the citation network. analyzing the citation network, citing statements from the citing articles are accumulated with the original text document to incorporate the research community's observation on that particular cited manuscript. these summaries are the abstracts of the original papers with additional information reflecting the research community's view. after that, we run the long-former @xcite encoder to generate sentence and word representations and train extractive and abstractive summarizers together. for the extractive summarizer, an heterogeneous graph neural network @xcite ) is used as it has the ability to preserve sentence-level dependencies utilizing additional semantic units as intermediate nodes in the graph representation. abstractive summaries are generated by the autoregressive decoder. the loss function is defined in such a way that both summarizers can achieve better rouge and me-teor scores. furthermore, we have developed a corpus containing 10k research articles along with their corresponding citation statements and is a subset of the semantic scholar network (ssn) corpus. the citation statements are collected utilizing the citation graph used in the ssn corpus. in short, the contributions of this work are: ## related work text summarization aims to distill a document's essence efficiently. recent nlp research has yielded effective neural summarization models, particularly those using transformer and bert-based architectures. work summarizing lengthy scientific documents often focuses on specific sections rather than the entire text @xcite or citation statements @xcite . ## methodology this section defines the problem of scientific document summarization using a citation graph. then, the two benchmark datasets used for the scientific article summarization experiments are discussed along with the pre-processing procedures. finally, the proposed deep learning model is explained. ## experimental results and analysis this section gives a brief description of the model parameters used in the experiments as well as the results achieved on cl-scisumm-2020 and the customized ssn datasets. ## conclusion and future work in this paper, we have introduced a summarizer model considering two intentions: first, summarize scientific documents incorporating citation contexts, and second, build a summarizer model which can generate both extractive and abstractive summaries by means of parallel training so that both counterparts can gain a performance boost. for this, we have utilized both the sentence-sentence and sentence-word correlations. furthermore, we have constructed a corpus comprising 10k scientific articles with their corresponding citation statements for the summarization task. the experimental results show that our model performs well compared to other well-known methods. though this work considers the research community's observations (citing statements), it doesn't consider the background information (references presented in the target article). in our future work, we are planning to use both sides of the citation graph (references as the background knowledge and the citing statements as the research community's views) while summarizing a scientific article.