Two-Stage Graph-Augmented Summarization of Scientific Documents (2024)

Abstract: Automatic text summarization helps to digest the vast and ever-growing amount of scientific publications. While transformer-based solutions like BERT and SciBERT have advanced scientific summarization, lengthy documents pose a challenge due to the token limits of these models. To address this issue, we introduce and evaluate a two-stage model that combines an extract-then-compress framework. Our model incorporates a “graph-augmented extraction module” to select order-based salient sentences and an “abstractive compression module” to generate concise summaries. Additionally, we introduce the *BioConSumm* dataset, which focuses on biodiversity conservation, to support underrepresented domains and explore domain-specific summarization strategies. Out of the tested models, our model achieves the highest ROUGE-2 and ROUGE-L scores on our newly created dataset (*BioConSumm*) and on the *SUMPUBMED* dataset, which serves as a benchmark in the field of biomedicine.

## introduction the scientific community has experienced an unprecedented surge in the number of scientific publications @xcite . this exponential growth has resulted in a state of information overload, presenting both opportunities and challenges for researchers. navigating the vast amount of information, filtering out relevant studies, and extracting essential insights have become increasingly challenging @xcite . to address this issue, researchers have turned to automatic summarization methods, which leverage various nlp techniques to condense the essential findings, methodologies, and contributions of research articles into concise and informative summaries. the use of transformer-based language models (lms), such as bert @xcite , bart @xcite , scibert @xcite , and t5 @xcite has significantly improved text summarization. particularly scibert's performance in handling sciencerelated documents, and bertsum @xcite , an extension of bert for extractive summarization, have advanced scientific summarization in terms of domain-specific performance, accuracy, and coherency @xcite . large language models (llms) have further reshaped the field of text summarization. for example, openai's gpt-3 and its successors @xcite have shown remarkable capabilities in generating coherent and informative summaries @xcite . these models can perform both extractive and abstractive summarization with high accuracy and fluency. for instance, @xcite highlight how llms can generate summaries that not only capture the main ideas of the source text but also reformulate them in novel ways, often providing additional context or explanations. despite recent advancements in text summarization, there are strong reasons to continue refining bert-based models for scientific summarization: scibert, for instance, is trained on scientific texts, which provides with an edge in identifying academic papers' unique language and structure. they are also computationally more efficient and have smaller memory requirements than llms, making them faster and more accessible for processing large volumes of scientific papers @xcite . additionally, such models offer greater interpretability, allowing for insights into the summarization process, which is crucial in the context of science @xcite . their open-source nature further aligns with scientific principles of transparency and reproducibility, reducing privacy concerns associated with proprietary models like gpt. although bert-based models offer numerous advantages for scientific summarization, they struggle with processing lengthy documents due to token limits. to address this issue, two-stage summarization models have been proposed, where the first stage focuses on identifying and extracting salient sentences or passages, and the second stage on generating a concise and coherent summary based on the extracted sentences @xcite @xcite . in this paper, we build upon the previous work and propose a model for summarizing scientific documents that incorporates the extractthen-compress framework. our model integrates a "graph-augmented extraction module" that first selects order-based salient sentences from the complete text of long documents in the initial step (see §4.1), and then employs an "abstractive compression module" to generate concise and relevant summaries using the extracted drafts (see §4.2). we test our model on two datasets: first, sumpubmed, a benchmark dataset for abstractive summarization of biomedical scientific articles @xcite . second, bioconsumm, a new dataset that we created, which was curated for the purpose of this work and is from the domain of biodiversity conservation. one limitation of existing scientific text summarization tasks is their strong reliance on datasets from well-established domains like computer science and biomedicine, neglecting other research areas. this lack of attention has resulted in a shortage of comprehensive exploration and rich datasets in these underrepresented domains. to address this disparity, we introduce the bioconsumm dataset, which also serves as a valuable resource for training and evaluating text summarization models across domains. with the bioconsumm dataset, we can investigate the unique challenges and characteristics of summarizing scientific texts in lowresourced fields, assess the effectiveness of existing summarization techniques, and develop domainspecific or domain-agnostic models. ## methodology this section presents our proposed framework for long document summarization, which follows the extract-then-compress paradigm. our model incorporates a graph-augmented extraction module that extracts salient sentences from the full text of long documents as drafts, and a subsequent abstractive compression module that generates concise and coherent summaries based on the extractive drafts. ## experiments in this section, we first introduce implementation details, and then show experimental results from both quantitative and qualitative evaluation. ## conclusions and future work this paper presents a novel dataset for summarizing scientific articles from the domain of biodiversity conservation, which distinguishes it from existing datasets in this field. additionally, we proposed a two-staged summarization model that employs the "extract-then-compress" approach to effectively summarize lengthy scientific documents. to evaluate the effectiveness of our model, we compared its performance using both our newly introduced dataset and a benchmark summarization dataset from the biomedical domain. the results demonstrate that our model outperforms well-established summarization methods.