SEHY : A Simple yet Effective Hybrid Model for Summarization of Long Scientific Documents (2022)

Abstract: Long document (e.g., scientific papers) summarization is obtaining more and more attention in recent years. Extractive approaches attempt to choose salient sentences via understanding the whole document, but long documents cover numerous subjects with varying details and will not ease content understanding. Instead, abstractive approaches elaborate to generate related tokens while suffering from truncating the source document due to their input sizes. To this end, we propose a Simple yet Effective HYbrid approach, which we call SEHY, that exploits the discourse information of a document to select salient sections instead sentences for summary generation. On the one hand, SEHY avoids the full-text understanding; on the other hand, it retains salient information given the length limit. In particular, we design two simple strategies for training the extractor: extracting sections incrementally and based on salience-analysis. Then, we use strong abstractive models to generate the final summary. We evaluate our approach on a large-scale scientific paper dataset: arXiv. Further, we discuss how the disciplinary class (e.g., computer science, math or physics) of a scientific paper affects the performance of SEHY as its writing style indicates, which is unexplored yet in existing works. Experimental results show the effectiveness of our approach and interesting findings on arXiv and its subsets generated in this paper.

## introduction long-document tasks (e.g., scientific papers summarization @xcite and long-text reading comprehension @xcite ) have become one of long-term challenging tasks in natural language processing (nlp) because long documents cover numerous subjects with varying details and will not ease content understanding. for * corresponding author: dongning rao. example, scientific papers, whose abstracts can be used as ground-truth summaries, is a representative type of long documents with discourse information showing the hierarchical structure composed of tokens, sentences, paragraphs, and sections @xcite . extractive summarization approaches select important units such as phrases or sentences from the original text, but long documents cover numerous subjects with varying details and will not ease content understanding @xcite . instead, abstractive summarization approaches concisely paraphrase the information content while suffering from truncating the source document due to their input sizes @xcite . hybrid models exhibit a combination solution via first extracting salient sentences with an extractive model (i.e., extractor) and then generating a summary based on extracted sentences with an abstractive model (i.e., generator) @xcite . however, on the one hand, training an extractive model may be expensive due to the complex salience analysis; on the other hand, an abstractive model may generate inappropriate summary words due to the dependence on extracted sentences. thus, pipeline-style errors can be propagated and accumulated, leading to hybrid models perform worse than current state-of-the-art (sota) abstractive models @xcite . this suggests that exploring simple yet effective extractive approaches is crucial to improve the overall performance and decrease the training cost of a hybrid model. recently, the success of pre-trained language models (ptms) such as transformer @xcite in nlp brings great gain for abstractive models in the summarization task. however, transformer-based models usually suffer from the quadratic dependency on the sequence length due to their full attention mechanism. sometimes, the model's performance is mainly con-strained by its limitation on the sequence length. for instance, the average document length on arxiv @xcite is more than 6000 tokens while bart @xcite , which combines bert @xcite and @xcite , has a comparatively smaller length limit, 1024 tokens. besides, for a hybrid model, extracted sentences from its extractive model are often difficult to maintain the coherence of the source document, thus leading to the poor semantic representations by its abstractive model @xcite . to alleviate these issues, we propose a novel simple yet effective hybrid approach, which we call sehy, that exploits the discourse information of a document to select salient sections instead sentences for summary generation. we use simple strategies for choosing sections, not only for decreasing the training cost of the extractor, but also for enhancing the input-sequence's coherence to the generator. motivated by @xcite , which identifies and selects specific sections that are more informative, we propose two strategies: choosing specific sections (e.g., introduction or conclusion) based on the salience analysis and using the beginning sections without concerning the salience. after this, we use strong abstractive models to generate the final summary. to demonstrate the effectiveness of sehy, we answer the following questions in this paper: as the contents indicate, q1 is used to evaluate the two section-extraction strategies, q2 is used to measure different abstractive models which are responsible to generate the final summary, and q3 is used to estimate writing styles of scientific papers in different disciplines. the joint of q1 and q2 acts as ablation studies on the proposed hybrid model sehy. while, q3 is not explored yet in existing works where all scientific papers on arxiv are summarized without distinguishing their disciplinary properties (e.g., computer science, math or physics). for instance, a well-written computer science paper usually presents summary sentences in the introduction or conclusion section, but no experimental work has ever confirmed this. ## related work automatic text summarization is the task of producing a concise and fluent summary while preserving key information content and overall meaning. it aims to transform lengthy documents into shortened versions, something which could be difficult and costly to undertake if done manually. in this section, we focus on recent summarization models. for more text summarization technologies, we refer interested readers to a survey on this @xcite . ## method in this section, we first present two strategies to implement our extractive model (for answering q1), then describe multiple paired abstractive models (for answering q2), and finally explain how to generate data subsets with regard to disciplinary categories of scientific papers (for answering q3). ## conclusion long documents introduce a lot of noise to the summarization process and large parts of the document are not really key to its narrative and thus should be ignored. in this paper, we propose a simple yet effective hybrid model called sehy for summarizing long scientific documents. in particular, we design two simple strategies for selecting sections: p sal (sec) and p inc (k), and use strong abstractive models for generating the final summary. we conduct excessive experiments with multiple variants of sehy on the full arxiv dataset publicly available and three disciplinary sub-datasets generated by ourselves. experimental results show that p sal (sec) is more effective than p inc (k) and our best models obtain the competitive results with regard to the current sota on arxiv. future works are as follows. first, we will evaluate the generalization of our approach on other academic-paper datasets such as scisummnet @xcite , which is a large-scale scientific paper summarization dataset with citation networks. second, we will develop more flexible and sophisticated extraction strategies such as using reinforce learning @xcite , to further improve our approach's performance.