Monash-Summ@ L ong S umm 20 S ci S umm P ip: An Unsupervised Scientific Paper Summarization Pipeline (2020)

Abstract: The Scholarly Document Processing (SDP) workshop is to encourage more efforts on natural language understanding of scientific task. It contains three shared tasks and we participate in the LongSumm shared task. In this paper, we describe our text summarization system, SciSummPip, inspired by SummPip (Zhao et al., 2020) that is an unsupervised text summarization system for multi-document in News domain. Our SciSummPip includes a transformer-based language model SciBERT (Beltagy et al., 2019) for contextual sentence representation, content selection with PageRank (Page et al., 1999), sentence graph construction with both deep and linguistic information, sentence graph clustering and within-graph summary generation. Our work differs from previous method in that content selection and a summary length constraint is applied to adapt to the scientific domain. The experiment results on both training dataset and blind test dataset show the effectiveness of our method, and we empirically verify the robustness of modules used in SciSummPip with BERTScore (Zhang et al., 2019a).

## introduction text summarization aims at automatically generating a fluent and coherent summary that mainly contains the salient information from the source document(s). two main categories are typically involved in the text summarization task, one is extractive approach @xcite which directly extracts salient sentences from the input text as the summary, and the other is abstractive approach @xcite @xcite which imitates human behaviour to produce new sentences based on the extracted information from the given document. in order to meet the requirements of modern data-driven methods, several large datasets have been presented. the majority of those datasets are for generic domain, but few available corpora from other task-specific domains. most of existing state-the-art summarization systems @xcite @xcite target news or simple documents, and they are less adequate for summarizing scientific work due to the length and complexity. those summarization systems cannot provide sufficient information conveyed in the scientific paper. the general domain have been paid enough attention, whereas the attention in scientific domain is far from enough. to address this point, the scholarly document processing (sdp) workshop @xcite is held to accelerate scientific discovery in research community, they appeal to researchers for designing a summarization system that can generate a relatively long summary for scientific work. since the release of transformer @xcite and bert @xcite , much research has been carried out on involving them in their system. @xcite modified the input sequence embedding and built several summarizationspecific layers for extractive summarization. similarly, @xcite present a novel document-level encoder based on bert @xcite for both extractive summarization and abstractive summarization. in their model structure, the lower transformer represents adjacent sentences and the higher layer with self-attention mechanism represents the multi-sentence discourse. these works leverage the advantage of deep neural network, not taking into account the linguistic information. in contrast, @xcite 1 construct semantic clusters and sentence graphs for multidocument summarization, which involves linguistic information and discourse markers. in this paper, we followed the framework of @xcite to construct our own unsupervised text summarization system. however, our model is different from the previous work: we modify the pipeline structure of multi-document summarization in the field of news to the single-document summarizer for summarizing scholarly documents, and we introduce two new steps to control the length of generated summary and to remove irrelevant sentences. our contributions in this work can be summarized in the following aspects: ## related work text summarization system most of recent text summarization systems leverage the advantages of deep neural networks, their encoderdecoder structures use either recurrent neural networks @xcite or transformer encoders @xcite . benefit of the sequence-to-sequence structure, a great progress in both extractive and abstractive document summarization is achieved. though abstractive summarization has more potentials to generate interpretations in a human-like fashion, it has been found that sometimes repeatedly produces the same phrase or sentence @xcite , which greatly reduces the comprehensibility and readability. in contrast, extractive summarization performs better in fluency aspect and it can grammatical and accurately represent the source text. one potential issue in extractive summarization is that not all of information from the extracted sentence is important, which leads more redundancy in the generated summary. in the work of @xcite , they apply graph structure and consider the discourse relationship between sentences rather than using encoderdecoder structure, and text compression is implemented in the final stage to reduce the redundancy in the generated sentences. however, their model is designed for multi-document summarization in the news domain, we extend their summpip to singledocument settings for scientific long articles. sentence embedding method term frequency-inverse document frequency (tf-idf) is widely used in traditional nlp, but it cannot capture the semantic information and contextual relationship between sentences. word2vec @xcite is used in summpip @xcite to capture contextualized relationship, but this embedding method cannot solve the polysemous problem. more recently, bert @xcite has achieved better performance in many nlp downstream tasks, but it is difficult to derive sentence embeddings. to solve this limitation, single sentences are passed to the bert and two common ways to extract sentence representation are widely used: averaging the outputs and using the output of the [cls] token @xcite . @xcite develops a repository, bert-as-aservice foot_1 , which accelerates the process of extracting token and sentence embeddings from bert @xcite . lately, in order to find a better way to derive semantically similar sentence from language models, reimers and gurevych (2019) present sbert. however, above works help facilitate workload in generic domain rather than task-specific domain. content selection graph is an intuitive structure for utilizing the relation information between sentences. some work @xcite focuses on selecting salient sentences by leveraging graph-based rank-characteristics extractive abstractive test dataset sci p ref s sci p ref s sci p range of corpus size (sentences) [37,629] [9,48] [24,792] [1,87] [119,345] median value of corpus size (sentences) 186 31 201 31 219 range of sentence length (words) [12,51] [15,48] [10,44] [0.5,54] [18,27] median value of sentence length (words) 26 27 26 21 22 where c is the set of candidate sentences, s is the set of extracted sentences, q is the query embedding, s i , s j are sentence embeddings of candidate sentences i and j, respectively. sim indicates the cosine similarity between two embeddings. though this approach have been proved that it outperforms generic summarization approaches in the information retrieval task, to our knowledge, there is no previous work compared it with pagerank algorithm on scientific long document summarization task. our work incorporates deep neural representations into both pagerank algorithm and mmr strategy and shows the comparison between these two methods in the field of scientific work for both extractive and abstractive summarization. ## dataset pre-processing the training dataset provided by the longsumm shared task consists of 2236 scientific papers, of which 1705 are for extractive method and 531 are for abstractive method. the reference extractive summaries are generated by talksumm @xcite that extracts sentences appeared in associated conference videos, while the abstractive summaries are collected from blogs written by researchers. ## system overview we adopt the summpip @xcite as our baseline model, and we modify the pipeline architecture for summarizing scholarly documents. two new steps are introduce for adapting scientific domain, one is to remove irrelevant sentences and the other is to control the length of generated summary. in the following subsections, we will specify each component in the scisummpip. ## conclusion and limitation in this paper, we have presented the modified unsupervised pipeline architecture, scisummpip, that leverages a transformer-based language model for summarizing scientific papers. we add content selection module and two steps to remove irrelevant sentences and to control the length of generated summary. after that, the linguistic knowledge will be incorporated into the process of multi-sentences compression for summarizing scientific work. the experiment results of automatic evaluation prove that our new pipeline significantly improves the overall performance on both training and blind test dataset. besides, through manual inspection we find that our model indeed capture the salient information from the given source document. however, we have to admit that the readability of generated summary needs to be improved. we incorporated deep neural representation into both mmr @xcite strategy and pagerank @xcite algorithm. even though mmr strategy performs better in information retrieval task, we empirically verified that it is not sufficient for our model to summarize scientific work. mmr is a query-biased approach and we chose the title as query in our implementation, thus the potential reason for worse performance may be the query we chose is not effective enough. to investigate a sentence embedding method for sufficiently summarizing scholarly document, we compared the performances among several embed-ding strategies and we also evaluated their performances on both rouge metric and bertscore metric. although averaging the output of scibert @xcite achieves better performance, the workload of using it to extract sentence embeddings is heavier than that of directly using sbert @xcite . there is enough work for generic domain while the attention paid for task-specific domain is far from enough, therefore we appeal to researchers for making more efforts on task-specific domain in their further research.