GUIR @ L ong S umm 2020: Learning to Generate Long Summaries from Scientific Documents (2020)

Abstract: This paper presents our methods for the LongSumm 2020: Shared Task on Generating Long Summaries for Scientific Documents, where the task is to generatelong summaries given a set of scientific papers provided by the organizers. We explore 3 main approaches for this task: 1. An extractive approach using a BERT-based summarization model; 2. A two stage model that additionally includes an abstraction step using BART; and 3. A new multi-tasking approach on incorporating document structure into the summarizer. We found that our new multi-tasking approach outperforms the two other methods by large margins. Among 9 participants in the shared task, our best model ranks top according to Rouge-1 score (53.11%) while staying competitive in terms of Rouge-2.

## introduction the task of document summarization aims at generating a short-form (summary) of a longer sequence of text (source) conveying the key points of the input text. this task can be generally performed in two ways: 1) extractive: where the system finds the salient sentences within the source and concatenates them to form the summary @xcite @xcite @xcite ; and 2) abstractive: where the model conducts text generation, paraphrasing, and produces novel words that are not necessarily present in the source text @xcite @xcite @xcite @xcite . over the recent years, the task of summarizing scientific papers has attracted researchers' attention. this is due to linguistic challenges inherent to scientific domain and the longer length of the documents (i.e., scientific papers) in comparison with the documents in other domains such as news. most prior works in scientific summarization have focused on producing short-form summaries which are around 200 tokens per summary @xcite @xcite , rather than long-form summaries. producing the summary at such length might be adequate when the source document is also of shorter form such as those in the news domain. nevertheless, when summarizing longer documents such as scientific papers, producing short-length summary (i.e., abstract-like) more favors a high-level view of the source document, rather than covering all the salient information within a given source text. producing such long summaries requires a deep and comprehensive understanding of specific scientific domain. generating long summaries of the paper is helpful for researchers who might want to learn more about the paper beyond abstract-level information, without the need to read the entire paper. the longsumm 2020 shared task foot_0 aims to encourage the research at generating longer-form summaries for scientific papers, and we progress this challenge by our participation in this challenge, utilizing pre-trained transformer encoders for summarization task. in our experiments, we explore three different methods for the challenge including 1) experimenting with different versions of pre-trained transformer encoders finetuned for summarization task as an extractive method @xcite ; 2) a two-stage method where an abstarctive summarizer is added to the extractive summarizer to produce abstractive summaries; and 3) a novel multi-task learning model which aims at jointly incorporating documents' discourse structure into the extractive summarizer. this is inspired by the fact that having close attention to the scientific paper's discourse information would result in improved summaries @xcite . we report comptetive results of our model, which achieves rouge scores of 53.11%, 16.17%, and 20.34% for rg-1, rg-2, and rg-l, respectively. with the obtained results, our system ranks 1st (rg-1), 2nd (rg-2), and 4th (rg-l) in terms of evaluation metrics. ## dataset the dataset provided for this challenge consists of two types of summaries: in our experiments, we use the extractive set along with 50% of abstracitve set as our training set, containing 1969 papers; and the other half of abstractive set is used as validation and test datasets. it has to be mentioned that the official test set (blind) also contains 22 abstractive papers. ## methodology in this section, we discuss our methods with different configurations submitted to the shared task. ## experimental setup as the initial parameters of the bertsum, and bart, we used the default hyper-parameters as denoted in the original papers @xcite . we used huggingface's transformers library for working with bart foot_2 , and also the open implementation for experimenting with bertsum foot_3 . in order to provide groundtruth labels for the task of section prediction, we utilized the external sequential-sentence package foot_4 by @xcite . it has to be mentioned that we classified the abstract and conclusion sections foot_5 into the same section without having their sentences labeled by the external package. for the joint model, we used loss weighting of 0.5 for two losses associated with each task as it resulted the highest scores in our experiments. for domaintuned summarization, we used 50,000 samples of training set of arxiv dataset, and 2,000 papers samples from the validation set. in all our models, we pick the checkpoint that achieves the best rg-l 6 results in this section, we present the performance of our submissions to the challenge, along with the scores achieved by the other participants. we then show the results of our systems over our internal test set that were constructed on the basis of abstractive set of summaries. note that the reported scores are the average scores over the 5 folds of cross validation sets. to this end, we report the summarization systems' performance in terms of rg-1 (f1), rg-2 (f1), and rg-l (f1) metrics. since the official test set is small, we also conducted analysis between variants of our model using the validation and an additional internal test set. we see in ## conclusion in this paper, we approached the problem of generating long summaries given a scientific dataset of extractive and abstractive summaries. our approaches explored using methods including 1) pretrained transformer encoders for the extractive summarization task using bertsum; and 2) an abstractive summarizer (i.e., bart) which runs over the outputs of extractive summarizer at the first stage, to produce abstractive summaries; and 3) our proposed novel multi-task learner where a section prediciton task is added to the extractive network, trying to jointly learn the sentence importance to be included in the summary, and the section associated with the sentence. while finetuning summarization model on external dataset does not yield promising results on this shared task, our best model is the one that jointly incorporates the section information into the extractive summarizer.