AUTH @ CLS ci S umm 20, L ay S umm 20, L ong S umm 20 (2020)

Abstract: We present the systems we submitted for the shared tasks of the Workshop on Scholarly Document Processing at EMNLP 2020. Our approaches to the tasks are focused on exploiting large Transformer models pre-trained on huge corpora and adapting them to the different shared tasks. For tasks 1A and 1B of CL-SciSumm we are using different variants of the BERT model to tackle the tasks of “cited text span” and “facet” identification. For the summarization tasks 2 of CL-SciSumm, LaySumm and LongSumm we make use of different variants of the PEGASUS model, with and without fine-tuning, adapted to the nuances of each one of those particular tasks.

## introduction for scholars in every scientific domain, the ever growing amount of articles published each year has made the long-lasting challenge of keeping up with the recent literature significantly harder. in addition, there is an increasing need for making research accessible and relevant to the general public and not just a small group of researchers and practitioners. for example, taxpayers want to know where federal money supporting research goes. as a result, there is a need for different types of summaries that can either facilitate scientific research by compressing the key ideas discussed in a scientific paper or make scientific research more relevant for a lay audience. it is obvious that tasking the author of a paper with writing multiple summaries of her work for different audiences is time-consuming. consequently, the interest for methods that automatically summarize scientific documents in different styles and variations has increased significantly. towards this direction, the 1 st scholarly document processing shared task (sdp 2020) @xcite introduces a number of different tasks that address these challenges. in addition to the original cl-scisumm sub-tasks of previous years, the 2020 version includes tasks that are targeting the summarization of complete papers as well as the generation of lay summaries. the different tasks of sdp 2020 can be summarized as follows. exploiting large language models that are pretrained on huge corpora of unlabelled data and then adapting them to solve nlp problems has proven to be a very successful strategy. this type of approach has yielded state-of-the-art results in a variety of nlp tasks such as question answering, machine translation and summarization and has proven to be especially beneficial when the training data are limited. in this work our main focus is to explore large pre-trained transformers like bert @xcite and pegasus @xcite and how they can be effectively used in the context of the sdp 2020 shared task. for cl-scisumm task 1a we fine-tune a bert pairwise classifier that is able to identify "cited text spans" of the rp given a number of "citing spans" from multiple cps. we further improve the efficiency of our system by adding a pre-filtering stage based on tf-idf that selects good "candidates" for the bert model. in task 1b we train a simple logistic regression classifier that uses embeddings from another pre-trained model, scibert @xcite , and is able to classify each cited text span into one of five distinct facets. we show that even an algorithm as simple as logistic regression can effectively learn a fairly complex task with very few training data when using features from a powerful pre-trained model such as scibert. we approach task 2 of cl-scisumm as well as laysumm and longsumm as abstractive summarization tasks. more specifically, we employ the pe-gasus pre-trained model that has demonstrated very good results in various abstractive summarization benchmarks. we use the pre-trained model without any additional training to generate a comprehensive summary of an article given the abstract as well as the information from the parts of the full text that are cited by other articles for task 2 of cl-scisumm. for the laysumm task, we fine-tune the pegasus model to compress and re-write the abstract of the given article in order to generate a summary suited for a lay audience. finally, for the longsumm task our approach makes use of the divide-and-conquer (dancer) @xcite summarization method in combination with the pegasus model aiming to generate an accurate and detailed summary of the article by separately summarizing important sections of the full text. the rest of this work is structured as follows. in section 2 we very briefly present different summarization approaches focusing on academic articles with emphasis on pre-trained transformer models. in sections 3 to 5 we describe our approaches to each one of the three tasks and in section 6 we discuss our experiments and results. ## related work the task of summarizing scientific articles has received increased attention lately. existing methods usually approach the problem in one of two fundamental ways. extractive methods @xcite @xcite focus mainly on identifying key sentences of the text and creating a summary by combining the extracted sentences. on the other hand, abstractive methods @xcite @xcite are using language models conditioned on the input text in order to generate a summary. both extractive and abstractive methods when applied to scientific articles are typically taking as input the abstract and/or the full text of the article and try to generate an abstract-like summary. in contrast, dancer @xcite learns to summarize different sections of the full text separately and combines the individual summaries into a single article summary. given the increased popularity and success of large pre-trained transformer models in various nlp tasks, multiple approaches have decided to use similar models for summarization. such approaches have either used pre-trained transformers as encoders combined with a classification decoder that selects sentences in an extractive manner @xcite or have employed full encoder-decoder models that are pre-trained on various tasks and fine-tuned for abstractive summarization @xcite @xcite . one notable such model is the pre-training with extracted gap-sentences for abstractive summarization sequence-to-sequence (pega-sus) @xcite model. pegasus is a transformer encoder-decoder model pre-trained on massive corpora of documents (web and news articles) that has demonstrated great potential on various summarization benchmarks. the pre-training of pegasus is based on optimizing the gap sentence generation (gsg) objective where whole sentences of the input are masked and the model attempts to generate these gap-sentences from the rest of the input. a number of summarization approaches we proposed for the cl-scisumm task b in the previous years of the challenge, including extractive methods based on probabilistic models @xcite and large pre-trained bert models @xcite . ## conclusion we have presented the systems we developed for the sdp 2020 shared task. for task 1a we implemented an efficient pairwise classification approach based on the bert model that tackles the "cited text identification" problem. for task 1b we show how a simple logistic regression classifier using pre-trained scibert embeddings as features can effectively learn to solve the problem of facet classification. for the summarization tasks we employ different variants of the pegasus model and adapt them to the nuances of each particular task. for task 2 we use of the pre-trained pegasus model in a zero-shot setting to generate a summary given the abstract of an article along with the cited text spans. for laysumm we propose a re-writing strategy based on the pegasus model that works on the abstract and generates a lay summary. finally, we showcase how the pegasus model can be used to summarize an academic paper in a distributed way and we demonstrate an end-to-end system that generates a "long" summary by selecting key sections, summarizing each section independently and combining them to form the final summary.