Summaformers @ L ay S umm 20, L ong S umm 20 (2020)

Abstract: Automatic text summarization has been widely studied as an important task in natural language processing. Traditionally, various feature engineering and machine learning based systems have been proposed for extractive as well as abstractive text summarization. Recently, deep learning based, specifically Transformer-based systems have been immensely popular. Summarization is a cognitively challenging task – extracting summary worthy sentences is laborious, and expressing semantics in brief when doing abstractive summarization is complicated. In this paper, we specifically look at the problem of summarizing scientific research papers from multiple domains. We differentiate between two types of summaries, namely, (a) LaySumm: A very short summary that captures the essence of the research paper in layman terms restricting overtly specific technical jargon and (b) LongSumm: A much longer detailed summary aimed at providing specific insights into various ideas touched upon in the paper. While leveraging latest Transformer-based models, our systems are simple, intuitive and based on how specific paper sections contribute to human summaries of the two types described above. Evaluations against gold standard summaries using ROUGE metrics prove the effectiveness of our approach. On blind test corpora, our system ranks first and third for the LongSumm and LaySumm tasks respectively.

## introduction popularity of data science in recent years has led to a massive growth in the number of published papers online. this has generated an epochal change in the way we retrieve, analyze and consume information from these papers. also wider interest in data science implies even lay persons (readers outside * the author also works as a researcher at microsoft the data science community) are significantly interested in keeping up with the latest developments. the readers have access to a huge amount of such research papers on the web. for a human, understanding large documents and assimilating crucial information out of them is often a laborious and time-consuming task. motivation to make a concise representation of huge text while retaining the core meaning of the original text has led to the development of various automated summarization systems. these systems provide users filtered, high-quality and concise content to work with at an unprecedented scale and speed. summarization methods are mainly classified into two categories: extractive and abstractive. extractive methods aim to select salient phrases, sentences or elements from the text while abstractive techniques focus on generating summaries from scratch without the constraint of reusing phrases from the original text. scientific papers are large, complex documents that tend to be geared towards a particular audience. this is a very small percentage of the population while majority of individuals are unable to fully comprehend the contents of long scientific documents. even among the people who are able to understand the material, the length of such documents often spanning several pages demand a great deal of time and attention. hence, tasks like layman summarization (laysumm) and long-form summarization (longsumm) are of great importance in today's world. typically scientific research papers are fairly structured documents containing standard sections like abstract, introduction, background, related work, experiments, results, discussion, conclusion and acknowledgments. thus, summarization of such documents should be aware of such sectional structure. an intuitive way is to pick a few sentences from each of the sections to be a part of the summary. but how do we decide how many sentences to pick from each section? also, which sentences to pick? can we rewrite sentences so as to obtain a concise abstractive summary? we investigate answers to these questions in this paper. multiple survey papers have provided a detailed overview of the automatic text summarization task @xcite @xcite . most of the practically usable summarization systems are extractive in nature. also, most summarization studies have focused on summarization of news articles. in this work, we mainly focus on two interesting aspects of text summarization: (1) summarization of scientific research papers, and (2) summarization for laymen. @xcite propose that section level processing of scientific documents is useful. further, @xcite conclude that not all sections are equally useful. also, recent papers have observed that a hierarchical summarization of scientific documents is highly effective where at the first level, an extractive summary of each section is independently generated and at the second level, the sectional output is abstracted into a brief summary @xcite . @xcite observe that while summarizing, local context is useful, but global is not. thus, in our approach at the sectional level, we use extracted information from only within the section text to obtain a section's extractive summary, ignoring the remaining text of the entire paper. for the laysumm task, we observe that abstract is the most relevant section of a scientific paper from a layman perspective. we therefore feed the abstract to a transformer-based model and generate an abstractive summary for the laysumm task. for the longsumm task, we first perform extractive summarization for each section and choose a selected number of sentences from each section into the final summary. on blind test corpora of 37 and 22 papers for the laysumm and longsumm tasks, our proposed system leads to a rouge r1 of 45.94 and 49.46 respectively. these results helped us bag the top positions on the leaderboards for the two tasks. ## related work in this section, we discuss related areas including text summarization and style transfer. ## datasets we first describe the datasets which were provided by the organizers of the 'workshop on scholarly document processing @emnlp 2020' foot_0 . ## system overview in this section, we present an overview of the proposed systems for the laysumm and longsumm tasks. ## experimental settings we used hugging face's foot_4 implementation of t5 and bart. we experimented with various length settings for training and generation. we found that minimum and maximum sequence lengths of 120 and 140 respectively for generation gave us the best results. we used adam optimizer with an initial learning rate of 5e-5 with learn rate scheduling based on rouge-1 values calculated on the validation split. in the hyper-parameter tuning phase, a repetition penalty of 1.8 while generating lay summaries provided the most optimal results. we used hpzhao's implementation of sum-marunner foot_5 with default hyper-parameter values. the 'topk' parameter was dynamically adjusted to set the summary length of each section based on the section-specific budget. ## results our system (with the team name: summaformers) ranks first and third for the longsumm foot_6 and lay-summ foot_7 tasks respectively. further details on these tasks can be found in the shared tasks overview paper @xcite . in this section, we present detailed results. ## conclusions in this paper, we studied two scientific document summarization tasks: laysumm and longsumm. we experimented with popular text neural models in a section-aware manner. our results indicate that modeling of the document structure with strong focus on which parts of a research paper to attend to while composing a summary gives a significant boost to the quality of the resultant output. on blind test corpora, our system ranks first and third for the longsumm and laysumm tasks respectively.