Using Pre-Trained Transformer for Better Lay Summarization (2020)

Abstract: In this paper, we tack lay summarization tasks, which aim to automatically produce lay summaries for scientific papers, to participate in the first CL-LaySumm 2020 in SDP workshop at EMNLP 2020. We present our approach of using Pre-training with Extracted Gap-sentences for Abstractive Summarization (PEGASUS; Zhang et al., 2019b) to produce the lay summary and combining those with the extractive summarization model using Bidirectional Encoder Representations from Transformers (BERT; Devlin et al., 2018) and readability metrics that measure the readability of the sentence to further improve the quality of the summary. Our model achieves a remarkable performance on ROUGE metrics, demonstrating the produced summary is more readable while it summarizes the main points of the document.

## introduction recent summarization techniques have greatly benefitted from the advancement of language models and successfully produced plausible summaries for both general news articles in real-life and technical scholarly documents in the expert domain. an informative but concise summary can help people to reduce the search time and boost the decision making by expeditiously providing more relevant documents @xcite @xcite . for processing scholarly documents, automatic summarization is promising since it can benefit researchers to cope with the pace of the exponentially growing number of publications @xcite . despite the recent advancement in automatic summarization literature, summarization for scholarly documents has been less explored compared to the works regarding summarization for ordinary news articles due to the absence of large-scale datasets. developing human-written lay summaries for scholarly documents is challenging since it involves expert knowledge to understand the technical jargon and the complex structure of scientific documents. because of these inherent challenges, existing summarization techniques for scientific documents is limited in a sense, which the produced summary is either too concise to provide important information @xcite or aiming to directly extract the content from abstract or citation sentences @xcite , which mostly resembles the abstract, making it hard for the public and researchers from outside of the particular domain to understand the main points of the scientific papers. although the readability of the abstracts in scientific papers had continuously decreased due to the increase in the use of technical jargon @xcite , the summarization of scientific papers for the public and researchers from outside of the certain field has been remained elusive. to provide a better summary for the public and researchers, we participated in the first computational linguistics lay summary challenge (cl-laysumm 2020) shared task @xcite and developed a summarization system that automatically produces lay summaries for scholarly documents. the main task of cl-laysumm is producing a corresponding lay summary given the full-text and abstract of the research paper. we employed the dataset from the cl-laysumm 2020 committee and performed experiments using recent summarization models including pre-training with extracted gap-sentences for abstractive summarization (pegasus; @xcite , extractive summarization with bidirectional encoder representations from transformers (bert; @xcite , and a new evaluation protocol that measures the readability of the sentence in the summary. we showcase how pegasus, bert, and readability metric improve the summarization system and demonstrate that the produced summary is more readable while it summarizes the main ideas of the documents. ## related work the type of recent benchmark datasets that are widely utilized for evaluating the performance of the summarization system can be categorized into two themes: news articles and scientific documents. summarization of news articles have been more actively explored since it is relatively easy to develop human-written summaries. @xcite and @xcite created a large-scale dataset that contains 200k news articles with manually written gold summaries. owing to the large-scale dataset and relatively simple structure of the articles, neural abstractive summarization using sequence models such as long term short memory (lstm, @xcite with attention mechanism @xcite has been actively used in abstractive summarization for news articles. the attention-based encoder-decoder network has been improved by others. @xcite used lstm with two different networks: pointer-generator network that produces accurate expression by pointing each word in the source and coverage network that avoids repetition. @xcite incorporated reinforcement learning (rl) into sequence models for summarization tasks and @xcite developed multi-agent encoders that communicate with each other by sharing outputs for each layer in the encoder network. after the advent of pre-trained language models such as transformer, bert, and bidirectional and auto-regressive transformers (bart), the summarization literature benefits from these pretrained language models that provide more contextual word representation @xcite @xcite . @xcite used bert model as an encoder, @xcite applied bert to both encoder and decoder networks, @xcite constructed generative adversarial networks using bert models, and @xcite appended semantic similarity layers on top of the pre-trained bart. while neural sequence models have been successfully applied to the summarization for news articles, applying the same techniques to scientific documents would be challenging since the scholarly documents are far longer than ordinary news articles and have a complicated structure. our work is different from the described works as we tackle summarization for scientific documents. although summarization for scientific texts is less explored, @xcite proposed hierarchical encoder-decoder network to address the long scholarly documents for constructing abstract summary, @xcite suggested summarization using abstract and citation sentences with graph convolutional networks (kipf and welling, 2016) and lstm, and released the medium-scale dataset that contains 1000 scientific papers in the computational linguistic domain with humanwritten summaries and citation sentences for each paper. @xcite implemented an extreme summarization system, which is tldr (too long; don't read) summarization, for scientific documents using multi-task learning with headline generation models @xcite . @xcite proposed pegasus by masking important sentences in the input document with a transformer-based encoder-decoder network to force the model to summarize main points of the contents given the remainder of the text. pega-sus tackled summarization for both news articles and scholarly documents but it only aimed to produce the abstract. in contrast, our work is distinct from the previous approaches as we aim to produce lay summaries for scientific documents rather than generating extremely short sentences or summaries that contain technical words which makes it difficult for lay audiences to understand. to facilitate scholarly document processing, there have been annual workshops regarding data mining, natural language processing (nlp), information retrieval for scientific publications: birndl (bibliometric-enhanced information retrieval and natural language processing for digital libraries), wosp (workshop on mining scientific publications), tac (text analytics conference). in particular, the annual cl-scisumm @xcite @xcite encouraged participants to research on scientific documents summarization. our work is closely related to the cl-laysumm 2020, which is the first lay summary challenge shared task. we employed the laysumm dataset provided by the workshop organizing committee and performed experiments using a variety of recent summarization models to develop the lay summarization system. ## method there are two main summarization models used in our system to generate the lay summary. we tried to use pegasus which is an abstractive summarization model and presumm @xcite for extractive summarization to produce the summary. we trained the summarization model on the lay summary dataset in a supervised way by pairing the abstract and the corresponding lay summary of the paper. after producing the lay summary using pegasus, we improved the quality of the produced summary by appending important sentences to the summary of which the number of words is under a certain threshold. for example, if the number of the lay summary generated by the abstractive model was under 90, we added the sentences from the corresponding summry geneated by extractive model up to this threshold. when appending the sentences to the produced summary, we prioritized the sentences in the abstract based on the score predicted by presumm model and readability metric and applied tri-gram blocking to avoid repetition @xcite . detailed descriptions of each summarization model and the readability metric are presented in the following sections.