TSTR : Too Short to Represent, Summarize with Details! Intro-Guided Extended Summary Generation (2022)

Abstract: Many scientific papers such as those in arXiv and PubMed data collections have abstracts with varying lengths of 50-1000 words and average length of approximately 200 words, where longer abstracts typically convey more information about the source paper. Up to recently, scientific summarization research has typically focused on generating short, abstract-like summaries following the existing datasets used for scientific summarization. In domains where the source text is relatively long-form, such as in scientific documents, such summary is not able to go beyond the general and coarse overview and provide salient information from the source document. The recent interest to tackle this problem motivated curation of scientific datasets, arXiv-Long and PubMed-Long, containing human-written summaries of 400-600 words, hence, providing a venue for research in generating long/extended summaries. Extended summaries facilitate a faster read while providing details beyond coarse information. In this paper, we propose TSTR, an extractive summarizer that utilizes the introductory information of documents as pointers to their salient information. The evaluations on two existing large-scale extended summarization datasets indicate statistically significant improvement in terms of Rouge and average Rouge (F1) scores (except in one case) as compared to strong baselines and state-of-the-art. Comprehensive human evaluations favor our generated extended summaries in terms of cohesion and completeness.

## introduction over the past few years, summarization task has witnessed a huge deal of progress in extractive @xcite @xcite @xcite and abstractive @xcite @xcite @xcite [introductory] neural machine translation ( @xcite ), directly applying a single neural network to transform the source sentence into the target sentence, has now reached impressive performance ( @xcite motivated by recent success in unsupervised cross-lingual embeddings ( @xcite ), the models proposed for unsupervised nmt often assume that a pair of sentences from two different languages can be mapped to a same latent representation in a shared-latent space ( @xcite ) @xcite although the shared encoder is vital for mapping sentences from different languages into the shared-latent space, it is weak in keeping the uniqueness and internal characteristics of each language, such as the style, terminology and sentence structure. @xcite for each language, the encoder and its corresponding decoder perform an ae, where the encoder generates the latent representations from the perturbed input sentences and the decoder reconstructs the sentences from the latent representations. experimental results show that the proposed approach consistently achieves great success. ## related work summarizing scientific documents has gained a huge deal of attention from researchers, although it has been studied for decades. neural efforts in scientific text have used specific characteristics of papers such as discourse structure @xcite and citation information @xcite @xcite to aid summarization model. while prior work has mostly covered the generation of shorter-form summaries (approx. 200 terms), generating extended summaries of roughly 600 terms for long-form source documents such as scientific papers has been motivated very recently @xcite . the proposed models for the extended summary generation task include jointly learning to predict sentence importance and sentence section to extract top sentences @xcite ; utilizing section-contribution computations to pick sentences from important section for forming the final summary (ghosh roy et al., 2020); identifying salient sections for generating abstractive summaries @xcite ; ensembling of extraction and abstraction models to form final summary @xcite ; an extractive model with textrank algorithm equipped with bm25 as similarity function @xcite ; and incorporating sentences embeddings into graph-based extractive summarizer in an unsupervised manner @xcite . unlike these works, we do not exploit any sectional nor citation information in this work. to the best of our knowledge, we are the first at proposing the novel method of utilizing introductory information of the scientific paper to guide the model to learn to generate summary from the salient and related information. ## background: contextualized language models for summarization contextualized language models such as bert @xcite , and roberta @xcite have achieved state-of-the-art performance on a variety of downstream nlp tasks including text summarization. @xcite were the first to fine-tune a contextualized language model (i.e., bert) for the summarization task. they proposed bertsum -a fine-tuning scheme for text summarization-that outputs the sentence representations of the source document (we use the term source and source document interchangeably, referring to the entire document). the bert-sumext model, which is built based on bert-sum, was proposed for the extractive summarization task. it utilizes the representations produced by bertsum, passes them through transformers encoder @xcite , and finally uses a linear layer with sigmoid function to compute copying probabilities for each input sentence. formally, let l 1 , l 2 , ..., l n be the binary tags over the source sentences @xmath0, sent 2 , ..., sent n } of a long document, in which n is the number of sentences in the paper. the bertsumext network runs over the source documents as follows (eq. 1), ## tstr: intro-guided summarization in this section, we describe our methodology to tackle the extended summary generation task. our approach exploits the introductory information 3 . the detailed illustration of our model is shown in figure 3 . to aid the extractive summarization model (i.e., right-hand box in figure 3 ) which takes in source sentences of a scientific paper, we utilize an additional bertsum encoder called introductory encoder (left-hand box in fig. 3 ) that receives x @xmath1, sent 2 , ..., sent m }, with m being the number of sentences in introductory section. the aim of adding second encoder in this framework is to identify the clues in the introductory section which point to the salient supplementary sentences foot_2 . the bertsum network computes the extraction probabilities for introductory sentences as follow (same way as in eq. 1) , in which hb , and h are the introductory sentence representations by bertsum, transformers encoder, respectively. p is the introductory sentence extraction probabilities. w j and b j are trainable matrices. after identifying salient introductory sentences, the representations associated with them are retrieved using a pooling function and further used to guide the first task (i.e., right-hand side in figure 3 ) as follows, bertsum transformers feed-forward introductory sentences task-2 (t ) : introductory sentence extraction perceptron (mlp ) 1 source sentences { sent [ cls ] sent [ cls ] ... sent } bertsum transformers perceptron (mlp ) 2 1 2 n { sent [ cls ] sent [ cls ] ... sent } 1 2 m feed-forward sigmoid task-1 (t ) : source sentence extraction output probablities introductory sentences output probablities source sentences       sigmoid select 2 1 figure 3: detailed illustration of our summarization framework. task-1 (t 1 ): source sentence extraction (right-hand gray box). task-2 (t 2 ): introductory sentence extraction (left-hand gray box). as shown, the identified salient introductory sentences at training stages are incorporated into the representations of source sentences by the select(•) function (orange box) with @xmath2. plus sign shows the concatenation layer. the feed-forward neural network is made of one linear layer. at the final stage, we concatenate the transformed introductory top sentence representations (i.e., ĥ) with each source sentence representations from eq. 1 (i.e., h i where i shows the ith paper sentence) and process them to produce a resulting vector r which is intro-aware source sentence hidden representations. after processing the resulting vector through a linear output layer (with w z and b z as trainable parameters), we obtain final introaware sentence extraction probabilities (i.e., p) as follows, in which mlp 2 is a multi-layer perceptron, influencing the knowledge from introductory sentence extraction task (i.e., t 2 ) into the source sentence extraction task (i.e., t 1 ). we train both tasks through our end-to-end system jointly as follows, ## experimental setup in this section, we explain the datasets, baselines, and preprocessing and training parameters. ## error analysis to determine the limitations of our model, we further analyze our system's generated summaries and report three common defects, along with the percentage of these errors among underperformed cases. we found that (1) our end-to-end system's performance is highly dependent on the introductory sentence extraction task's performance (i.e., task t 2 in figure 3 ) as identification of salient introductory sentences (i.e., oracle introductory sentences) sets up a firm ground to explore detailed sentences from the non-introductory parts of the paper. in other words, identification of non-salient introductory sentences leads to a drift in finding supplemental sentences from the non-introductory parts. our model often underperforms when it cannot find important sentences from the introductory part (65%); (2) in underperformed cases, our model fails in selecting motivation, objective sentences from the introductory part, and only identifies the contribution sentences (i.e., describing paper's contributions), such that the final generated summary is composed of contribution sentences, rather than objective sentences. this observation hurts the system in cohesion and completeness (40%); and (3) as discussed, our model matches introductory sentences with sentences from non-introductory parts of the paper. given that two sentences within a scientific paper might conceptually convey the exact same information, but are just paraphrased of each other, our model samples both to form the final summary as a high semantic correlation exists between them. this phenomenon leads to sampling two sentences that convey the same information without providing more details; hence, information redundancy (35%). ## conclusion in this work, we propose a novel approach to tackle the extended summary generation for scientific documents. our model is built upon the fine-tuned contextualized language models for text summarization. our method improves over strong and state-of-the-art summarization baselines by adding an auxiliary learning component for identifying salient introductory information of long documents, which are then used as pointers to guide the summarizer to pick summary-worthy sentences. the extensive intrinsic and human evaluations show the efficacy of our model in comparison with the stateof-the-art baselines, using two large scale extended summarization datasets . our error analysis further paves the path for future reseacrh.