What Are You Token About? Differentiable Perturbed Top- k Token Selection for Scientific Document Summarization (2024)

Abstract: Scientific document summarization aims to condense complex and long articles in both technical and plain-language terms to facilitate the accessibility and dissemination of scientific findings. Existing datasets suffer from a deficiency in source heterogeneity, as their data predominantly stem from a single common resource, hindering effective model training and generalizability. First, we introduce SciLay, a novel dataset that includes documents from multiple natural science journals with expert-authored technical and lay summaries. Second, we propose PrunePert, a new transformer-based model that incorporates a differentiable perturbed top- k encoder layer to prune irrelevant tokens in end-to-end learning. Experimental results show that our model achieves a nearly 2x speed-up compared to a state-of-the-art linear transformer, remaining comparable in effectiveness. Additional examinations underscore the importance of employing a training dataset that includes different sources to enhance the generalizability of the models. Code is available at https://github.com/disi-unibo-nlp/sci-lay.

## introduction abstractive summarization aims to meticulously condense documents by discerning and rephrasing their salient points. although this task is relatively easy when summarizing concise texts-such as news facts @xcite -synthesizing scientific articles presents formidable challenges for humans @xcite . in fact, experts must conduct thorough reviews, encountering technical terms and formulas @xcite , to fully grasp the foundational information contained within the text. to mitigate the considerable time and effort required for this endeavor, scientific document summarization (sds) emerges as an indispensable tool. ## scilay scilay is a new dataset created to summarize scientific papers for both technical and lay audiences. collection unlike existing datasets in the literature (see readability we compare the readability of technical with their lay counterparts using the following standard metrics: flesch-kincaid grade level @xcite , coleman-liau index doc tech summary lay summary dataset samples # words # words # sents # words # sents pubmed (cohan et al., 2018) 133,215 2640.8 177.3 6.7 --arxiv (cohan et al., 2018) 215,913 5282.3 237.8 8.9 --laysumm (chandrasekaran et al., 2020) 572 4426.1 --82.2 3.8 plos (goldsack et al., 2022) 27,525 5366.7 --175.6 7.8 elife (goldsack et al., 2022) 4828 7806.1 --347.6 15.7 scitechnews (cardenas et al., 2023) 2431 7570.3 --216.8 7.9 scilay (ours) 43,790 7530.4 239.1 8.8 145.7 5.7 characterization to compare the alignment of the technical and lay summaries with the input article, we calculate the extractive fragment coverage and density @xcite . technically, an extractive fragment is defined as the set of words shared between two texts. coverage gauges the proportion of words in the summary that constitute an extractive fragment. density employs the square of common fragment lengths, ensuring that summaries with longer common fragments receive higher values than those with more numerous but shorter common fragments. furthermore, we compute the percentage of novel n-grams to assess the amount of information present in the summary that is not explicitly stated in the source document. ## prunepert in laying the groundwork for understanding our solution (section 3.2), we provide a conceptual preliminary with the needed foundations. ## conclusion we first introduced scilay, a new sds dataset designed to benchmark models in the creation of technical and lay summaries from heterogeneous sources. second, we present prunepert, a new plm enriched with a token-pruning layer within the encoder stack that allows the model to select only a summary-worthy subset of input tokens for synthesis generation. quantitative and qualitative analysis attest to the comparable performance between our method and a cutting-edge linear transformer; yet, prunepert is notably more efficient, with an average speed-up of almost 2x. further investigation reveals the nature of the selected tokens and the importance of having training sets covering multiple sources to enhance model generalizability.