Guiding Large Language Models via External Attention Prompting for Scientific Extreme Summarization (2024)

Abstract: Scientific extreme summarization, the task of generating concise one-sentence summaries (TLDRs) for scientific papers, presents significant challenges due to the need for deep domain-specific understanding and the ability to distill salient information. This study identifies the critical role of titles and keywords in enhancing TLDR generation through quantitative analysis. We propose a novel method, External Attention Prompting (EAP), which leverages LLMs by guiding them to focus on the most critical parts of the source text through varying degrees of attention signals. Our method employs Markdown emphasis syntax to annotate attention levels, enabling LLMs to prioritize salient information effectively. Extensive experiments demonstrate that EAP significantly outperforms baseline methods across various LLMs and metrics in both zero-shot and few-shot settings. Further evaluations by GPT-4 demonstrate that EAP can enable LLMs to generate TLDRs of higher human-aligned quality.

## introduction the rapid growth of scientific literature has made it increasingly difficult for researchers to keep up with the research frontiers and quickly identify the most relevant and impactful information. in this context, recent work @xcite @xcite @xcite @xcite has studied the problem of scientific extreme summarization, which involves generating a concise one-sentence summary (tldr: too long; didn't read) that captures the key aspects of a scientific paper. scientific extreme summarization is a challenging task that requires a deep understanding of domain-specific content to accurately identify and distill the most salient points @xcite . this task demands advanced methods capable of * corresponding author leveraging domain knowledge and prioritizing the most important content. previous studies have found that key content such as the titles and keywords of source documents are helpful to improve performance in summarization task, as they often contain key information @xcite @xcite @xcite . our analysis on the scitldr dataset @xcite further indicates that their contributions to generating highly compressed summaries differ as they contain diverse density of key information (see section 2), suggesting that they should be given unequal levels of importance. in this work, we leverage large language models (llms) to tackle the scientific extreme summarization task, as they have demonstrated emergent abilities such as in-context learning and made remarkable advances in solving various nlp problems @xcite . we propose a novel method called external attention prompting (eap) that enhances the performance of llms by guiding them to pay varying degrees of attention to different segments of the source text. our method involves constructing prompts that assign different levels of attention signals to the n-grams in the source text based on their importance, using markdown emphasis syntax to annotate the attention strength. this approach enables the llms to prioritize the salient information and generate more accurate and informative tldrs. experimental results demonstrate that our proposed prompting method significantly enhances the performance of llms in this task, outperforming baseline methods. the main contributions of this work are as follows: ## source contribution analysis generating high-quality tldrs requires capturing precisely the salient aspects of the document. in this section, we conduct a preliminary experiment to investigate whether components such as the title, abstract and keywords contribute differently to the formation of a tldr. by identifying the most influential parts, we aim to enhance the summarization process by directing the model's attention to the most critical information. to do this, we firstly introduce the concept of contribution density, which measures the averaged contribution of tokens from a specific source input components (e.g., title, abstract, and keywords) to the gold tldr. we first tokenize the input components and remove stopwords. then, the contribution density cd(c) for an input component c is calculated as: the calculation of contribution density draws inspiration from methods used in previous works @xcite @xcite to assess abstractiveness and extractiveness of generated summaries. however, in this context, it serves as an analytical tool rather than a direct evaluation metric for summary quality assessment. the term "contribution" represents how frequently the words from source input appear in the gold tldr. we use contribution density as a proxy to quantify the relative importance of different source input components in contributing to extreme summarization. we take the training set of scitldr @xcite for analysis, and calculate the contribution density for title and abstract, which are 49.2 and 11.9, respectively. the results align with our intuition that the title often captures the most core idea of a paper. while titles may contain less overall information than abstracts, the information they do contain is more densely packed with content relevant to the tldr. in other words, any given n-gram from the title is more likely to contribute to the tldr than a random n-gram from the abstract. abstracts may contain more information that is either less important or not sufficiently crucial to be included in the tldr. additionally, we calculate the contribution densities for the top-k keywords from abstract using textrank algorithm @xcite , the results are shown in figure 1 . the contribution density decreases as k increases, indicating diminishing returns with more keywords. this highlights the importance of selecting a concise set of highly relevant keywords. the above findings from our source contribution analysis underscore the necessity of focusing on the most important parts of the input text when generating a tldr. ## external attention prompting based on the insight from section 2, we propose a novel method called external attention prompting (eap). the core idea of our method is to guide llms to focus on the most critical parts of the source text when generating tldrs. as illustrated in figure 2 , our method leverages external signals derived from the paper's title and keywords to modulate the attention of the llm. by emphasizing different segments of the source text with varying degrees of importance, we aim to enhance llms' ability to distill the essence of the scientific paper into a concise and informative tldr. ## conclusion in this work, we have presented external attention prompting (eap), a novel method for improving the performance of large language models (llms) in the task of scientific extreme summarization. by