Salient Information Prompting to Steer Content in Prompt-based Abstractive Summarization (2024)

Abstract: Large language models (LLMs) can generate fluent summaries across domains using prompting techniques, reducing the effort required for summarization applications. However, crafting effective prompts that guide LLMs to generate summaries with the appropriate level of detail and writing style remains a challenge. In this paper, we explore the use of salient information extracted from the source document to enhance summarization prompts. We show that adding keyphrases in prompts can improve ROUGE F1 and recall, making the generated summaries more similar to the reference and more complete. The number of keyphrases can control the precision-recall trade-off. Furthermore, our analysis reveals that incorporating phrase-level salient information is superior to word- or sentence-level. However, the impact on summary faithfulness is not universally positive across LLMs. To enable this approach, we introduce Keyphrase Signal Extractor (SigExt), a lightweight model that can be finetuned to extract salient keyphrases. By using SigExt, we achieve consistent ROUGE improvements across datasets and LLMs without any LLM customization. Our findings provide insights into leveraging salient information in building prompt-based summarization systems.

## introduction abstractive summarization aims to generate concise summaries that capture the most salient information from lengthy source documents. prior work has shown that emphasizing keywords from source documents can enhance summarization performance on supervised finetuned (sft) models @xcite . however, existing approaches @xcite @xcite require extensive modifications to the architecture † work done during an internship at aws ai labs. and loss functions, hindering widespread adoption, especially for large language models (llms) with billions of parameters. recent work @xcite trains a separate network using reinforcement learning (rl) to generate keyphrases for llm prompts, but training rl model is non-trivial due to convergence and stability issues @xcite . emphasizing salient information in the prompt can help zero-shot llms generate more complete summaries, and steer llms to generate summaries that align with the desired use case. however, there is also a lack of analysis on how emphasizing salient information in prompts would affect the llm behavior. we first address the challenge of applying salient information to llms. we obtain keyphrases using a stand-alone keyphrase signal extractor called sigext, and prompt the llms to consider these keyphrases when generating summaries. unlike prior work relying on complex keyphrase generators optimized for specific llms, sigext is llmagnostic, allowing leveraging salient information with large api-based models that cannot be finetuned. we demonstrate consistent improvement in rouge scores on 4 representative summarization datasets and 3 recent llms -claude, mistral @xcite , and falcon @xcite -highlighting the wide adaptability of our approach. secondly, we conduct comprehensive experiments using sigext to gain insights into how keyphrases in prompts affect different aspects of summary quality. we show that adding keyphrases improves rouge f1 and recall, making the generated summaries more similar to the reference and more complete. adjusting the number of keyphrases influences the trade-off between precision and recall. including additional keyphrases in the prompt tends to produce more detailed summaries, enhancing recall. our findings indicate that using phrase-level salient information is more effective than word-or sentence-level approaches. ## method in this section, we introduce sigext -a keyphrase extractor designed for boosting summarization quality of prompt-based llms. figure 1 gives an overview. sigext tokenizes the source document into phrases (phrase tokenization is detailed in section 2.1), and simultaneously predict whether each phrase is important. to train the model, we create target labels by identifying phrases appear in both the source document and the summary, then optimizing the cross entropy loss. compared to previous a keyphrase generator that uses rl @xcite , sigext allows easier control of keyphrase numbers, faster training and inference, and better consistency across domains. we directly incorporate keyphrases in prompt, making it generalizable across llms. to handle longer input lengths while maintaining efficiency, we build sigext using longformer, so that training and inference can be done on a single gpu. ## related work leveraging keyword in abstractive summarization has been explored in many works. switching generator-pointer @xcite and copy-net @xcite modify a recurrent neural network model @xcite to directly copy keywords from the source text. more recent work has adopted transformer architectures @xcite , which have become dominant in natural language processing. @xcite introduces a bias in the attention matrix to help transformer models focus on keywords. all these models need to be trained or finetuned on large-scale training data. while finetuned models typically achieve higher rouge scores than prompting a pretrained model, prompt-based summarizers are preferred in some industrial use cases due to their flexibility and reduced need for data collection. incorporating keyphrases in the prompt can effectively control the length and content coverage of the summary, something that fine-tuning methods cannot easily achieve. therefore, we cannot compare with these methods using metrics like rouge. instruction finetuned llms @xcite @xcite have shown strong performance on summarization purely via prompts, without finetuning data. such models are often offered via apis, enabling easier development and deployment of summarization applications. keyphrases are still helpful for these large models, as @xcite show that a keyphrase generater trained with reinforcement learning can improve summarization performance. there has been interest in 2-stage extractivethen-abstractive approaches @xcite @xcite @xcite . these first extract keyphrases or sentences before abstractively summarizing them. these methods are trained end-to-end for domainspecific use cases, while our method can be pretrained for general purpose zero-shot use cases. practically, any keyword extractor, for example keybert or llmbert @xcite , can be used for the first stage to enhance the summarization in the second stage. the 2-stage methods could also be implemented as chain-of-thought (cot) by generating intermediate hints and final results in the same prompt, such as @xcite . in our experiments, we compare our method with a 2-stage prompting approach -first generating keywords using one prompt, then using those keywords for summarization in the second prompt. while slightly different from previous work, the 2-stage baseline effectively captures the use of intermediate reasoning steps of llms. ## conclusion in this paper, we propose a lightweight approach to incorporate keyphrases into the prompt for llmbased abstractive summarization. sigext involves training a phrase extractor using supervised learning to identify salient keyphrases from the input text. these keyphrases are then injected into the prompt provided to the llm for summary generation. we demonstrate that this approach can effec-tively improve the rouge scores of the generated summaries, indicating a higher similarity to reference summaries. introducing keyphrases in the prompt enhances the faithfulness of the summary by ensuring that important information is captured. additionally, our approach offers control over the length and precision/recall trade-off of the summary. notably, our pretrained keyphrase extractor -gp-sigext-can improve summarization performance out-of-the-box without any finetuning, even in cases where training data is not available.