QA -prompting: Improving Summarization with Large Language Models using Question-Answering (2025)

Abstract: Language Models (LMs) have revolutionized natural language processing, enabling high-quality text generation through prompting and in-context learning. However, models often struggle with long-context summarization due to positional biases, leading to suboptimal extraction of critical information. There are techniques to improve this with fine-tuning, pipelining, or using complex techniques, which have their own challenges. To solve these challenges, we propose QA-prompting - a simple prompting method for summarization that utilizes question-answering as an intermediate step prior to summary generation. Our method extracts key information and enriches the context of text to mitigate positional biases and improve summarization in a single LM call per task without requiring fine-tuning or pipelining. Experiments on multiple datasets belonging to different domains using ten state-of-the-art pre-trained models demonstrate that QA-prompting outperforms baseline and other state-of-the-art methods, achieving up to 29% improvement in ROUGE scores. This provides an effective and scalable solution for summarization and highlights the importance of domain-specific question selection for optimal performance.

## introduction language models (lms) have revolutionized the application of natural language processing. with instruction tuning @xcite , prompting @xcite , and in-context learning @xcite , llms perform well in most of the conditional generation tasks out-of-the-box. specifically, in abstractive summarization, this approach yields highly fluent, consistent, and relevant summaries (tanya goyal, 2022) that are even preferred over summaries given the following article, first answer the questions. then, using the article and answers as key pointers, generate a summary of the article. {article} q1: {question_1} q2: {question_2} a: language model a: a1: ... a2: .... summary: ..... top_k figure 1 : framework for qa-prompting: relevant questions are extracted from a corpus based on the domain of article, and a user defined top-k value. a prompt is constructed to first answer the questions, followed by generating summary. despite the metrics rating the summaries low, several works argue that lm summaries are of high quality @xcite , fluent, grammatically correct, and largely relevant. but where facts are important, key information is sometimes not present. this problem proves to be more pronounced in the summarization of long context tasks. one of the factors contributing to this is the positional bias in lms, where more recent tokens play a more critical role in the generation of the next token @xcite . it is also established that summaries are more faithful to tokens at the start and the end @xcite , ignoring the text in the middle. this phenomenon may be more pronounced in small to medium size lms (ai@meta, 2024; @xcite because they don't have the scale to extract deep contextual information ef-fectively, leading to the generation of sub-optimal summaries. one way to handle this is by adapting lms to the task via weight updates (fine-tuning), but this poses massive compute and data requirements, and does not generalize to all use cases. one can also do complex pipelining, or iterative refinement @xcite , but these bring additional complications and inefficiencies. however, generating fluent, factually correct, contextually rich summaries efficiently is important to support generalization and scalability. this poses the following key challenges: (c1) how to mitigate the positional bias of lm to generate a good summary? (c2) how to make this approach generalize so that it can work with the summarization of all types of text? (c3) how to achieve this with resource efficiency? to solve these challenges, we propose qa-prompting -an approach to summarization by using question-answering and in-context learning with a single lm call. first, we show that vanilla prompting or in-context learning (icl) generates sub-optimal summaries of articles. then, using a manually crafted, potentially relevant set of questions, we identify top k questions that can aid summarization. question answering is different from summarization, as it tries to focus on a specific part of the article rather than comprehending its global context at once. we use this property to extract relevant content from the article. then, using the article and these questions, we prompt the lm to first generate answers to the questions, followed by generating a summary. by trying to answer the relevant questions first, the lm extracts useful information and keeps it in recent context, which further helps it to generate good summaries (addressing c1). it also filters out noise from the long context article that is not relevant for the summary. using small lms in the 0.5b-12b range, we show that this approach significantly improves the quality of summaries (addressing c2). qaprompting uses a single lm call for a task and works with a pre-trained model without modifications (addressing c3). we also conduct a detailed ablation study to validate our design choices. our questions are domain-specific, i.e., they are different for tasks belonging to different domains (e.g., news and research). identifying top k questions is an overhead, but needs to be done only once for a domain. we show that this domain-level adoption of qa-prompting is necessary and performs better than keeping a standard set for all tasks. the key contributions of this work is: ## method overview in this section, we will describe qa-prompting in detail. the first step is to sample candidate questions that will aid the generation of effective summaries, followed by using these questions to construct a prompt that will summarize the text. ## experimental setup this section is to describe the experimental setting to validate our proposed method. all the artifacts used are cited as per their licensing agreements for academic research. ## results this section discusses the results, followed by an ablation study, ending with aspects related to domain-specificity and question selection. ## conclusion in this work, we present qa-prompting, an effective approach to improve summarization with large language models by incorporating questionanswering as an intermediate step. our method addresses positional bias in language models by extracting key information through domain-specific questions, ensuring critical details remain in recent context before summary generation. experiments across multiple models and diverse domains demonstrate that qa-prompting consistently outperforms other methods by up to 29%. qa-prompting's success lies in domain-specific question selection. tailored question sets significantly enhance performance, especially for larger models that can better process and utilize extracted information. our efficient method uses pre-trained models without instruction-tuning or fine-tuning, operating in a single lm call and making it scalable for real-world applications. future research directions include automating question selection and exploring dynamic top-k optimization for different tasks. by bridging the gap between question-answering and summarization, our work opens new possibilities for leveraging intermediate reasoning steps to improve lm performance across diverse applications.