Exploring Neural Models for Query-Focused Summarization (2022)

Abstract: Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization. While recently released datasets, such as QMSum or AQuaMuSe, facilitate research efforts in QFS, the field lacks a comprehensive study of the broad space of applicable modeling methods. In this paper we conduct a systematic exploration of neural approaches to QFS, considering two general classes of methods: two-stage extractive-abstractive solutions and end-to-end models. Within those categories, we investigate existing models and explore strategies for transfer learning. We also present two modeling extensions that achieve state-of-the-art performance on the QMSum dataset, up to a margin of 3.38 ROUGE-1, 3.72 ROUGE2, and 3.28 ROUGE-L when combined with transfer learning strategies. Results from human evaluation suggest that the best models produce more comprehensive and factually consistent summaries compared to a baseline model. Code and checkpoints are made publicly available: https://github.com/salesforce/query-focused-sum .

## introduction text summarization aims at transforming long documents into short snippets that contain only the most important information from the source document. the field has seen substantial progress driven by the availability of large-scale models pre-trained on vast amounts of data @xcite , the development of summarizationspecific pre-training strategies @xcite , and computationally efficient neural architectures @xcite . the majority of recent research efforts in text summarization assume an unconstrained setting * equal contribution in which models are given only a source document as input and are expected to generate a general summary covering the salient aspects from the source. the performance of such models has been evaluated on benchmark datasets spanning various domains: news articles @xcite @xcite , legal documents @xcite , scientific writing @xcite , or creative writing @xcite . however, it has been shown that summarization in an unconstrained setting is an ill-defined task where multiple generated summaries are equally relevant @xcite . this in turn hinders the ability to evaluate and understand the models' content selection capacity. in addition, such generic summarization models lack control mechanisms that would allow end users to customize summaries to their particular needs and expectations. query-focused summarization (qfs) is a subtask within text summarization that focuses on generating summaries where the summary content is tailored to a user-specified query that is passed alongside the source document as input to the model. each source document can be associated with multiple unique queries inquiring about different information from that document. in this setting, end users are enabled to explicitly specify their preferences for the summary, and the relevance of the output summary may be evaluated more precisely with respect to the input query. research on this task has been accelerated by the recently introduced highquality datasets, such as qmsum @xcite and aquamuse @xcite . in this work we conduct a systematic, exploratory study of different approaches to queryfocused text summarization, considering both twostep and end-to-end neural methods. we present two models, relreg and segenc, which achieve state-of-the-art rouge scores on the qmsum dataset, up to a margin of 3.38 r-1, 3.72 r-2, and 3.28 r-l when combined with transfer learning methods. the relreg model uses a two-step approach to solving the problem, where the first step extracts content relevant to the given query and the next step synthesizes the extracted fragments into a coherent summary. the segenc method follows an end-to-end framework in which individual document segments are separately encoded to avoid the computational bottleneck of long input documents, and the decoder jointly attends to all encoded segments when producing the summary. through quantitative studies, we compare our models with other baselines and discuss the trade-offs of the end-to-end methods and pipelined approaches. we also perform human evaluation to understand the qualitative differences between the models. together with this manuscript, we share the code base and model checkpoints to enable future research in this area. ## methodology we present existing methods and propose modeling extensions to address the challenges of queryfocused summarization. ## model exploration in this section, we first analyze the effects of modelspecific architectural and hyperparameter choices on the performance of two-stage ( §4.1) and end-toend models ( §4.2). next, we study the task-specific knowledge transfer capabilities of different pretraining strategies in §4.3. lastly, we conduct a final evaluation and comparison of all discussed models in §4.4. all experiments and analyses presented in this section were conducted on qmsum. ## further analysis in this section we conduct further analysis of the best performing models from section 4. first, we offer additional insights into the performance of those models on the qmsum dataset through a human-based study. next, we discuss the generalization abilities of those models by running experiments on the aquamuse dataset. ## conclusion in this work, we conducted an exploratory study of neural models for query-focused summarization. we studied two categories of models: two-stage and end-to-end, and presented two architectures, relreg and segenc, both of which improve rouge performance over prior state of the art by a substantial margin. we also explored taskspecific transfer learning, which further improved model performance. besides model performance, we discussed issues of computational efficiency that practitioners may factor into their modeling choices. finally, we conducted a human study suggesting that the summaries produced by the bestperforming models are more factually correct and complete than a baseline model by a substantial margin. we hope that the analysis and modeling contributions of this paper will be a resource for future research on query-focused summarization.