Generating Query Focused Summaries from Query-Free Resources (2021)

Abstract: The availability of large-scale datasets has driven the development of neural models that create generic summaries from single or multiple documents. In this work we consider query focused summarization (QFS), a task for which training data in the form of queries, documents, and summaries is not readily available. We propose to decompose QFS into (1) query modeling (i.e., finding supportive evidence within a set of documents for a query) and (2) conditional language modeling (i.e., summary generation). We introduce MaRGE, a Masked ROUGE Regression framework for evidence estimation and ranking which relies on a unified representation for summaries and queries, so that summaries in generic data can be converted into proxy queries for learning a query model. Experiments across QFS benchmarks and query types show that our model achieves state-of-the-art performance despite learning from weak supervision.

## introduction the neural encoder-decoder framework has become increasingly popular in generic summarization @xcite @xcite thanks to the availability of large-scale datasets containing hundreds of thousands of document-summary pairs. training data of this magnitude is not readily available for query focused summarization (qfs; dang 2005) which aims to create a short summary from a set of documents that answers a specific query. existing corpora @xcite @xcite are relatively small for modern data-hungry neural architectures and have been mostly used for evaluation purposes. a major bottleneck in leveraging generic summarization data for qfs is the absence of queries @xcite ; the majority of existing datasets consist of document-summary pairs, while qfs summaries are expected to answer specific queries. recent work @xcite @xcite sidesteps this problem by resorting to distant supervision from query-relevant nlp resources including question answering @xcite and paraphrase identification @xcite . such approaches incorporate query modeling in the summarization process but are even more data hungry compared to generic summarization ones, since they additionally require access to qa datasets which can be extremely costly to create @xcite . moreover, there is often a mismatch between queries in qa datasets and those in qfs scenarios @xcite ; the two types of queries are not identically distributed and it is practically infeasible to find appropriate query-related resources for all domains and topics. in this work we do not assume access to any resources other than those available for generic summarization. we further decompose abstractive qfs into two subtasks: (1) query modeling (i.e., finding supportive evidence within a set of documents for a query) and (2) conditional language modeling (i.e., generating an abstractive summary based on found evidence). under this formulation, we use generic summarization data not only for conditional language modeling, but also for learning an evidence ranking model. inspired by the cloze task and its applications in nlp @xcite @xcite , we propose marge, a masked rouge regression framework for evidence estimation and ranking. marge intro-masked summary -the da vinci code was published in 2003, and within six years brown had booted john grisham f rom t he no. 1 sl ot on t he l ist of writ ers whose books were most often donated t o oxf am's 700 shops. -the independent in 2012 reported brown's best -sel l er was the most-donated book for t he f ourt h year running. what hydroelectric projects are planned or in progress and what problems are associated with them? figure 1 : overview of our abstractive qfs approach. summaries and queries are rendered with unified masked representation (umr) for training and testing, respectively. the summarization framework consists of a query model and a controllable generator. the query model ranks sentences in the input document(s) which provide evidence to answer the query; the generator operates over evidence bearing sentences to generate the final summary. our contributions in this work are threefold: we propose a weakly supervised system for abstractive qfs where no query-related resources are required; we discover a new type of connection between generic summaries and qfs queries, and provide a universal representation for them which allows generic summarization data to be exploited for qfs; we provide experimental results on qfs benchmarks, and show that across query types and domains our system achieves state-of-the-art results on both evidence ranking and abstractive qfs. ## related work the majority of previous qfs approaches have been extractive, operating over queries and document clusters from which they select query-relevant sentences to compose a summary. they mostly differ in the way centrality and relevance are estimated and incorporated, e.g., via manifold ranking @xcite , using a look-ahead strategy @xcite , uncertainty prediction @xcite , or attention mechanisms @xcite . more recently @xcite propose a coarse-to-fine framework that leverages distant supervision from question answering to extract summary-worthy content. abstractive qfs has received significantly less attention. this is due to generation models being particularly data-hungry @xcite and the scarcity of qfs training data. the increasing availability of pre-trained models has prompted the development of pipeline-style frameworks for qfs which use resources from a wider range of nlp tasks. for example, @xcite fine-tune bart @xcite on cnn/dailymail @xcite , a single-document summarization dataset, and generate abstracts for qfs by iteratively summarizing paragraphs to a budget. they learn a query model for paragraph selection based on a plethora of qa and machine reading datasets @xcite . similarly, laskar et al. (2020) fine-tune bertsum on cnn/dailymail, and propose a three-stage system which uses supervision from qfs data (typically reserved for evaluation) and related qa and paraphrase identification tasks. we also focus on abstractive qfs, however, we do not assume access to any additional training resources over and above generic summarization datasets, even for query modeling. moreover, our system is able to generate long qfs abstracts all at once, instead of iteratively creating bullet-style summaries which often lack coherence. ## problem formulation let {(s, d)} denote a generic summarization dataset where @xmath0, d 2 , . . . , d m } is a collection of documents with corresponding summaries s. |d| = 1 for single-document summarization (sds) and |d| > 1 for multi-document summarization (mds). in qfs, a query q additionally specifies an information request, {(s, d, q)}. it is often assumed (e.g., in duc benchmarks) that q consists of a short title (e.g., amnesty international ), and a query narrative which is longer and more detailed (e.g., what is the scope of operations of amnesty international and what are the international reactions to its activities? ). in this work, we propose to decompose qfs into two sub-tasks, namely query modeling and conditional language modeling. the query model q θ (d|q; θ) estimates whether textual units (e.g., sentences) within document cluster d are relevant to query q, while p φ (s|d, q; φ) generates summary s conditioned on evidence provided by the query model and (optionally) the query itself (see figure 1(b ) for an illustration). when s ⊥ ⊥ q, we have a query-agnostic conditional language model p φ (s|d; φ). otherwise, the conditional language model is query-guided. our query model is trained with distant supervision derived from generic summarization data which is easier to obtain (e.g., from online sources) compared to qa datasets which must be annotated from scratch (e.g., for different types of questions and domains). although queries are not verbalized in generic summarization, we hypothesize that the summaries themselves constitute a response to latent queries. so, how can we reverse-engineer the queries from the summaries? inspired by the standard cloze task @xcite and its recent variants @xcite , we render queries and summaries in a unified masked representation (umr) which enables summaries to serve as proxy queries for model training, as shown in figure 1(a) . we further assume that the answer to these queries can be found in sentences which form part of the document collection d. although we do not know for certain what these sentences are we can assume that if they have a high rouge score against the reference summary they are likely to contain an answer. we therefore use rouge as a distant supervision signal, and train a model that takes a query and document sentence as input and estimates their relevance. at inference time, we also render actual queries in umr and rank all sentences in the document collection with our trained model. the most relevant sentences serve as input to a conditional language model to generate query focused abstractive summaries. ## query modeling as explained earlier, we train a query model q θ (d|q; θ) on summary-sentence pairs via distant supervision. we use a summary-based proxy query umr s during training and an actual query umr q during testing. in the following, we first describe how umrs are obtained and then discuss how the query model is trained. algorithm 1 generate masked summary 1: function masksummary(s, γ) summary sentences and mask ratio 2: parse each @xmath1: reveal budget @xmath2: initialize revealed word number @xmath3: initialize masked summary m to s and fill with [mask] 6: initialize @xmath4: while true do 8: @xmath5: for s ← sa do 10: @xmath6, i.e., #tokens 11: if @xmath7: if eom then start post-process 13: the umr for a summary is the concatenation of its sentential umrs. to convert a sentence from natural language to umr, we parse it with open information extraction (open ie; @xcite to a set of propositions consisting of verbs and their arguments. the latter are considered candidate information slots i. we initialize algorithm 1, by replacing all such slots with a [mask] token. we subsequently sample and reveal a set of slots subject to a budget constraint. we define the budget as @xmath8, 1] modulates the proportion of tokens to be revealed within i slots (and is optimized on the development set). finally, in order to keep the representation of umr s and umr q consistent (see next paragraph), we merge adjacent [mask] tokens to one [mask] resulting in a partially masked summary. we mask qfs queries by considering their structure and lexical makeup. queries in duc benchmarks often contain interrogative words (e.g., how is a and what is b ) and request words (e.g., describe a and tell me b ). following this observation, we manually collect a small set of such query words and replace them with @xcite . for queries with a title and a narrative, we first mask the narrative and then prepend "[mask] t .", where t is a sequence of title tokens. figure 1(a) shows examples of a masked query and summary. evidence ranking we represent sentences in a document collection and umr queries with a pretrained bert model @xcite . specifically, we concatenate a umr query and a candidate sentence to sequence "[cls] u [sep] c [sep]" where u is a sequence of tokens within a umr query and c a sequence of tokens in a document sentence (we pad each sequence in a minibatch of l tokens). the [cls] vector serves as input to a single layer neural network which estimates whether the sentence contains sufficient evidence to answer the query (see figure 1 (b) right). we use the mean-square error to compute the loss and update the encoding parameters in bert via standard backpropagation: previous work @xcite ) has used rouge-2 as training signal for paragraph ranking. however, sentences are significantly shorter than paragraphs, and we observe a number of instances with a rouge-2 score of 0. we therefore perform label smoothing and define y as the f1 interpolation of rouge-2 and rouge-1: @xmath9, c) + λ * r 1 (s, c) where λ is optimized on the development set. at inference time, we use the trained model to compute the affinity score between umr q and all candidate sentences in d and rank them accordingly. the highest ranked sentences are deemed query-relevant and passed on to our summary generation model. 2 query narrative expansion in some cases queries may be relatively short and narratives absent. this can be problematic for our setup since query proxies (in the form of summaries) are typically long and detailed. for datasets with short queries we automatically create query narratives in an unsupervised fashion. we employ lexrank @xcite to select a subset of representative sentences under a word budget and concatenate them to form narratives (which we append to the original queries). ## query focused generation we also leverage generic summarization datasets to fine-tune a pretrained language model for abstractive qfs. in experiments we employ the publicly released unilmv2 @xcite to instantiate the controllable generator shown in figure 1(b) , however any other language model could have been used instead. with transformer @xcite as the backbone network, unilmv2 is jointly pretrained for natural language understanding and generation. specifically, a bidirectional model is employs an autoencoding objective (ae; identical to @xcite , while a partially autoregressive (par) sequence-to-sequence model decomposes the probability of masked tokens in input sequence x as: at inference, unilmv2 operates over sentences deemed relevant by the query model and decodes summaries autoregressively (see figure 1(b) left ). ## experimental setup datasets we performed experiments on the duc 2005-2007 qfs benchmarks and td-qfs @xcite . duc benchmarks contain long query narratives while td-qfs focuses on medical texts with short keyword queries. statistics for both datasets are given in ## results our experiments evaluate both components of the proposed approach, namely query modeling and summary generation. we assess the evidence ranker and the effectiveness of the unified masking. we also compare our summaries against competitive abstractive and extractive systems using automatic and human-based evaluation. ## conclusions in this work we proposed an abstractive framework for query focused summarization. we provided a unified mask representation for summaries and queries, which enables summaries to serve as proxy queries for model training. as a result, a query model can be trained with generic summarization data without relying on additional question-answering resources. experimental results across datasets show that the proposed system yields state-of-the-art performance despite the weakly supervised setting, and produces more relevant and coherent summaries compared to existing approaches. in the future, we would like to push this low-resource approach even further and attempt to generate abstractive summaries without access to any summarization datasets. we show in the paper the top k retrieval performance of different models when @xmath10, 30}. in some cases, when top sentences are relatively short, the maximum input length to unilm (which is set to 768) allows for more than 30 sentences to be selected. therefore, in