Coarse-to-Fine Query Focused Multi-Document Summarization (2020)

Abstract: We consider the problem of better modeling query-cluster interactions to facilitate query focused multi-document summarization. Due to the lack of training data, existing work relies heavily on retrieval-style methods for assembling query relevant summaries. We propose a coarse-to-fine modeling framework which employs progressively more accurate modules for estimating whether text segments are relevant, likely to contain an answer, and central. The modules can be independently developed and leverage training data if available. We present an instantiation of this framework with a trained evidence estimator which relies on distant supervision from question answering (where various resources exist) to identify segments which are likely to answer the query and should be included in the summary. Our framework is robust across domains and query types (i.e., long vs short) and outperforms strong comparison systems on benchmark datasets.

## introduction query focused multi-document summarization (qfs; dang 2006) aims to create a short summary from a set of documents that answers a specific query. it has various applications in personalized information retrieval and recommendation engines where search results can be tailored to an information need (e.g., a user might be looking for an overview summary or a more detailed one which would allow them to answer a specific question). neural approaches have become increasingly popular in single-document text summarization @xcite @xcite @xcite , thanks to the representational power afforded by deeper architectures and the availability of large-scale datasets containing hundreds of thousands of document-summary pairs @xcite @xcite . unfortunately, such datasets do not exist in qfs, and one might argue it is unrealistic they will ever be created for millions of queries, across different domains, and languages. in addition to the difficulties in obtaining training data, another obstacle to the application of end-to-end neural models is the size and number of source documents which can be very large. it is practically unfeasible (given memory limitations of current hardware) to train a model which encodes all of them into vectors and subsequently generates a summary from them. in this paper we propose a coarse-to-fine modeling framework for extractive qfs which incorporates a relevance estimator for retrieving textual segments (e.g., sentences or longer passages) associated with a query, an evidence estimator which further isolates segments likely to contain answers to the query, and a centrality estimator which finally selects which segments to include in the summary. the vast majority of previous work @xcite @xcite creates summaries by ranking textual segments (usually sentences) according to their relationship (e.g., similarity) to other segments and their relevance to the query. in other words, relevance and evidence estimation are subservient to estimating the centrality of a segment (e.g., with a graph-based model). we argue that disentangling these subtasks allows us to better model the query and specialize the summaries to specific questions or topics @xcite . a coarse-to-fine approach is also expedient from a computational perspective; at each step the model processes a decreasing number of segments (rather than entire documents), and as a result is insensitive to the original input size and more scalable. our key insight is to treat evidence estimation as a question answering task where a cluster of po-tentially relevant documents provides support for answering a query @xcite . advantageously, we are able to train the evidence estimator on existing large-scale question answering datasets @xcite @xcite , alleviating the data paucity problem in qfs. existing qfs systems @xcite @xcite employ classic retrieval techniques (such as tf-idf) to estimate the affinity between query-sentence pairs. such techniques can handle short keyword queries, but are less appropriate in qfs settings where query narratives can be long and complex. we argue that a trained evidence estimator might be better at performing semantic matching @xcite between queries and document segments. to this effect, we experiment with two popular qa settings, namely answer sentence selection @xcite and machine reading comprehension @xcite which operates over passages than isolated sentences. in both cases, our evidence estimators take advantage of powerful pre-trained encoders such as bert @xcite , to better capture semantic interactions between queries and text units. our contributions in this work are threefold: we propose a coarse-to-fine model for qfs which we argue allows to introduce trainable components taking advantage of existing datasets and pre-trained models; we capitalize on the connections of qfs with question answering and propose different ways to effectively estimate the query-segment relationship; we provide experimental results on several benchmarks which show that our model consistently outperforms strong comparison systems across domains (news articles vs. medical text) and query types (long narratives vs. keywords). ## related work existing research on query-focused multidocument summarization largely lies on extractive approaches, where systems usually take as input a set of documents and select the sentences most relevant to the query for inclusion in the summary. in figure 1 (a), we provide a sketch of classic centrality-based approaches which have generally shown strong performance in qfs. under this framework, all sentences within a document cluster, together with their query relevance, are jointly considered in estimating centrality. a vari-ety of approaches have been proposed to enhance the way relevance and centrality are estimated ranging from incorporating topic-sensitive information @xcite @xcite , predictions about information certainty @xcite , manifold-ranking algorithms @xcite @xcite , and wikipedia-based query expansion @xcite . more recently, @xcite estimate the salience of text units within a sparsecoding framework by additionally taking into account reader comments (associated with news reports). @xcite use a cascaded neural attention model to find salient sentences, whereas in follow-on work @xcite employ a generative model which maps sentences to a latent semantic space while a reconstruction model estimates sentence salience. there are also feature-based approaches achieving good results by optimizing sentence selection under a summary length constraint @xcite . in contrast to previous work, our proposal does not simultaneously perform segment selection and query matching. we introduce a coarse-to-fine approach that incorporates progressively more accurate components for selecting segments to include in the summary, making model performance relatively insensitive to the number and size of input documents. drawing inspiration from recent work on qa, we take advantage of existing datasets in order to reliably estimate the relationship between the query and candidate segments. we focus on two qa subtasks which have attracted considerable attention in the literature, namely answer sentence selection which aims to extract answers from a set of pre-selected sentences @xcite @xcite and machine reading comprehension @xcite @xcite , which aims at answering a question after processing a short text passage @xcite . qa and qfs are related but ultimately different tasks. qa aims at finding the best answer in a span or sentence, while qfs extracts a set of sentences based on user preferences and the content of the input documents under a length budget @xcite . qa questions are often short and fact-based while qfs narratives can be longer and more complex (see the example in section 3) and as a result simply localizing an answer within a cluster is not optimal. the classic approach involves a relevance estimator nested within a summarization module while our framework takes document clusters as input, and sequentially processes them with three individual modules (relevance, evidence, and centrality estimators). the blue circles indicate a coarse-to-fine estimation process from original articles to final summaries where modules gradually discard segments (i.e., sentences or passages). with regard to evidence estimation, we adopt pretrained bert @xcite which is further fine-tuned with distant signals from question answering. ## problem formulation let q denote an information request and @xmath0, d 2 , . . . , d m } a set of topic-related documents. it is often assumed (e.g., in duc competitions) that q consists of a short title (e.g., amnesty international ) highlighting the topic of interest, and a query narrative which is considerably longer and detailed (e.g., what is the scope of operations of amnesty international and what are the international reactions to its activities? ). we illustrate our proposed framework in figure 1(b) . we first decompose documents into segments, i.e., passages or sentences, and retrieve those which are most relevant to query q (relevance estimator). then, a trained estimator quantifies the semantic match between selected segments and the query (evidence estimator) to further isolate segments for consideration in the output summary (centrality estimator). we propose two variants of our evidence estimator; a context agnostic variant infers evidence scores over individual sentences, while a context aware one infers evidence scores for tokens within a passage which are further aggregated into sentence-level evidence. passages might allow for semantic relations to be estimated more reliably since neighboring context is also taken into account. ## experimental setup datasets we performed qfs experiments on the duc 2005-2007 benchmarks and the topically diverse qfs dataset (td-qfs; @xcite . duc benchmarks contain long query narratives over 50 clusters with 32-25 documents each, and cover multiple domains. td-qfs focuses on medical texts, contains short keyword queries over 4 clusters with 185 documents each. as a result, td-qfs clusters are less topically concentrated, with larger amounts of query-irrelevant information @xcite . although our approach is motivated by the desire to better model long and complex queries, experiments on td-qfs examine whether it generalizes to out-of-domain queries and clusters. we used duc 2005 as a development set to optimize hyperparameters and evaluated performance on duc 2006-2007 and td-qfs. a summary of the characteristics of these datasets ## conclusions in this work, we proposed a coarse-to-fine estimation framework for query focused multi-document summarization. we explored the potential of leveraging distant supervision signals from question answering to better capture the semantic relations between queries and document segments. experimental results across datasets show that the proposed model yields results superior to competitive baselines contributing to summaries which are more relevant and less redundant. we have also shown that disentangling the tasks of relevance, evidence, and centrality estimation is beneficial allowing us to progressively specialize the summaries to the semantics of the query. in the future, we would like to generate abstractive summaries following an unsupervised approach @xcite and investigate how recent advances in open domain qa @xcite can be adapted for query focused summarization. gpus with 11gb memory. for the answer sentence selection model, bert was fine-tuned with a learning rate of 3 × 10 - foot_5 and a batch size of 16 for 3 epochs @xcite . for span selection, we adopted a learning rate of 3 × 10 -5 and a batch size of 64 for 5 epochs. during inference, the confidence threshold for the relevance estimator was set to θ = 0.75 @xcite for both sentence and passage retrieval. for the evidence estimator, k qa was tuned on the development set. we obtained 90 and 110 evidence sentences from the sentence selection and span selection models, respectively. for the centrality estimator, the influence of the query was set to φ = 0.15 @xcite . the td-qfs dataset used in this work is publicly available at https://www.cs.bgu.ac. il/ ˜talbau/td-qfs/dataset.html.