Document Summarization with Latent Queries (2022)

Abstract: The availability of large-scale datasets has driven the development of neural models that create generic summaries for single or multiple documents. For query-focused summarization (QFS), labeled training data in the form of queries, documents, and summaries is not readily available. We provide a unified modeling framework for any kind of summarization, under the assumption that all summaries are a response to a query, which is observed in the case of QFS and latent in the case of generic summarization. We model queries as discrete latent variables over document tokens, and learn representations compatible with observed and unobserved query verbalizations. Our framework formulates summarization as a generative process, and jointly optimizes a latent query model and a conditional language model. Despite learning from generic summarization data only, our approach outperforms strong comparison systems across benchmarks, query types, document settings, and target domains.1

## introduction recent years have witnessed substantial progress in generic summarization @xcite @xcite thanks to neural architectures based on the encoder-decoder paradigm @xcite and the availability of large-scale datasets containing hundreds of thousands of document-summary pairs. unfortunately, training data of this magnitude is not readily available for the related task of query-focused summarization (qfs; dang 2005) which aims to create a summary from one or multiple document(s) that answers a specific query. existing qfs benchmarks @xcite @xcite have been constructively used for evaluation but are relatively small for training large neural models. to make up for the absence of labeled qfs data, recent work has resorted to distant supervision provided by pretrained models, paraphrase identification, and question-answering datasets @xcite @xcite . other work induces proxy queries @xcite ) from generic summarization datasets, without additional question-answering resources that can be also extremely expensive to acquire @xcite . despite this progress, building and scaling qfs systems remains challenging due to the many different ways natural language queries express users' information needs. for instance, queries can have one or multiple keyword(s) @xcite , a simple question @xcite , or a longer narrative composed of multiple sub-queries @xcite (see the examples in in this work, we provide a unified modeling framework for generic summarization and qfs, under the assumption that only data for the former is available. specifically, we treat generic summarization as a special case of qfs where the query is latent. we model queries as discrete latent variables over document tokens, and learn representations compatible with observed and unobserved query verbalizations. our 2 related work @xcite and @xcite were among the first to apply the neural encoder-decoder architecture to abstractive summarization. @xcite enhance their approach with a pointer-generator model, essentially a copy mechanism allowing words from the source document to be copied directly in the summary. @xcite incorporate a content selection model that decides on relevant aspects of the source document. they frame this task as a word-level tagging problem, with the objective of separately identifying tokens from a document that should be part of its summary; at test time, they produce content selection probabilities for each word, which are then used to restrict the copy mechanism by performing hard masking over the input document. another line of research controls summary generation via topics @xcite , retrieve-and-edit methods @xcite , factual relations (jin et al., 2020), keywords, relational triples, or preselected source sentences @xcite . the majority of previous qfs approaches have been extractive and compose summaries by selecting central and query-relevant sentences @xcite @xcite . more recently, @xcite propose a coarse-to-fine framework that leverages distant supervision from question answering for summary sentence extraction. abstractive qfs has received significantly less attention in comparison, due to generation models being particularly data-hungry @xcite . as a result, resources from a wider range of nlp tasks have been used. @xcite rank document paragraphs against queries with the aid of qa and machine reading datasets @xcite , and then iteratively summarize selected paragraphs. similarly, laskar et al. (2020b) jointly exploit supervision from qfs data (typically reserved for evaluation) and related qa and paraphrase identification tasks. because query-related resources can be also costly to obtain @xcite , @xcite use none whatsoever. instead, they create proxy queries by selectively masking information slots in generic summaries. despite promising system performance, their approach assumes prior knowledge of target queries (proxies are created to match their length, and content), and a development set is used @xcite . also, their system is particularly tailored to multi-document qfs and includes a sophisticated evidence selection component. our work is closely related to theirs in that we also do not take advantage of query-related resources. we go a step further and do not require a development set either, allowing our model to be independent of specific query verbalizations and produce qfs summaries in zero-shot settings. our approach is generally applicable to singleand multi-document qfs. for any summarization task we assume that queries are latent and estimate these jointly via a summarization and (weakly supervised) tagging task. the latter draws inspiration from @xcite under the assumption that document tokens found in the summary also provide evidence for the (latent) query that gave rise to it. finally, our model is fundamentally different from approaches that rely on document-based guidance to improve the informativeness @xcite or faithfulness @xcite of summaries. while these models exploit guidance from supervision signals in training data, we are faced with the problem of estimating queries when there are none available (at least during training). ## problem formulation let {(d, q, s)} denote a summarization dataset, where document d is a sequence of tokens, and s its corresponding summary; query q additionally specifies an information request. in generic summarization, @xmath0, whereas in qfs q can assume various formats, ranging from keywords to composite questions (see our model learns from generic summarization data alone, while robustly generalizing to a range of tasks at test time, including out-of-domain qfs. a shared characteristic between generic summarization and qfs is the fact that user intent is underspecified. even when queries are available (i.e., @xmath1, they are incomplete expressions of intent as it is unlikely to specify queries to the level of detail necessary to compose a good summary @xcite . we thus identify latent query signals from d, and optionally take advantage of q as additional observation for belief update. generative model we model an observed input document d as a sequence of random variables @xmath2; x 2 ; . . . ; x m ] where x i is a token and m the length of the document. we define the latent query as a sequence of discrete latent states over input document tokens: @xmath3; z 2 ; . . . ; z m ]. specifically, from each document token x i , we generate a binary query variable z i , whose distribution p(z i ) represents the belief that x i contributes to a potential query for document d. modeling latent queries at the token-level allows us to regularize the model-by taking into account weak supervision in the form of token-level tagging @xcite . it also renders the model independent of the query form, thereby enabling zero-shot inference (see section 4). the output summary @xmath4; y 2 ; . . . ; y t ] is then generated from {x, z} using teacher-forcing at training time. at test time, we may additionally be presented with a query q; we ground this optional information to the input document via discrete observed variables @xmath5; z2 ; . . . ; zm ], and generate y by additionally conditioning on z (if it exists) in an autoregressive manner. our model estimates the conditional distribution p θ (y|x) according to the generative process just described (and illustrated in figure 1 ) as: inference model the posterior distribution of latent variable z is calculated as: unfortunately, exact inference of this posterior is computationally intractable due to the joint probability p θ (x, y). we therefore approximate it with a variational posterior q φ (z|x, y). inspired by β-vae @xcite , we maximize the probability of generating summary y, provided the distance between the prior and variational posterior distributions is below a small constant δ: because we cannot solve equation ( 4 ) directly, we invoke the karush-kuhn-tucker conditions ( @xcite and cast the above constrained optimization problem into unconstrained optimization, with the following elbo objective: we discuss how this tagger is learned in section 4. for now, suffice it to say that weak supervision is a form of posterior regularization adding an extra term in the objective, which we rewrite as: as can be seen from equation ( 6 ), we decompose summarization into two modeling objectives, namely, latent query modeling and conditional language modeling. inside the query modeling term, hyperparameter ω controls the influence of weak supervision ẑ, while β controls the strength of label smoothing on the weak annotations. ## latent query model in this section we discuss how the inference network for latent queries is constructed. we also explain how query-focused document representations are obtained, our attempts to mitigate posterior collapse via weak supervision o(ẑ|x, y) (see equation ( 6 )), and how query belief is updated when queries are available at test time. inference network for latent queries we construct a neural network model to infer for each token in the input document whether it constitutes a query term. given a contextual token representation matrix h @xmath6, we project it to r m ×2 with a two-layer mlp as a scoring function: let g(0) denote the standard gumbel distribution, and g ℓ ∼ g(0), ℓ ∈ [0, 1] is i.i.d. drawn gumbel noise. we normalize π to form a variational distribution as: query-focused view as explained earlier, in addition to a canonical, query-agnostic encoding of the input document d (which we discuss in section 5), we further introduce a query-focused encoding factorized via latent queries z. specifically, for the ith token, we take the continuous relaxation of its discrete latent variable z i , and ground foot_4 it to the input document via: as we can see, the query-focused view explicitly models the dependency on latent queries. from a learning perspective, this factorization leads to the following partial derivatives of the query encoder states with respect to the query-focused view: token tagging as weak supervision although it is possible to optimize latent queries solely based on conditional language modeling (our approach is fully differentiable), we additionally exploit weak supervision to label tokens in the document as query-specific or not. weak supervision is advantageous as it imposes extra regularization on the posterior (see equation ( 6 )), thereby mitigating its collapse (i.e., the decoder may learn to ignore the query-focused view and instead solely rely on the query-agnostic view). let t 1 , . . . , t n denote binary tags for each of the source tokens, that is, 1 if a token is query-specific and 0 otherwise. we could learn such a tagger from training data generated by aligning query tokens to the document. in default of such gold-standard data, we approximate queries by summaries and obtain silver standard token labels by aligning summaries to their corresponding documents. specifically, inspired by @xcite , we assume a token in the document is query-specific if it is part of the longest common sub-sequence (lcs) of tokens in the summary. our tagging model is built on top of a pretrained language model, and thus operates on subwords. we first byte-pair encode (bpe; @xcite documents and summaries, and then search for the lcs over bpe sequences. if there exist multiple identical lcss, only the one appearing at the earliest document position is differentiable with biased gradient estimation. however, it did not yield better results than continuous relaxation. tagged as positive. we refer to this tagging scheme as bpe-lcs. note that although we model query variables at the token level, we take phrases indirectly into account through lcs, which identifies subsequences of tokens (or phrases) as query annotations. our our tagging model is therefore able to capture dependencies between tokens, albeit indirectly. training to optimize the variational inference model, that is, the mlp defined in equations (7-9), we use a cross entropy loss for token tagging, with the posterior entropy term from equation ( 6 ). formally, we write the query modeling loss as follows: in the initial stages of training, the tagger might lead to inaccurate posterior probability assignments q φ (z i |x), and, consequently, hurt the summarization model, which relies heavily on a high-quality query-focused view. to address this issue, we introduce a posterior dropout mechanism that replaces the estimated posterior with weak supervision o(ẑ|x) according to probability α. we initialize α to 1, so that only o(ẑ|x) is used in the beginning of training, and the tagger is supervised via equation ( 12 ). we then linearly anneal α over optimization steps so that the gradients from the summarization objective (which we introduce in section 5) can jointly optimize the tagger. zero-shot transfer we now explain how queries are taken into account at test time by performing query belief updates δ(z i |x, z). in the case of generic summarization where no queries are available, we simply perform no update. when @xmath7, some tokens in the document become more relevant and we consequently set δ(z @xmath8, z) = 1, ∀w @xmath9, q), and all other tokens to zero. we further incorporate query information via a simple calibration as: note that our calibration is non-parametric, since it is not realistic to assume access to a development set for each query type (e.g., in order to perform hyper-parameter tuning). this enables zero-shot transfer to qfs tasks with varying characteristics. ## conditional language model in this section we describe our conditional language model, which estimates the log-likelihood expectation of a summary sequence over the variational posterior (see equation ( 6 )). as mentioned earlier, we adopt an encoder-decoder architecture tailored to document summarization with latent queries. encoder we encode two views of the input document, a generic query-agnostic view d, and a query-focused one q (see equation ( 10 )). as shown in figure 1 (c), our encoder module consists of three encoders: a shared encoder, a document encoder, and a query encoder. because both views are created from the same document, we use a shared encoder for general document understanding that also reduces model parameters. the shared document representation serves as input to more specialized encoders. each encoder contains one or multiple transformer layers @xcite , each composed of a multi-head attention (mha) layer and a feed-forward (ffn) layer: decoder we adopt a decoder structure similar to @xcite to handle multiple inputs. our decoder sequentially attends to the two encoded views of the same document: after taking the context of the previous generation h (dec) into account, the decoder will first attend to signals coming from query q, then to original document d (based on guidance provided by the query). the final summary generation objective is calculated autoregressively as: ## automatic evaluation before analyzing our model under various zero-shot settings, we first confirm it can indeed produce good quality generic summaries in a supervised setting. there is no point in contemplating zero-shot scenarios if our approach underperforms when full supervision is available. following standard practice, we use f1 rouge as our automatic evaluation metric @xcite . unigram and bigram rouge (r-1 and r-2) are a proxy for assessing informativeness and the longest common subsequence (r-l) represents fluency. for multi-document qfs, we follow duc @xcite and report r-su4 (based on skip bigram with maximum skip distance of 4) instead of r-l. 5 the third block includes various abstractive systems (see section 2 for an overview). ptgen @xcite and bottomup @xcite do not use pretrained lms, while bertabs @xcite ) is built on top of a pretrained bert encoder. bart @xcite is fine-tuned on cnn/dm, while gsum @xcite is initialized with bart parameters. our latent query summarization model (lqsum) outperforms bart by a large margin, which demonstrates the effectiveness of latent queries even for generic summarization. it also performs on par with gsum, under identical training resources and configurations. gsum is a state-of-the-art abstractive model, which relies on matchsum @xcite , a high-performance extractive model to provide guidance to the decoder. compared to gsum, lqsum can be trained end-to-end and requires significantly less parameters (406 m for lqsum versus 625 m for gsum; see ## human evaluation following previous work @xcite , we also evaluated query-focused summaries in a judgment elicitation study via amazon mechanical turk. native english speakers (self-reported) were asked to rate query-summary pairs on two dimensions: succinctness (does the summary avoid unnecessary detail and redundant query: effectiveness: do earmarks allocate spending effectively? gold: earmarks are often unrelated to legislation; holds up bill. bertabs: earmarks can be fully examined. gsum+lexrank q : sometimes a good piece of legislation that receives the support of a majority of congressman will be held up and voted down. lqsum: congressmen are using earmarks to hold up bills they don't like, says rep. ruben gallego. in addition, participants were asked to assess the relevance of the summary to the query. crowdworkers read a summary and for each sentence decided whether it is relevant (i.e., provides an answer to the query), irrelevant (i.e., does not answer the query), or partially relevant (i.e., unclear it directly answers the query). relevant sentences were awarded a score of 5, partially relevant ones a score of 2.5, and 0 otherwise. sentence scores were averaged to obtain a relevance score for the whole summary. we view relevance as as more critical for qfs than coherence or succinctness. this is why we obtained per-sentence ratings which we then aggregated to an overall summary score. to make this task manageable, raters were asked to provide more coarse-grained ratings. participants assessed summaries created by lqsum (our model), gsum+lexrank q (a competitive abstractive system), lexrank q (an extractive baseline), and gold (the ground-truth upper bound). we also compared against bertext on wikiref, bertabs on debatepedia, and marge-sum on duc and td-qfs. 9 we sampled 40 query-document pairs from wikiref and debatepedia, 40 query-cluster pairs from duc @xcite 20 from each set), and 40 pairs from td-qfs and collected three responses per pair. 10 we show our results in ## conclusion we propose a deep generative formulation for document summarization that supports generic and query-focused applications. we represent queries as discrete latent variables, whose approximated posterior distribution can be calibrated with query observations at test time without further adaptation. our approach does not rely on any query-related resource and can be applied in zero-shot settings. experimental results across summarization datasets show that the proposed model yields state-of-the-art qfs performance in zero-shot settings. directions for future work are many and varied. one research challenge is to push this low-resource approach even further and generate abstractive summaries without access to any summaries or queries. we would also like to extend the proposed framework to cross-lingual settings, and satisfy the information needs of users with different language backgrounds through effective query understanding and summary generation.