KL earn: Background Knowledge Inference from Summarization Data (2020)

Abstract: The goal of text summarization is to compress documents to the relevant information while excluding background information already known to the receiver. So far, summarization researchers have given considerably more attention to relevance than to background knowledge. In contrast, this work puts background knowledge in the foreground. Building on the realization that the choices made by human summarizers and annotators contain implicit information about their background knowledge, we develop and compare techniques for inferring background knowledge from summarization data. Based on this framework, we define summary scoring functions that explicitly model background knowledge, and show that these scoring functions fit human judgments significantly better than baselines. We illustrate some of the many potential applications of our framework. First, we provide insights into human information importance priors. Second, we demonstrate that averaging the background knowledge of multiple, potentially biased annotators or corpora greatly improves summaryscoring performance. Finally, we discuss potential applications of our framework beyond summarization.

## introduction summarization is the process of identifying the most important information pieces in a document. for humans, this process is heavily guided by background knowledge, which encompasses preconceptions about the task and priors about what kind of information is important @xcite . despite its fundamental role, background knowledge has received little attention from the summarization community. existing approaches largely focus on the relevance aspect, which enforces similarity between the generated summaries and the source documents @xcite . in previous work, background knowledge has usually been modeled by simple aggregation of large background corpora. for instance, using tf•idf @xcite , one may operationalize background knowledge as the set of words with a large document frequency in background corpora. however, the assumption that frequently discussed topics reflect what is, on average, known does not necessarily hold. for example, commonsense information is often not even discussed @xcite . also, information present in background texts has already gone through the importance filter of humans, e.g., writers and publishers. in general, a particular difficulty preventing the development of proper background knowledge models is its latent nature. we can only hope to infer it from proxy signals. besides, there is, at present, no principled way to compare and evaluate background knowledge models. in this work, we put the background knowledge in the foreground and propose to infer it from summarization data. indeed, choices made by human summarizers and human annotators provide implicit information about their background knowledge. we build upon a recent theoretical model of information selection @xcite , which postulates that information selected in the summary results from 3 desiderata: low redundancy (the summary contain diverse information), high relevance (the summary is representative of the document), and high informativeness (the summary adds new information on top of the background knowledge). the tension between these 3 elements is encoded in a summary scoring function θ k that explicitly depends on the background knowledge k. as illustrated by fig. 1 , the latent k can then be inferred from the residual differences in information selection that are not explained by relevance and redundancy. for example, the black information unit in fig. 1 is not selected in the summary despite being very prominent in the source document. intuitively, this is explained if this unit is already known by the receiver. to leverage this implicit signal, we view k as a latent parameter learned to best fit the observed summarization data. ## related work the modeling of background knowledge has received little attention by the summarization community, although the problem of identifying content words was already encountered in some of the earliest work on summarization @xcite . a simple and effective solution came from the field of information retrieval, using techniques such as tf•idf on background corpora @xcite . similarly, @xcite proposed the loglikelihood ratio test to identify highly descriptive words. these techniques are known to be useful for news summarization @xcite . later approaches include heuristics to identify summary-worthy bigrams @xcite . also, @xcite proposed a supervised model for predicting whether a word will appear in a summary or not (using a large set of features including global indicators from the new york times corpus) which can then serve as a prior of word importance. @xcite proposed to model background knowledge by aggregating a large random set of news articles. @xcite used bayesian topic models to ensure the extraction of informative summaries. finally, @xcite investigated background knowledge for update summarization with bayesian surprise. these ideas have been generalized in an abstract model of importance @xcite discussed in the next section. ## background this work builds upon the abstract model introduced by @xcite , whose relevant aspects we briefly present here. let t be a text and a function mapping a text to its semantic representation of the following form: the semantic representation is a probability distribution p over so-called semantic units {ω j } @xmath0. many different text representation techniques can be chosen, e.g., topic models with topics as semantic units, or a properly renormalized semantic vector space with the dimensions as semantic units. in the summarization setting, the source document d and the summary s are represented by probability distributions over the semantic units, p d and p s . similarly, k, the background knowledge, is represented as a distribution p k over semantic units. 1 intuitively, p k (ω j ) is high whenever ω j is known. a summary scoring θ k (s, d) (or simply θ k (s) since the document d is never ambiguous) can be derived from simple requirements: in this work, we fix α = β = 1. ## the klearn framework as laid out, in our framework, texts are viewed as distributions over a choice of semantic units {ω j } @xmath1. we aim to infer a general k as the distribution over these units that best explains summarization data. we consider two types of data: with and without human judgments. ## comparison of approaches to compare the usefulness of various k's, we need a way to evaluate them. fortunately, there is a natural evaluation setup: (i) plug k into θ k , the summary scoring function described by eq. 2, (ii) use the induced θ k to score summaries s i , and (iii) compute the agreement with human scores h i . to distinguish between the algorithms introduced in sec. 4, we adopt the following naming convention for scoring functions: if the background knowledge k was computed using algorithm a, we denote the corresponding scoring function by θ a ; e.g., θ hpl is the scoring function where k was inferred by hpl. data. we use two datasets from the text analysis conference (tac) shared task: tac-2008 and tac-2009. 2 they contain 48 and 44 topics, respectively. each topic was summarized by about 50 systems and 4 humans. all system summaries and human-written summaries were manually evaluated by nist assessors for readability, content selection with pyramid @xcite , and overall responsiveness @xcite . in this evaluation, we focus on the pyramid score, as the framework is built to model the content selection aspect. semantic units. as in previous work @xcite , we use words as semantic units. in sec. 7, we also experiment with topic models. however, different choices of text representations can be easily plugged in the proposed methods. words have the advantage of being simple and directly comparable to existing baselines. baselines. for reference, we report the summary scoring functions of several baselines: lexrank (lr) @xcite ) is a graph-based approach whose summary scoring function is the average centrality of sentences in the summary. icsi @xcite scores summaries based on their coverage of frequent bigrams from the source documents. kl(s d) and js(s d) @xcite measure divergences between the distribution of words in the summary and in the sources. js divergence is a symmetrized and smoothed version of kl divergence. additionally, we report the performance of choosing the uniform distribution for k (denoted θ u ) and an idf-baseline where k is built from the document frequency computed using the english wikipedia (denoted as θ idf ). for reference, we report the performance of training and evaluating θ hpl on all data (denoted as optimal). this measures the ability of hpl to fit the training data. improvements over baseline are also obtained by hpl, which leverages the fine-grained information of human judgments. however, even without benefiting from supervision, ms|d performs similarly to hpl without significant difference. also, as expected, the preference learning setup θ hpl is stronger and more robust than the regression setup θ hreg , which does not significantly outperform the uniform baseline θ u . therefore, we use hpl when human judgments are available and ms|d when only documentsummary pairs are available. mary s is different from k (kl(s k) is large) but still similar to the document d (kl(s d) is small). furthermore, the regularization term in eq. 3, with @xmath2, makes minimal commitment as to what k should look like, i.e., no a-priori information except the documents is assumed. viewing these distributions as points in euclidean space, the optimal arrangement for s, d, and k is on a line with d in between s and k. since human-written summaries s and documents d are given, inferring k intuitively consists in discovering the point in high-dimensional space matching this property for all document-summary pairs. interestingly, we can easily test whether this geometrical structure appears in real data with our inferred k. to do so, we perform a simultaneous multi-dimensional scaling (mds) embedding of documents d i , human-written summaries s i , and k. in this space, two distributions are close to each other if their kl divergence is low. we plot such an embedding in fig. 2 for 6 randomly chosen topics from tac-09 and k inferred by hpl. we indeed observe documents, summaries, and k nicely aligned such that the summaries are close to their documents but far away from k. this finding also holds for k inferred by ms|d. these observations are important for two reasons. (1) they show that general framework introduced in fig. 1 is an appropriate model of the summarization data: for any given topic, the reference summaries are arranged on one side of the document. they deviate from the document in a systematic way that is explained by the repulsive action of the background knowledge. human-written summaries contain information from the document but not from the background knowledge which puts them on the border of the space. (2) our models can be seen to infer an appropriate background knowledge that is common to a wide spectrum of topics, as shown by the fact that k occupies the central point in the embedding of fig. 2 . ## applications we now investigate some applications arising from our framework. as k is easily interpretable, we explore which units receive high or low scores. one can also use different subsets (or aggregations) of training data. here, we look into annotator-specific k's and domain-specific k's. ## conclusion we focus on the often-ignored background knowledge for summarization and infer it from implicit signals from human summarizers and annotators. we introduced and evaluated different approaches, observing strong abilities to fit the data. the newly-gained ability to infer interpretable priors on importance in a data-driven way has many potential applications. for example, we can describe which topics should be extracted more frequently by systems to improve their agreement with humans. using pretrained priors also helps systems to reduce overfitting on the frequency signal within source documents as illustrated by initial results in appendix d. an important application made possible by this framework is to infer k on any meaningful subset of the data. in particular, we learned annotatorspecific k's, which yielded interesting insights: some annotators exhibit large differences from the others, and averaging several, potentially biased k's results in generalization improvements. we also inferred k's from different summarization datasets and also found increased performance on the news domain when averaging k's from diverse domains. for future work, different choices of semantic units can be explored, e.g., learning k directly in the embedding space. also, we fixed α = β = 1 to get comparable results across methods, but including them as learnable parameters could provide further performance boosts. investigating how to infuse the fitted priors into summarization systems is another promising direction. more generally, inferring k from a commonsense task like summarization can provide insights about general human importance priors. inferring such priors has applications beyond summarization, as the framework can model any information selection task.