Factorizing Content and Budget Decisions in Abstractive Summarization of Long Documents (2022)

Abstract: We argue that disentangling content selection from the budget used to cover salient content improves the performance and applicability of abstractive summarizers. Our method, FactorSum, does this disentanglement by factorizing summarization into two steps through an energy function: (1) generation of abstractive summary views covering salient information in subsets of the input document (document views); (2) combination of these views into a final summary, following a budget and content guidance. This guidance may come from different sources, including from an advisor model such as BART or BigBird, or in oracle mode – from the reference. This factorization achieves significantly higher ROUGE scores on multiple benchmarks for long document summarization, namely PubMed, arXiv, and GovReport. Most notably, our model is effective for domain adaptation. When trained only on PubMed samples, it achieves a 46.29 ROUGE-1 score on arXiv, outperforming PEGASUS trained in domain by a large margin. Our experimental results indicate that the performance gains are due to more flexible budget adaptation and processing of shorter contexts provided by partial document views.

## introduction casting summarization as a language transduction problem is convenient given the existence of powerful neural sequence-to-sequence models that produce high-quality textual outputs @xcite . however, this framework conflates multiple steps of the summarization decision-making process into a single feedforward step without taking into account the contextual factors involved @xcite . one such decision depending on context factors is the quantity of information to be included in a summary, reflecting on the generated outputs' length. this factor is particularly relevant for long documents that cover many different aspects of interest for which different summaries may be suitable. for instance, in samples from summarization datasets such as pubmed and arxiv @xcite , there are many abstracts including terse passages about the background or methods of the research. in contrast, others will add more details about those aspects. often, those choices are due to the author's preferences and do not necessarily represent an ideal summary for the document. furthermore, current evaluation protocols based on n-gram overlap are sensitive to summary lengths @xcite . generating summaries that match ground-truth lengths increases performance (see section 4.1). thus, recent progress in summarization may be the effect of better length prediction and not the actual summarization desideratum: a reductive transformation of the source text that keeps the important information @xcite . to address this issue, we propose to avoid budget information as a confounding factor as much as possible in sequence-to-sequence training. instead, we treat budget decisions as extrinsic guidance during summary generation, that is, an objective that is unrelated to the content of the documents. in this setting, the neural abstractive model is responsible for the generation of short passages (summary views) capturing relevant topics of the input document (intrinsic importance objective), while the extrinsic importance objective will encourage the adherence of generated summaries to context factors such as budgets or aspect coverage. specifically, we formulate factorsum, a factorized energy-based model @xcite aiming to find a summary that maximizes the total importance given a source document, a reference dataset, and contextual factors such as budget and content guidance. a key piece of our model is the sampling of random document views (and corresponding reference summary views) which allows the abstractive model to focus on shorter summarization tasks with less influence of varying summary lengths. also, this approach allows the processing of long documents without truncation, which is a recurring problem in summarization @xcite . the sampling procedure is detailed in section 2.1. our model comprises two optimization procedures: learning and inference. in the learning phase, the model parameters are optimized so that summary views with important content (as informed by reference summaries) will have lower energies. in practice, this is implemented by training a neural sequence-to-sequence model to predict summary views. during inference, a greedy optimization algorithm is used to find the combination of summary views that maximize the compatibility with the target budget and other types of guidance. this process is illustrated in figure 1 . our experimental results on the pubmed, arxiv, and govreport summarization benchmarks @xcite show that our approach using budget guidance alone is competitive with resource-intensive baselines such as pega-sus @xcite . the results confirm that matching reference summary lengths significantly impacts rouge scores, often more than different modeling approaches. we also investigate the use of existing baselines as additional guidance during summary generation. in contrast to teacher models in knowledge distillation literature @xcite , we leverage existing model prediction during inference only, and thus, we adopt the term advisor model to refer to our summarization guidance approach. when guided by bigbird or bart, our model obtains state-of-the-art results on pubmed, arxiv, and govreport. finally, we perform domain adaptation experiments in which models trained on pubmed, arxiv, and govreport have no access to samples from the evaluation dataset during training. our results indicate that factorsum can adapt better to outof-domain data, outperforming strong baselines trained in domain on both pubmed and arxiv. this finding suggests a good generalization capacity and is evidence that we achieved our objective to disentangle content selection from budget decisions. ## intrinsic and extrinsic importance since our objective is to explicitly model budget decisions, we need a definition of importance that accounts for context factors. inspired by the information-theoretic notion of importance developed by @xcite , we introduce the notion of importance with respect to intrinsic and extrinsic semantic units. intrinsic semantic units are those specific to the document (e.g., salient topics) whereas extrinsic units are related to a priori external preferences or require domain knowledge and grounding that is hard to capture from the textual corpora alone. we argue that the usual setting of end-to-end supervised summarization evaluated by rouge @xcite optimizes for intrinsic importance. in this work, budget and content guidance provided by advisor model summaries (section 2.1.2) play the role of extrinsic information. formally, we define the best summary s * for a document d as the summary that minimizes the following factorized energy @xcite : an overview of the model components and the summary inference procedure is represented in figure 1 . given a document d, n d document views are generated, each covering a random subset of sentences from the original document (section 2.1). then, the intrinsic importance model generates summary views s v , each partially covering salient content from the original document d (section 2.1.1). finally, the extrinsic importance model will optimize the final summary s so that it maximizes the alignment of the content to a target budget and content guidance. in the following sections, we detail the model components as well as the training and inference procedures. ## experimental setup we turn to discuss the experimental setup to assess the effectiveness of the factorized model proposed in section 2. we specify details about datasets, baselines, and the evaluation protocol. (see appendices c and d for further implementation details). datasets our experiments are performed on the pubmed and arxiv datasets, consisting of documents extracted from the homonymous scientific repositories @xcite . to further test the generalization capacity of the model, we also perform the experiments on govreport, a dataset containing long reports published by u.s. government accountability office (gao; @xcite . the only preprocessing applied is to filter out documents with empty articles or summaries, algorithm 1: greedy summary generation. input parameters are the set of summary views v d for document d, content guidance c, budget guidance b, redundancy threshold t, and patience p. see the "non-redundancy" paragraph in section 2.1.2 for a discussion about the is_redundant function. input: vd, c, b, t, p output: ## results in this section, we analyze the contribution of different guidance factors on summarization performance by controlling summary budget and content guidance. we continue by conducting an ablation study, examining the summary views and the greedy summary generation contributions to the overall performance. finally, we discuss domain adaptation results. sample summaries are provided in appendix f. ## related work ranking and aggregating candidate summaries recent work explores the idea of sampling summary candidates that are scored to form a final summary. the simcls @xcite ) model generates several summary candidates using diverse beam search @xcite , which are ranked according to a learned evaluation function. also, the perturb-and-select summarizer @xcite uses similar ideas in a multi-document opinion summarization task. instead of a diverse beam search, it performs random perturbations in the model inputs to generate candidates ranked according to a coherence model. @xcite presented coop, a framework that improves the aggregation methods for multi-document opinion summarization representations by maximizing their input-output word overlap. our approach differs from these models as our sampled summary views are not full summary candidates but partial views with important semantic content from the original document. also, the simcls ranking uses intrinsic importance only (similarily with respect to the original document). ## conclusion and future work in this work, we embrace the idea that generalpurpose summary is an elusive goal and that contextual factors such as preferences for summary lengths are essential in the design of summarization systems @xcite . we propose a framework to separate budget decisions from selecting important content in the document using neural sequence-to-sequence models as building blocks. our results suggest improved performance in both in-domain and cross-domain summarization of long documents. in future work, we plan to investigate the effects of domain-specific extrinsic guidance that would encourage the summaries to cover aspects of interest.