Do Multi-Document Summarization Models Synthesize? (2024)

Abstract: Multi-document summarization entails producing concise synopses of collections of inputs. For some applications, the synopsis should accurately synthesize inputs with respect to a key aspect, e.g., a synopsis of film reviews written about a particular movie should reflect the average critic consensus. As a more consequential example, narrative summaries that accompany biomedical systematic reviews of clinical trial results should accurately summarize the potentially conflicting results from individual trials. In this paper we ask: To what extent do modern multi-document summarization models implicitly perform this sort of synthesis? We run experiments over opinion and evidence synthesis datasets using a suite of summarization models, from fine-tuned transformers to GPT-4. We find that existing models partially perform synthesis, but imperfectly: Even the best performing models are over-sensitive to changes in input ordering and under-sensitive to changes in input compositions (e.g., ratio of positive to negative reviews). We propose a simple, general, effective method for improving model synthesis capabilities by generating an explicitly diverse set of candidate outputs, and then selecting from these the string best aligned with the expected aggregate measure for the inputs, or abstaining when the model produces no good candidate.

## introduction multi-document summarization (mds) models aim to distill inputs into concise synopses that preserve key content. examples of mds include summarizing news articles @xcite @xcite , answering questions from multiple sources @xcite , and producing overviews of scientific literature @xcite @xcite @xcite . we expect summarization models to produce outputs consistent with inputs @xcite , e.g., discussing the same types of entities @xcite and allowing one to answer questions similar in a way that is consistent with individual inputs @xcite . in some applications models must synthesize inputs-i.e., aggregate potentially conflicting information-to yield an accurate synopsis (figure 1 ). consider the meta-reviews of movies featured on rotten tomatoes, foot_0 which provide a consensus view of individual critic opinions. these reviews should reflect the mean and range of sentiment implicit in the input critiques: a summary of mostly negative reviews (e.g., gigli) should communicate that the film was widely panned; a summary of mixed reviews (the fifth element) ought to convey that critics disagreed and discuss the main positive and negative attributes. a more consequential example is summarizing the evidence presented in clinical trials. individual trials will often present conflicting evidence about whether or not a particular health intervention is effective. an ideal summary would appropriately weigh the findings presented in individual studies and reflect the evidence on balance. what are the desiderata of multi-document synthesis? first, summaries produced by models should be consistent with the input data, with respect to the latent property of interest. in the case of rotten tomatoes, the sentiment of the summary should be in line with the aggregate sentiment expressed in the individual critic reviews. a corollary to this is that models should be sensitive to changes in the composition of inputs, e.g., removing most of the negative reviews from a set of inputs should yield a summary with a corresponding increase in the expressed sentiment. in this work we evaluate neural mds models with respect to these criteria. to this end we use a meta-reviews dataset from rotten tomatoes @xcite and a dataset of systematic reviews (meta-analyses) summarizing the evidence about medical interventions @xcite . for the former we probe the degree to which generated meta-review sentiment agrees with the expected aggregate sentiment score; for the latter we evaluate whether the generated summary indicates that the input evidence suggests, on balance, that the intervention under consideration was effective. our main contributions are: ## synthesis and summarization in standard multi-document summarization, we assume inputs (x i , y i ); x @xmath0, . . . , x i|x i | }. we then typically train a summarization model with parameters θ, to consume x i and yield 2 shah et al. (2021a) study a low-resource health and nutrition setting, in which they extract relational tuples, apply a manual rule set for aggregation, and then generate a surface form following this result. see section 6 for a discussion of opinion summarization work which considers synthesis as a target but not measure of summarization performance. we are interested in settings in which there is an additional, latent property z ij implicit in the constituent input texts x ij . for example, z ij might reflect the sentiment in critique j of the film indexed by i. summaries should synthesize this aspect, i.e., the generated summary ŷi should implicitly convey an aggregated in both cases considered heresummaries of film critiques and synopses of clinical trials evidence-g can reasonably be assumed to be a (weighted that is, summaries should roughly reflect the average sentiment and reported treatment effect in the cases of movie reviews and clinical trial reports, respectively. we investigate the following questions. (1) do model summaries ŷi reflect the anticipated aggregate aspect of interest? that is, how well calibrated is the aspect communicated in the generated summary (z iŷ ) compared to the expected z i ? (2) do these same results apply to other (not solely transformer) mds architectures? (3) can we improve the ability of summarization models to synthesize by explicitly incorporating synthesis targets z i into the decoding process? we propose a simple inference-time procedure to explicitly preference output candidates that align with the expected aggregate property of interest (e.g., average sentiment), and report promising results under both automatic and manual evaluation. this strategy naturally lends itself to cautious summarization, i.e., approaches where the model can abstain from generating an output if it does not produce any candidates that reflect the anticipated aggregate measure. ## improving synthesis in summarization we propose a straightforward post-hoc approach to improving the synthesis performed by multi-document summarization models: (1) generate an explicitly diverse set of output candidates; for (1), we rely on an existing technique for generating diverse outputs c i from input x ⊕ i : diverse beam search (dbs) @xcite . this method modifies standard beam search to maintain multiple groups of beams. during decoding, a term is added to the next-token log probabilities, penalizing production of strings similar to candidates in other groups. 16 than some : |g(ŷ i,1,..,t )z i | < . we elide these results here as they were often disfluent. 16 this penalty requires a hyperparameter λ that encodes the relative importance of diversity; we use λ = 0.5. to in (2) we would like to select the output that best synthesizes the property of interest; this requires an approach to specify what we expect the synthesized property be, given the inputs. for example, if we know the sentiment scores associated with input movie reviews, we might enforce that the output sentiment agrees with the average of these. to realize this intuition, we can select as final output from c i the string that best aligns with this aggregate property (sentiment score or significance finding). operationally, this requires an external model to estimate the aspect of interest as latent in a given candidate output. this enable fair comparison with standard beam search (5 beams, in all experiments), we used 5 groups, 1 beam per group. we exclude qt as it is an extractive model, and plansum as it does not readily support diverse beach search. for acesum and reflect we modify these codebases to use the diverse beam search implementation from huggingface. for gpt-4 we sample five responses with a temperature of 0.6. @xcite and then select the text that best agrees with the predicted aggregate property of interest (here, sentiment). we can also abstain when the model fails to yield an appropriate output. is a limitation of the approach, but in many settings it may be feasible to identify or construct a model; we were able to do so for both tasks considered here. it may be that any member of c i will align well with the anticipated aggregated property. in such cases, we have no means of producing an output consistent with respect to synthesis, and it may be desirable to abstain from outputting anything at all in such cases; that is, to be a cautious summarizer @xcite . we consider this strategy in the case of generating narrative synopses of evidence, as this constitutes a case in which (a) one would very much prefer not to produce a misleading summary of clinical evidence @xcite , and, (b) we observe many cases where the diverse decoding strategy yields an output that seems to communicate (at a granular level) the aggregate findings expected. movie reviews. we use bert @xcite , fine-tuned on imdb (maas et al., 2011) 17 to predict the sentiment inputs x ij , using the proportion of x @xmath1. for each diverse prediction c i , we predict its sentiment ziŷ via our regression model (2.1), and select the prediction closest to the estimated target sentiment |z iŷz iŷ |. we find this improves model synthesis performance ( figure 9 : distributions of outputs for the candiate summaries. movie reviews (left) show a histogram for the range of differences between lowest and highest output sentiments. systematic reviews (right) show histograms of the fractions of outputs reporting significant results. and diverse outputs. 18 we find a moderate agreement cohen's κ = 0.59, and a statistically significant preference for the diverse summaries (@xmath2.003). ## conclusions we have outlined and investigated the problem of synthesis as related to some summarization tasks. we showed that existing models are partially able to synthesize implicitly, but do so imperfectly: the aggregation they perform is sensitive to input ordering, and they are not as sensitive to perturbations in the composition of inputs as one would hope. some models specifically designed for these tasks (acesum, qt, reflect) are less sensitive to these perturbations, but offer worse overall performance than an equivalently sized transformer model (compare led and reflect -reflect integrates a model with the same base llm parameters as a portion of its synthesis model). furthermore, increasing model size within an architecture can lead to fairly substantial improvements (led to primera, t5 small to base, similarly for flan-t5). pretraining methods have some impact as well: t5 and flan-t5 do not perform identically despite an identical model structure and comparable sizes, and gpt-4 clearly outperforms all models in this case, including the bespoke ones. we proposed and validated a straightforward inference time method to improve model synthesis capabilities by preferentially outputting summary candidates that align with a predicted aggregate measure, and demonstrated empirically that this offers gains in performance. these gains are primarily limited by the underlying models' behaviors, but potentially bring performance on these single, task-specific metrics, on par to human performance, when the model is capable of providing a response that aligns with the proxy metrics. we hope this work encourages additional research into summarization models that explicitly optimize to accurately synthesize potentially conflicting evidence. we are particularly interested in understanding why models fail to synthesize-they clearly learn to produce synthesis-like text, but fail to yield the best option, even among their top candidates. we use summary reranking as a means to surface these more-appropriate summaries, but this is solely post-hoc as opposed to controlling for a more suitable generation, or ideally improving base model performance. our methods focus solely on improving performance at single specific task measures, potentially at a cost to other review qualities. users of such systems may have auxiliary goals, perhaps requiring multiple measures of synthesis quality, other measures of overall review quality, or a greater (or lesser) willingness to abstain. abstinence can be a feature beyond the case of systematic reviews; systems may have other specific rules for when to abstain: e.g. toxic language, challenging to verify statements, or distance from an overall objective (i.e. abstaining in the movie reviews case). this work has several limitations. we have made an effort to fine-tune several popular summarization models, but limited our analysis to models of relatively modest size (due to the gpu memory required to train long sequence summarization models). these behaviors appear to change with larger models (e.g., the small vs base-sized models, gpt-4 (openai, 2023)), but building robustness to perturbations while maintaining sensitivity to input composition is a non-obvious challenge. we also have reported results on only english-language tasks. finally, we focused on a relatively narrow behavior (synthesis of a single aspect); models may succeed in this respect while failing in other ways.