D O S UMMARIZATION M ODELS S YNTHESIZE ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Multi-document summarization entails producing concise synopses of collections of inputs.
For some applications, the synopsis should accurately synthesize inputs with respect to a key property or aspect.
For example, a synopsis of film reviews all written about a particular movie should reflect the average critic consensus.
As a more consequential example, consider narrative summaries that accompany biomedical systematic reviews of clinical trial results.
These narratives should fairly summarize the potentially conflicting results from individual trials.
In this paper we ask: To what extent do modern multi-document summarization models implicitly perform this type of synthesis?
To assess this we perform a suite of experiments that probe the degree to which conditional generation models trained for summarization using standard methods yield outputs that appropriately synthesize inputs.
We find that existing models do partially perform synthesis, but do so imperfectly.
In particular, they are over-sensitive to changes in input ordering and under-sensitive to changes in input compositions (e.g., the ratio of positive to negative movie reviews).
We propose a simple, general method for improving model synthesis capabilities by generating an explicitly diverse set of candidate outputs, and then selecting from these the string best aligned with the expected aggregate measure for the inputs, or abstaining when the model produces no good candidate.
This approach improves model synthesis performance.
Our hope is that by highlighting the need for synthesis (in some summarization settings), this work motivates further research into multi-document summarization methods and learning objectives that explicitly account for the need to synthesize.
I NTRODUCTION
Multi-document summarization (MDS) models aim to distill inputs into concise synopses that preserve key content.
Examples of MDS include summarizing news articles (Dang, 2005; Fabbri et al., 2019; Ghalandari et al., 2020;
Evans et al., 2004), answering questions from multiple sources (Dang, 2006), and producing overviews of scientific literature (Liu et al., 2018;
Lu et al., 2020; Moll´a &
Santiago-Mart ´ ınez, 2012; Wallace et al., 2020; DeYoung et al., 2021).
We expect summarization models to produce outputs consistent with inputs (Kryscinski et al., 2020; Nan et al., 2021a), e.g., discussing the same types of entities (Nan et al., 2021b) and allowing one to answer questions similar in a way that is consistent with individual inputs (Wang et al., 2020a;
Scialom et al., 2021).
In some applications models must synthesize inputs—i.e., aggregate potentially conflicting information—to yield an accurate synopsis (Figure 1).
As a simple example, consider the metareviews of movies featured on Rotten Tomatoes, 1 which provide a consensus view of individual critic opinions.
These reviews should therefore reflect the mean and range of sentiment implicit in the input critiques: A summary of mostly negative reviews (e.g., Gigli ) should communicate that the film was widely panned; a summary of mixed reviews (as in the case of The Fifth Element ) ought to convey that critics disagreed and discuss the main positive and negative attributes.
A more consequential example is the task of summarizing the evidence presented in clinical trials.
Individual trials will frequently present conflicting evidence about whether or not a particular health intervention is effective.
An ideal summary of the evidence would appropriately weigh the findings presented in the constituent inputs and reflect the evidence on balance.
[Figure]
What are the desiderata of multi-document synthesis ?
First, summaries produced by models should be consistent with the input data, with respect to the latent property of interest.
In the case of Rotten
Tomatoes, the sentiment of the summary should be in line with the aggregate sentiment expressed in the individual critic reviews.
A corollary to this is that models should be sensitive to changes in the composition of inputs, e.g., removing most of the negative reviews from a set of inputs should yield a summary with a corresponding increase in the expressed sentiment.
In this work we evaluate neural MDS models with respect to these criteria.
To this end we use a meta-reviews dataset from Rotten Tomatoes (Leone, 2020) and a dataset of systematic reviews (meta-analyses) summarizing the evidence about medical interventions (Wallace et al., 2020).
For the former we probe the degree to which generated meta-review sentiment agrees with the expected aggregate sentiment score; for the latter we evaluate whether the generated summary indicates that the input evidence suggests, on balance, that the intervention under consideration was effective.
Our main contributions are summarized as follows.
(1) To the best of our knowledge, this is the first work to investigate implicit synthesis in summarization, and the degree to which modern models are capable of this.
We show that “off-the-shelf” neural MDS models are somewhat inconsistent and insensitive with respect to performing synthesis in summarization.
(3) We propose and evaluate a simple and general technique which involves generating a diverse set of output candidates (Vijayakumar et al., 2016) and then selecting from these on the basis of agreement with an expected aggregate measure (based on inputs), with promising results.
S YNTHESIS AND S UMMARIZATION
In standard multi-document summarization, we assume inputs ( X i , y i ) , where X i
= { x i 1 , ..., x i |
X i | } .
We then typically train a summarization model with parameters θ , to consume X i and yield summaries ˆ y i as similar as possible to targets y i .
More precisely, the standard objective entails finding estimates for θ which maximize target token log-probabilities.
Assuming the input documents x ij in X i have been linearized (i.e., concatenated, usually with adjoining special tokens to demarcate individual inputs) into a string x
⊕ i of input tokens, this objective takes the form: (cid:80) |
y i | t =1 log p θ ( y it | y i 1 , ..., y i ( t − 1) , x ⊕ i ) , where p θ is a probability assigned to the token at position t in the (linearized) target x ⊕ i by a summarization model with parameters θ .
By myopically focusing on encouraging the model to produce tokens that mimic the targets, this objective aligns with standard (but flawed) measures of automated summary quality like ROUGE (Lin, 2004), which quantify n -gram overlap between targets y i and outputs ˆ y i .
We are interested in settings in which there is an additional, latent property implicit in the constituent input texts x ij , z ij .
For example, z ij might reflect the sentiment in critique j of the film indexed by i .
Summaries should synthesize this aspect, i.e., the generated summary ˆ y i should implicitly convey an aggregated z i which reflects a synthesis or aggregation G over Z i
= { z i 1 , ...
z i | X i | } .
That is, we assume z
i =
G ( Z i ) .
In both cases considered here—summaries of film critiques and synopses of clinical trials evidence— G can reasonably be assumed to be a (weighted) mean,
G ( Z i ) =
| X i | (cid:80) |
X i | j =
α ij z ij .
That is, summaries should roughly reflect the average sentiment and reported treatment effect in the cases of movie reviews and clinical trial reports, respectively.
[Table]
We investigate the following questions.
(1) Do model summaries
ˆ y i reflect the anticipated aggregate aspect of interest?
That is, how well calibrated is the aspect communicated in the generated summary ( z i ˆ y ) compared to the expected z i ?
(2) Can we improve the ability of summarization models to synthesize by explicitly incorporating synthesis targets
z i into the decoding process?
We propose a simple inference-time procedure to explicitly preference output candidates that align with the expected aggregate property of interest (e.g., average sentiment)
[Table]
and report promising results for the approach.
This strategy also naturally lends itself to cautious summarization, i
[Table]
e., approaches in which we allow the model to abstain from generating an output if it does not produce any candidates that reflect the anticipated aggregate measure.
D ATASETS AND M EASUREMENTS 3. M OVIE R EVIEWS
We first consider a dataset comprising movie reviews and associated meta-reviews summarizing these from Rotten Tomatoes.
An in-house staffer summarizes audience reviews 3 into meta-reviews.
These meta-reviews synthesize the constituent input reviews, and reflect the aggregate critic reception of a film.
Each meta-review is associated with a numerical “Tomatometer” score, which is an overall measure of what percent reviews were positive for the corresponding film ( G then is an average of the positive indicator per review).
The Rotten Tomatoes dataset we use comprisesmovies with meta-reviews constructed from 244,000 individual reviews (Table 1).
Measuring sentiment in movie reviews.
As our measure g we train a BERT model (Devlin et al., 2019) using the continuous (fine-grained) sentiment targets provided in the SST dataset (Socher et al., 2013).
4 We trained this model for 3 epochs using a learning rate of 5e-5 using the
Huggingface library 5 with no hyperparameter tuning.
While the raw text of the SST dataset is in-domain, the targets themselves are not.
We find a reasonably strong correlation between our sentiment estimates and the “true” meta-review sentiment (“Tomatometer” score): The R 2 (centered) is 0.696, mean squared error (MSE) of 0.022, and Pearson’s r of 0.836 (Figure 2, upper left).
3. B IOMEDICAL S YSTEMATIC R EVIEWS OF T REATMENTS
Our second dataset is a collection of systematic reviews from the Cochrane Collaboration.
This dataset comprises roughly 2600 systematic reviews summarizing a total of 16,500 clinical trials evaluating interventions in healthcare (Table 1).
Each review includes both a natural language sum-
substantial
[Figure]
t
[Figure]
S
[Figure]
n t i m e n t mary and accompanying statistical meta-analysis results.
The latter provides an aggregate statistical summary of the individual (study-level) data extracted from the trials included in each review.
The natural language summary should accurately convey and contextualize the findings of the metaanalysis.
Therefore, the (lack of) treatment efficacy communicated in a given summary should generally agree with the direction of the corresponding meta-analytic point estimate.
Measuring effects in evidence syntheses For systematic reviews of clinical trials, we resort to a less granular classification model g ( x ij ) , g ( y i ) which attempts to infer whether a given piece of text reports a significant result or not.
In particular we use RobotReviewer (Marshall et al., 2017; DeYoung et al., 2020).
Given a narrative describing a clinical trial result (or a systematic review summary of such results), RobotReviewer predicts whether the reported result indicates a significant effect of the treatment being investigated, or not.
We can compare this prediction to the “truth”, which here is derived from the meta-analytic result (specifically by checking whether p < 0 . 05 ).
Applying this off-the-shelf model to the manually composed summaries accompanying the meta-analyses in our Cochrane set, we observe a macro-average F1 score of 0.577 (Table 10,
Appendix D), providing a reasonable (if weak) measure for this task.
M ODELS
We evaluate a suite of transformer (Vaswani et al., 2017) summarization models: Longformer (Beltagy et al., 2020), Pegasus (Zhang et al., 2020), PRIMERA (Xiao et al., 2021), and T5 (Raffel et al., 2020).
PRIMERA was designed and pre-trained specifically for multi-document summarization specifically.
And while not explicitly designed as multi-document summarization models, both Pegasus Zhang et al. (2020) and T5 7 have been used on multi-document tasks, while Longformer has been used for a related multi-document summarization task (DeYoung et al., 2021).
For all models we mostly use hyperparameters defaulted to in their respective huggingface implementations.
We conduct a hyperparameter sweep over optimization steps and learning rate, selecting the best model by ROUGE1 performance on the dev set (Appendix C, Tables 8. 9).
E XPERIMENTS 5. H OW WELL DO SUMMARIZATION MODELS SYNTHESIZE ?
We report sentiment performance for all models in Table 2.
These are metrics quantifying the strength of the relationship between (a) the continuous sentiment inferred (via our text-regression 7 https://huggingface.co/osama7/t5-summarization-multinews
[Figure]
[Figure]
g ) over model generated or reference (human written) summaries and (b) the reference sentiment (Tomatometer) score.
Across these metrics, correlations between the sentiment measured in model generated outputs and the Tomatometer score are considerably lower than that between the same measurement over human-composed summaries and said score.
Based on these metrics, human authors do a better job of synthesis than the models when composing their summaries.
For systematic reviews (Section 3.2), we are able to measure g whether a text appears to report significant treatment effect or not, and we can compare this against the p -value from the corresponding statistical meta-analysis.
This permits only a coarse assessment of synthesis, as we are unable to measure correlations.
Instead we report classification metrics describing how often the effect significance inferred from a summary (generated or manually written) matches the ground truth derived from the meta-analysis (Table 2).
The results are qualitatively similar to the sentiment case, in that the humans appear to do a better job of synthesis — as best we can measure, the significance reported in their summaries better aligns with the statistical results than in model generated summaries.
5. S ENSITIVITY TO I NPUT O RDERING
Synthesis of inputs should be invariant to ordering (e.g., the critics’ consensus on a film does not depend on the order in which one reads the reviews).
Here we evaluate if models are sensitive to input orderings with respect to the synthesized aspect of interest ( z i ˆ y ) in the resultant outputs.
Specifically, X i =
{ x i 1 , ..., x i |
X i | } will constitute an arbitrary ordering of inputs reflected in the linearized version x ⊕ i .
This ordering should not affect the aggregate aspect
z i ˆ y in the summary.
To evaluate if models realize this invariance, we permute the instance i inputs X i (and, consequently, the linearized x ⊕ i ) one hundred times, randomizing input orderings.
For each such permutation ˜ X i (and associated ˜ x ⊕ i ), we generate a summary ˆ y i and estimate of the resultant aspect ˜
z
i ˆ y , using the corresponding measurement model.
By repeating this process for each instance i , we can construct an empirical distribution over ˜ z
i ˆ y ’s under different random orderings.
Movie reviews.
We zero-mean the ˜ z
i ˆ y ’s inferred over each instance, and combine the distributions from all instances into a histogram (Figure 3 left).
This shows the spread of sentiments inferred over outputs under random input orderings minus the corresponding instance mean sentiment.
Were
[Table]
a model completely invariant to ordering, the empirical distribution over these differences would collapse to 0.
Instead, we observe a relatively wide spread in the sentiment measured over outputs generated from different permutations, indicating a counter-intuitive sensitivity to orderings.
Systematic reviews .
For each X i we have 100 order permutations and associated summaries; we infer whether these report significant results or not, and record the fraction that do ( p i ).
If models were invariant to ordering, this fraction would always be 0 or 1.
Values in-between suggest the model flips the report conclusion as a result of different input orderings.
We calculate the entropy of p i to quantify this.
Figure 3 (right) shows a histogram of these entropies calculated over the subset of examples where the associated meta-analysis indicates a significant effect.
9 Densities away from zero indicate sensitivity to ordering.
5. S ENSITIVITY TO I NPUT C OMPOSITION
Synthesis models should be responsive to changes in the distribution of the attribute to be synthesized in the input composition: If we increase the ratio of positive to negative reviews in an input set, we would anticipate a concomitant change in the sentiment communicated in the meta-review z i ˆ y .
To assess if models meet this synthesis desiderata, we manipulate model inputs X i in such a way to induce an expected change in the target measure
z i ˆ y ; we then measure if the output yields a summary that aligns with this expected change.
Movie reviews .
We manipulate the ratio of positive to negative reviews and observe the resultant change in the property of interest latent in the corresponding output.
We take movies with mixed reviews, and delete 10%, 20%, 30%, ..., 100% of the positive inputs, retaining the negative inputs; we then repeat the process but instead remove negative inputs.
For each of these permutations, we measure the input sentiment, the meta-review sentiment, and how well they correlate (Table 3).
Figure 4 plots the relationship between the fraction of positive reviews in the (manipulated) input sets and the granular sentiment score inferred over the resultant outputs.
The models are generally undersensitive to changes in their input: rather than having a change in meta-review sentiment equivalent in size to changes in input sentiment (a slope of 1, as we observe when we fit a model to the human written summaries).
Models tend to have trouble changing their sentiment, and require a large change in input distribution to substantially change the sentiment communicated in the output.
These are the more interesting entire dataset in Appendix Figure 9.
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
+JFw+KePWnePPfuG1z0NYHA4/3ZpiZFySCa3Ccb6u0tr6xuVXeruzs7u1X7YPDro5TRVmHxiJWvYBoJrhkHeAgWC9RjESBYA/B5GbmPzwypXks7yFLmBeRkeQhpwSM5NvVUX0wJpBnUz/nYnrm2zWn4cyBV4lbkBoq0Pbtr8EwpmnEJFBBtO67TgJeThRwKti0Mkg1SwidkBHrGypJxLSXzw+f4lOjDHEYK1MS8Fz9PZGTSOssCkxnRGCsl72Z+J/XTyG88nIukxSYpItFYSowxHiWAh5yxSiIzBBCFTe3YjomilAwWVVMCO7yy6uk22y4543m3UWtdV3EUUbH6ATVkYsuUQvdojbqIIpS9Ixe0Zv1ZL1Y79bHorVkFTNH6A+szx/Uo5Mz</latexit>
Systematic Reviews .
To measure sensitivity to changes in input composition, we manipulate our inputs
X i such that the meta-analysis result (target z i ˆ y ) flips from a significant effect to no effect, or from no effect to an effect.
Operationally, we do this by first taking of a subset of the reviews that have conflicting evidence (yielding 139 unique reviews).
We then order inputs in these by (weighted) effect sizes, 10 and remove subsets which ought to flip the significance result.
I MPROVING S YNTHESIS IN S UMMARIZATION
We propose a simple post-hoc approach to improving the synthesis performed by multi-document summarization models.
This involves the following steps: (1) Generate an explicitly diverse set of output candidates 11 ; (2) Select from these as the final output the candidate that best agrees with the expected synthesis result (as predicted by an external model).
For (1), we rely on a previously proposed technique for generating diverse outputs
C i from input x ⊕ i , namely Diverse Beam Search (DBS)
(Vijayakumar et al., 2016).
This method modifies standard beam search to maintain multiple groups of beams.
During decoding, a term is added to the next-token log probabilities which effectively penalizes production of (partial) strings similar to candidates on beams in other groups.
In (2) we would like to select the output that best synthesizes the property of interest; this requires a mechanism for specifying what we expect the synthesized property be, given the inputs.
For example, if we know the sentiment scores associated with input movie reviews, we might enforce that the sentiment expressed in the output agrees with the average of these.
To realize this intuition, we can select as final output from C i the string that best aligns with this anticipated aggregate property (sentiment score or significance finding).
Operationally, this requires an external model to measure—or estimate—the aspect of interest as latent in a given candidate output.
This is a limitation of the approach, but in many settings it may be feasible to identify or construct a model; we were able to do so for both tasks considered in this paper.
There is no guarantee that any member of C i will align well with the anticipated aggregated property.
In such cases, we have no means of yielding an output consistent with respect to synthesis, and it may be desirable to abstain from outputting anything at all in such cases; that is, to be a cautious summarizer (Ferri et al., 2004; Hechtlinger et al., 2018).
We consider this strategy in the case of generating narrative synopses of evidence, as this constitutes a case in which (a) one would very analysis above.
[Figure] [Figure]
much prefer not to produce a misleading summary of clinical evidence (Kell et al., 2021), and, (b) we observe many cases where the diverse decoding strategy yields an output that seems to communicate (at a granular level) the aggregate findings expected.
Movie Reviews For movie reviews we use a BERT (Devlin et al., 2019) model trained on IMDB (Maas et al., 2011) 14 to predict the sentiment of each input x ij , using the proportion of x ij ∈ X i with a positive score as an approximation for the target sentiment z
i ˆ y .
For each diverse prediction C i , we predict a sentiment ˜
z
i ˆ y using our sentiment regression model (Section 3.1), and select the prediction cloest to the estimated target sentiment | ˜
z i ˆ y − z i ˆ y | .
We find this improves model performance to human-like levels in terms of synthesis, as best we can measure (Table 4, Figure 6).
Systematic Reviews .
In the case of systematic reviews, we can have only a binary measure of significant effect (or not).
As a proxy for z
i ˆ y , we again use RobotReviewer to extract an effect for each of the model inputs x ij , using the majority vote (i.e., do the plurality of x ij ∈
X
i indicate that there was an effect).
We classify each output candidate in C i again using RobotReviewer to estimate
˜ z
i ˆ y .
We then select for output the highest probability candidate in C i which agrees with the majority vote of the inputs, and abstain where there are no viable candidates.
For the models we do choose a summary for, we find performance similar to our measure (Table 5).
Movie reviews show a wide range of sentiments; systematic reviews show some improvement but are biased towards no effect (qualitatively observed in Appendix G).
R ELATED W ORK
Automatic (multi-document) summarization (Nenkova & McKeown, 2011; Maybury, 1999) has been an active subfield within NLP for decades.
We have focused our analysis on modern, neural abstractive models for conditional text generation (Bahdanau et al., 2015).
In light of their empirical
[Table] [Table]
success, we have specifically evaluated a recently et et et al., 2021; et al There has been some work on highlighting conflicting evidence in health literature specifically (Shah et al., 2021b;a), though this was focused primarily on highlighting conflicting evidence, and explicitly aggregating extracted content.
Sentence fusion One view on synthesis might be that is a particular kind of sentence fusion (Barzilay & McKeown, 2005).
However, past work on “fusing” sentences has assumed that the aim is to generate an output that contains the information common to similar sentences (Thadani & McKeown, 2013).
This is intuitive in the context of, e.g., summarizing multiple news articles covering the same event.
But here we are interested in the more challenging setting in which the output should reflect an aggregate measure of potentially conflicting evidence or opinions.
Interpretation and analysis of neural models for NLP This work is also related to the emerging body of work on analyzing neural NLP models, their behaviors, “knowledge”, and “abilities” in general e.g., Linzen et al.
; Tenney et al.
; Petroni et al.
; Niven & Kao (2019);
Meng et al.
There has been some work specifically on analyzing neural summarization models.
Xu et al.
(2020a) investigated when a model is likely to extract (copy) rather than abstract (generate).
Xu & Durrett (2021) furthered this analysis by assessing when models were relying on the local input to produce particular output tokens, and when they instead rely on mostly on a background language distribution acquired in pre-training.
Factuality of neural summarizers Neural conditional generation models have proven adept at producing fluent outputs, but in the context of summarization they are prone to hallucinating content unsupported by input documents (Maynez et al., 2020; Kryscinski et al., 2019).
Automated metrics such as ROUGE do not reliably capture such phenomena (Falke et al., 2019; Maynez et al., 2020).
This has motivated several efforts to design automated factuality metrics (e.g., Wang et al.
(2020b);
Xu et al. (2020b); see Pagnoni et al.
(2021) for an overview
C ONCLUSIONS
We have outlined and investigated the problem of synthesis as related to some summarization tasks.
We showed that existing models are partially able to synthesize implicitly, but do so imperfectly: For instance, the aggregation they perform is sensitive to input ordering, and they are not as sensitive to perturbations in the composition of inputs as one would hope.
We proposed and validated a straightforward inference time method to improve model synthesis capabilities by preferentially outputting summary candidates that align with a predicted aggregate measure, and demonstrated empirically that this offers gains in performance.
Our hope is that this work encourages additional research into summarization models that explicitly optimize to accurately synthesize potentially conflicting evidence and information.
A N OTATION
[Table]
The i th set of input documents, corresponding to instance i
[Table]
The target summary y of the i th instance
[Table]
The t th token of target y
i
[Table]
A generated summary of the i th instance
[Table]
The j th input document for instance
i
[Table]
A particular linearization of the input documents
X
i
[Table]
Model & parameters
[Table]
Probabily under parameters θ
[Table]
Standard auto-regressive prediction of the next token given an input and partial summary
[Table]
Latent property (sentiment, significance finding) of x ij
[Table]
Aggregated latent property (sentiment, significance finding) of X
i
[Table]
Latent property measured over summary ˆ y
[Table]
Aggregation function over latent properties z ij , yields z i
[Table]
Auxillary function to measure latent property z ij of x ij , or z i ˆ y of ˆ y
[Table]
ij
A weight for z ij
B A DDITIONAL R ELATED W ORK
Shah et al.
(2021a) created a ”nutri-bullets” system for generating consensus-based summaries of health and nutrition related content.
They assume a low-supervision setting in which one has a set of tuples extracted from texts with which to train a content extractor, and where one can design heuristic rule-based aggregation strategies on top of extracted tuples mapping onto discrete categories like “consensus”.
By contrast, we assume a more typical supervised summarization setting and are interested in continuous aggregation of a latent attribute of interest, and we do not assume (or have access to) relational tuples over inputs.
Indeed, recent work Wolhandler et al.
(2022) has shown that systematic reviews are categorically different than news summarization, and that relational tuple extractors do not perform well in the medical domain.
First, Shah et al.
(2021a)
’s focus primarily on settings in which training data is (severely) limited, and motivate their pipeline approach on the basis of this limited supervision assumption.
For this reason they define separate modules: The first performs content selection (tuple extraction; this does require manual annotations of tuples on a subset of texts to train such an extractor); The second applies (manually composed) deterministic aggregation rules over these extracted tuples to combine them; a final module then generates a “surface realization” conditioned on the aggregated result.
We have investigated more typical supervised settings (with thousands of input and summary pairs), and we are training modern end-to-end transformer-based summarization models.
We have empirically assessed the extent to which model outputs in this typical training regime are consistent with the continuous synthesis result anticipated.
We do not have annotated tuples on our inputs (as would be required to use the Shah et al.
(2021a) approach, as it assumes a trained content extractor module).
And while applying discrete (manually composed) aggregation operators over inputs makes sense in some settings, we are explicitly interested in the ability of models to aggregate variables of interest continuously, for example producing “very positive” summaries when movie reviews are overwhelmingly positive, and merely “positive” summaries when they are only mostly positive.
In sum, the approach proposed by Shah et al.
(2021a) is appropriate in, and designed for, lowsupervision settings (which we do not consider here) where there are natural “tuples” to be extracted from inputs and supervision for this sub-task (which we do not have) and where discrete aggregation categories of inputs is natural (whereas we are interested in continuous aggregation, e.g., mean sentiment).
Wolhandler et al.
(2022) attempts to measure how challenging multi-document summarization is, as a function of the unique knowledge (represented as relational tuples) required to produce a summary. Model HuggingfaceCheckpoint Optimizer
Schedule
Warmup
Smoothing LED allenai/led-baseAdam
P.nomial/decay0.
50steps 0.1 T t5-base
Adam
{+Linear 50steps 
+}
PRIMERA allenai/PRIMERA
Adam
{+Linear 50steps 
+}
Pegasus sshleifer/distill-pegasus-cnn-16Adafactor
Linear 500steps 0.
This work measures how many new tuples each input document might add in contrast to subsets of other inputs.
By greedily building subsets of inputs as a function of new information added, they find that standard multiple document summarization datasets merely need to select two to four documents from inputs of up to ten, whereas their approach breaks down in the case of systematic reviews.
They find that due to both technical constraints for relation extraction, as well as the inability to model contradiction, relational extraction and aggregation methods are insufficient for producing evidence syntheses.
Oved & Levy (2021) introduce the Perturb and Selection Summarizer (PASS) system for summarizing Amazon product reviews.
It works by perturbing model inputs (i.e. keep random subsets of the input), generating a summary for each perturbation (via standard beam search), and then selecting amongst outputs (via a ranker) to produce a coherent, self-consistent, and fluent summary.
PASS is similar to our work in that it generates multiple outputs and selects amongst them.
However it differs in several key respects.
The key conceptual difference between PASS and our work is that
PASS’s target is a summary’s self-consistency (a product review might contradict itself on some aspect, e.g. simultaneously discussing a product fitting well in addition to the product running a size small), whereas our target is a continuous fact derived from the inputs as a whole (e.g. aggregate sentiment or effect sizes). PASS is designed to produce summaries that are plausible, as opposed (and complementary) to summaries that reflect inherent contradiction in the input data. As PASS produces summaries from subsets of each instance’s input, it cannot perform an explicit synthesis on its own, as opposed to our work, wherein each summary was produced with access to the whole of each instance’s input. C M ODELS
We train all models using a modified Huggingface Transformers libraryWolf et al.
For the
Pegasus model, we use a distilled version provided by Huggingface (Table 7).
All models were trained using their default hyperparameters, except for batch size, optimization steps, learning rates, and any parameters specified in Table 7.
We fix our batch size to 16, using gradient accumulation over single instances at a time, with floating point 16 (fp16) precision (due to data size), and perform an approximate (subject to resource constraints) grid search over learning rates and training steps (Tables 8, 9), keeping the model highlighted in bold.
Earlier experimentation was performed ad-hoc with Longformer and T5 models only; we found that while lower numbers of steps could perform well, they had high variance and were more sensitive to hyperparameter changes than longer runs.
All training was performed on 48G NVIDIA RTX8000 GPUs, most models are unable to fit single instance gradient information into fewer than 40G, even at reduced precision.
D D ETAILED R ESULTS
Measure Validation As our results rely on using proxy metrics, we measure the quality of these proxies.
See Figure 8 for movie meta-review sentiment correlation with human results, and Table 10 for how well the automatic significance measures correlate with the underlying truth.
Diversity Sampling.
We include detailed results for the importance of diversity sampling; the diversity sampling procedure produces better metrics in every dimension (Table 11 top left vs. bottom left.).
In the systematic reviews, most metrics drop slightly and abstention increases substantially (Table 12).
model steps lr rouge rouge rougeL led 1e25.
led 1e25.
led 3e25.
led 5e25.
led 1e25. led 1e25.
led 3e24.
led 5e23.
led 1e24.
led 1e25.
led 3e25.
pegasus 1e23.
pegasus 1e22.
pegasus 1e18. pegasus 1e18.
pegasus 1e26. pegasus 1e26.
pegasus 1e24.
pegasus 1e23.
pegasus 1e24.
pegasus 1e27.
pegasus 1e25.
pegasus 1e23. pegasus 1e23.
pegasus 1e27.
pegasus 1e25.
pegasus 1e24.
primera 1e23.
primera 1e25.
primera 1e24. primera 1e24.
primera 1e25. primera 1e25.
primera 1e23.
primera 1e24.
primera 1e25.
t  1e25.
t  1e24.
t  1e22.
t  5e25.
t  1e25.
t  1e24.
t  1e23.
t  5e25.
t  1e25. t  1e25.
t  1e23.
[Table]
5e24.
[Table]
1e
[Table] [Table]
[Table]
1e
[Table] [Table]
5e
[Table] [Table]
1e
[Table] [Table]
[Table]
1e
[Table] [Table]
5e
[Table]
Model steps lr
ROUGE
ROUGE
ROUGEL led 5e24.
led 5e23.
led 5e25.
led 5e26.
led 5e30.
pegasus 1e20.
pegasus 1e21. pegasus 1e21.
pegasus 1e22. pegasus 1e22.
primera 1e23.
primera 1e22.
primera 1e26.
primera 1e27. primera 1e27.
primera 5e22.
primera 5e24. primera 5e24.
primera 5e27.
primera 5e28.
t  5e23. t  5e22.
t  5e0.
[Table] [Figure]
E R OUGE R ESULTS
We report mean differences in ROUGE outputs for both datasets in Figure 10.
Ideally, these would have all mass at zero.
[Table]
[Table]
[Table]
[Figure]
[Figure]
F E XAMPLES OF D IVERSE M OVIE S UMMARIES Summary Sentiment
The Private Lives of Pippa Lee relies on a strong ensemble cast to deliver witty and poignant observations about life and relationships. The Private Lives of Pippa Lee relies on a strong ensemble cast to deliver witty and poignant observations about life and relationships.
With a strong cast and Robin Wright Penn’s sharp performance, The Private
Lives of Pippa Lee succeeds as both a witty tribute to lost characters and a showcase for Robin
Wright Penn.
With a strong cast and Robin Wright Penn’s empathetic direction, The Private
Lives of Pippa Lee succeeds as both a humorous look at domestic issues and a poignant look at relationships.
The Private Lives of Pippa Lee benefits from Robin Wright Penn’s superb performance, as well as a strong ensemble cast that includes Keanu Reeves, and
Faye Dunaway.
The Private Lives of Pippa Lee has an affecting ensemble cast and Robin Wright
Penn delivers a noteworthy performance, although the film is a bit too episodic.
Summary
[Table]
You Don’t Mess With the Zohan’s handful of laughs are almost enough to compensate for its inconsistent tone and stale, obvious jokes.
[Table]
You Don’t Mess with the Zohan has a handful of crotch thrusts, but not enough of them land.
[Table]
You Don’t Mess With the Zohan’s handful of laughs are almost enough to compensate for its aimless, crass script.
[Table]
You Don’t Mess with the Zohan has its moments, but not all of them – and the jokes are embarrassingly crass and often crude.
[Table]
You Don’t Mess with the Zohan has its moments, but not all of them – and the jokes are embarrassingly crass and often crude.
The script
[Table]
G E XAMPLES OF D IVERSE S YSTEMATIC R EVIEWS Generated
Effect
Ketanserin versus placebo in the Raynaud’s phenomenon is neither effective nor safe.
The Raynaud’s phenomenon is associated with significant adverse effects including dizziness and pain.
The effectiveness of ketanserin for the Raynaud’s phenomenon is unknown.
no significant difference
Ketanserin versus placebo in the Raynaud’s phenomenon is neither effective nor safe.
The Raynaud’s phenomenon is associated with significant adverse effects including dizziness and pain.
no significant difference
Ketanserin and serotonin receptor antagonists in the Raynaud’s phenomenon treatment of systemic scleroderma reduce the incidence of ischaemic ulcers and may reduce the frequency of adverse events.
significant difference
The Raynaud’s phenomenon is associated with a small number of adverse effects when administered orally to patients with Raynaud’s phenomenon.
The frequency of Raynaud’s phenomenon is similar to that of other drugs.
However, there is little evidence to aid the treatment of
Raynaud’s phenomenon.
no significant difference
The Raynaud’s phenomenon is associated with a small number of adverse effects when administered orally to patients with Raynaud’s phenomenon.
The frequency of Raynaud’s phenomenon is similar to that of other drugs.
no significant difference
Generated
Effect
The overall evidence supports the use of topical antibiotics in surgical patients who have undergone minor surgery, compared to no treatment.
The effect on other outcomes, other than infection rate, is consistent.
The safety profile of topical antibiotics is also of concern.
Further well-designed RCTs are needed to assess effectiveness of topical antibiotics in surgical patients.
no significant difference
A single application of topical antibiotics in surgical site wounds reduces the risk of infection, and the risk of other complications, including wound dehiscence.
The risk of infection recurrence is low.
The use of topical antibiotics outside of surgery should be restricted to surgical site wounds.
no significant difference
A single application of topical antibiotics in surgical site wounds reduces the risk of infection, and the risk of other complications, including wound dehiscence.
The risk of infection recurrence is low.
no significant difference
The overall evidence supports the use of topical antibiotics in surgical patients to reduce the risk of infection, and the risk of other complications, especially in high-risk patients.
There is a lack of evidence in low-risk patients to support the use of topical antibiotics in this setting.
significant difference
A single application of topical antibiotics in surgical site infection prevention has been demonstrated to reduce the risk of infection in patients who have undergone surgery.
The number of patients who have been treated with topical antibiotics has been small but this is due to risk of bias in the trials.
Ointment use should be limited to patients whose primary wound is irradiated.
significantly difference