Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations (2023)

Abstract: Evaluating multi-document summarization (MDS) quality is difficult. This is especially true in the case of MDS for biomedical literature reviews, where models must synthesize contradicting evidence reported across different documents. Prior work has shown that rather than performing the task, models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. Better automated evaluation metrics are needed, but few resources exist to assess metrics when they are proposed. Therefore, we introduce a dataset of human-assessed summary quality facets and pairwise preferences to encourage and support the development of better automated evaluation methods for literature review MDS. We take advantage of community submissions to the Multi-document Summarization for Literature Review (MSLR) shared task to compile a diverse and representative sample of generated summaries. We analyze how automated summarization evaluation metrics correlate with lexical features of generated summaries, to other automated metrics including several we propose in this work, and to aspects of human-assessed summary quality. We find that not only do automated metrics fail to capture aspects of quality as assessed by humans, in many cases the system rankings produced by these metrics are anti-correlated with rankings according to human annotators.

## introduction multi-document summarization (mds) requires models to summarize key points across a set of related documents. variants of this task have drawn significant attention in recent years, with the introduction of datasets in domains like newswire @xcite , wikipedia @xcite , science @xcite , medical literature reviews (deyoung et al., 2021; wallace et al., 1 dataset and analysis are available at https://github.com/ allenai/mslr-annotated-dataset . rankings from commonly reported automated metrics like rouge and bertscore are not correlated or anticorrelated with human-assessed system rankings (d). 2020), and law @xcite ; and substantial methodological work to design model architectures tailored to this task @xcite @xcite . in this work, we focus on mds for literature reviews (mslr), a challenging variant of the task in which one attempts to synthesize all evidence on a given topic. when manually performed, such reviews usually take teams of experts many months to complete. good review summaries aggregate the results of different studies into a coherent passage, while the evidence presented in the input studies will often be in conflict @xcite @xcite , complicat-ing the synthesis task. 2 evaluating conditional text generation models is notoriously difficult, impeding progress in the field. prior work on summarization evaluation has proposed various lexical and modeling-based approaches to assess generation quality, but these metrics predominately use correlation with humanassessed quality facets over relatively small numbers of examples to demonstrate utility @xcite @xcite . this limitation of current metric evaluation implies that existing automated measures may not generalize well. further, evaluation in the multi-document setting adds additional complexity, e.g., prior work has shown that mds models may sometimes exploit shortcuts that do not reflect as detectable changes in automated metrics @xcite . to address these challenges, we collect human annotations to evaluate current models and to support automated metrics development for the medical mds task. we construct a dataset of such evaluations using public submissions from the 2022 mslr shared task on literature review mds. 3 selecting top-performing models, we label the summary quality of a sample of these models' outputs on the cochrane subtask @xcite . as part of our analysis, we compare system rankings produced by automated metrics and human evaluations. strikingly, our results highlight consistent and significant disagreements between automated metrics and humans, motivating the need for better automated evaluation metrics in this domain. we contribute the following: ## background the mslr shared task was introduced to bring attention to the challenging task of mds for literature reviews. the shared task comprised two subtasks, based on the cochrane @xcite and msˆ2 @xcite datasets. the cochrane dataset consists of 4.6k reviews from the cochrane database of systematic reviews. inputs are abstracts of papers cited by the review and target summaries are the authors' conclusions subsections of review abstracts. the msˆ2 dataset includes 20k reviews and is semi-automatically constructed from biomedical literature reviews indexed by pubmed. we refer the reader to the original publications for details concerning dataset construction @xcite . shared task organizers provided training and validation splits for both datasets, and solicited model submissions to two public leaderboards, where models were evaluated on a hidden test split. models were ranked on the leaderboard using rouge (-1, -2, -l; lin 2004), bertscore @xcite , and delta-ei @xcite , a metric based on evidence inference @xcite classifications. ## dataset we construct our dataset from system submissions to the cochrane subtask leaderboard for the 2022 mslr shared task (provided to us by task organizers). we only sample from the cochrane subtask due to the greater number and variety of successful submissions. we include all summaries from the leaderboard, though we only perform human evaluation on summaries generated by 6 models (discussion in §3.1). we define and apply two human evaluation protocols to a sample of summaries from these 6 systems. the first ( §3.2) is a facet-based evaluation derived from the analysis conducted in @xcite and the second ( §3.3) is a pairwise preference assessment. ## analysis of generated summaries we perform lexical analysis of input abstracts, system generated summaries, and target summaries in our dataset, summarizing our findings below. input copying and synthesis to assess similarity between inputs and summaries, we first apply the evidence inference pipeline (lehman et al., 2019; deyoung et al., 2020) 6 to identify an evidence statement in each input document and classify it with an effect direction. between each input evidence statement and the target and generated summaries, we compute rouge-1 scores. we compute the synthesis rate as how often the effect direction agrees between the most similar evidence statement (by rouge-1 score) and the generated summary. in to determine the origin of these long n-grams, we calculate their overlap with summaries in the train set and their corresponding input documents. while overlap with inputs is nearly zero, up to 90% of long n-grams are also found in train set summaries (figure 6 in app. c). interestingly, models with global attention (led or primera-based) seem to replicate more long sequences from the train set summaries than bart-based ones, while in the pegasus-based system (scispace) a smaller amount of self-repetition can be explained by fine-tuning. finally, we observe that though the distributions of self-repeating n-grams in the target summaries of the test set and train set are very similar (figure 3 ; left), in generated summaries the rate of self-repetition increases up to 500x compared to occurrence in the train set summaries (figure 3 ; right). models amplify repeating patterns from the train set to unnatural proportions! ## automated evaluation metrics we compute automated metrics for each generated summary and include instance-level scores in our dataset. we investigate how these metrics correlate with other metrics ( §5.1) and with human evaluation facets ( §5.2). metrics from the mslr leaderboard: rouge: the leaderboard reported system-level rouge-1, rouge-2, and rouge-l f-scores @xcite . we report these same three metrics; in some plots, due to space constraints, we show the average of these three rouge metrics, which we call avg-rouge-f. bertscore: we compute and report bertscore-f @xcite for each generated summary as computed using the roberta-large model. ## comparing system rankings evaluation metrics for summarization can be used in two settings, to judge performance at the instance level (comparing individual summaries) or at the system level (comparing model performance over many instances). here, we compare systemlevel rankings produced by automated metrics, human facet evaluation, and pairwise preference annotations to determine whether automated metrics effectively rank systems as humans would. system rankings are computed by averaging the instance-level metric values or scores across all review instances for each system, and ranking from best to worst average score (direction depends on metric; higher is better for all scores except delta-ei). we only average metrics over the subset of reviews for which we have human annotations. this ensures a fair comparison in the circumstance where we have selected an annotation sample that a system performs particularly well or poorly on. by doing this, the system rankings we present here are different than those computed using the same metrics from the mslr leaderboards. we do not intend our computed rankings to be interpreted as the true system ranking; our analysis focuses on whether automated metrics and human evaluation are able to produce similar rankings of systems. rankings by automated metrics are not correlated with rankings by human evaluation in general, system rankings from commonly reported automated metrics are not correlated or anti-correlated (lighter blue) with system rankings produced by human judgments. system rankings from automated metrics are highly correlated among themselves (ρ close to 1), aside from pio-overlap. pio-overlap rankings are strongly correlated with rankings from human pio agreement. pio-overlap and delta-ei ranks also correlate with the combined pairwise rankings, again suggesting that these two metrics may be the most promising for capturing human notions of summary quality. ## related work beyond rouge @xcite and bertscore @xcite , an extensive list of n-gram @xcite and model-based @xcite @xcite @xcite summarization evaluation metrics have been proposed in the literature. in particular, model-based approaches that use question generation and question answering @xcite @xcite or nlibased models @xcite have been proposed to assess summary factual consistency. @xcite and @xcite provide more thorough evaluations of many of these metrics on select summarization tasks. we perform evaluations using metrics previously reported on the mslr task, and leave a systematic evaluation of metrics on this task and others to future work. in @xcite , the authors performed fact verification on generated radiology reports using an information extraction module, by aligning the extracted entities with entities found in the reference summary. our pio-overlap metric similarly uses a pio entity extraction module to assess concept overlap between generated and reference summaries. @xcite proposed to use nli models to rank summaries by average entailment score per sentence against the input documents; this shares similarities with the delta-ei score we evaluated, which attempts to quantify agreement relative to the reference summary with respect to the direction of evidence reported. @xcite investigated system-level rankings produced by automated metrics and human evaluation and found minimal correlation between them, a finding corroborated by our work. @xcite introduced the robust summarization evaluation (rose) benchmark, containing human judgments for system outputs on the cnn/dm, xsum, and samsum datasets. we extend such work into a novel domain (medical mds for literature review) and demonstrate differences in automated metric performance and hu-man evaluation in our domain and task. for example, though rouge correlates with human preferences in single-document (cnn/dm) and multidocument (multinews) news summarization, we find that it is poorly correlated with human judgments and preferences in the mslr task. recent developments in large language modeling have also shifted the goalposts for evaluation. @xcite found that although humans overwhelmingly prefer zero-shot gpt-3 summaries for news summarization, automated metrics were unable to capture this preference; they introduced a benchmark of human judgments and rationales comparing system outputs on the singledocument news summarization task. more recently, @xcite demonstrated that gpt-3 can be adapted for the mslr task, and though the model outputs are generally found by human annotators to be faithful to the inputs, in the mds setting the evidence direction often disagrees with the reference. detecting these disagreements and developing automated metrics that can capture such disagreements are valuable pursuits and one of the motivations for our work. further investigation into whether automated metrics developed using limited human evaluation benchmarks such as the dataset we introduce here will be a goal for future work. ## discussion mds for literature review may involve notions of summary quality not readily captured by standard summarization evaluation metrics. for example, our lexical analysis of generated summaries reveals a concerning level of self-repetition behavior, which is not penalized by standard metrics. through two independent human evaluations (facetbased and pairwise preferences), we also show that automated metrics such as rouge and bert-score are poorly correlated or even anti-correlated with human-assessed quality. this is not to say that these metrics do not provide any utility. rather, further work is needed to understand what aspects of summary quality these metrics capture, and how to use them in combination with other metrics, novel metrics yet unintroduced, as well as human evaluation to better assess progress. we note that ours is not a systematic analysis of all automated summarization evaluation metrics, but is a focused study on evaluation metrics reported for the mslr shared task and which we introduce under the hypothesis that they may be useful for capturing some quality facets associated with this task. for those interested in the former, please refer to studies such as @xcite or @xcite . a positive finding from our work is the promise of the pio-overlap and delta-ei metrics. delta-ei shows some potential to capture evidence directional agreement between summaries, though the metric as currently implemented is noisy and does not cleanly separate summaries that agree and disagree on direction. pio-overlap, a metric we introduce, correlates with human-assessed pio agreement, suggesting that it could be a performant, scalable alternative to human evaluation of this quality facet. still, more work is needed to probe how variants of these metrics could be adapted to evaluate performance on mslr and other mds tasks. finally, we note that human evaluation is difficult because people value different qualities in summaries. the rank-based analysis we perform does not account for interactions between related quality facets and is unable to elicit relationships between overall quality and individual quality facets. the majority of pairwise preference annotations in our dataset also include short free text justifications for preference decisions, which could be used to further study this problem. other promising directions for future work involve studying how to optimally elicit human preferences, such as how to sample instances for labeling to maximize our confidence in the resulting system-level rankings. ## conclusions there have been major recent advances in the generative capabilities of large language models. models like chatgpt, 8 @xcite , and pubmedgpt foot_7 demonstrate aptitude on many tasks but have also been shown to confidently produce factually incorrect outputs in specialized and technical domains. 10 medicine is a specialized domain where incorrect information in generated outputs is difficult to identify and has the potential to do harm. there is therefore a pressing need for the community to develop better methods to assess the quality and suitability of generated medical texts. our investigation confirms that there is significant room for improvement on medical mds evalua-tion. we hope that the resources and findings we contribute in this work can assist the community towards this goal.