Overview of MSLR 2022: A Shared Task on Multi-document Summarization for Literature Reviews (2022)

Abstract: We provide an overview of the MSLR2022 shared task on multi-document summarization for literature reviews. The shared task was hosted at the Third Scholarly Document Processing (SDP) Workshop at COLING 2022. For this task, we provided data consisting of gold summaries extracted from review papers along with the groups of input abstracts that were synthesized into these summaries, split into two subtasks. In total, six teams participated, making 10 public submissions, 6 to the Cochrane subtask and 4 to the MSˆ2 subtask. The top scoring systems reported over 2 points ROUGE-L improvement on the Cochrane subtask, though performance improvements are not consistently reported across all automated evaluation metrics; qualitative examination of the results also suggests the inadequacy of current evaluation metrics for capturing factuality and consistency on this task. Significant work is needed to improve system performance, and more importantly, to develop better methods for automatically evaluating performance on this task.

## introduction systematic literature reviews aim to comprehensively summarize evidence from all available studies relevant to a research question. in medicine, such reviews constitute the highest quality evidence used to inform clinical care. reviews are expensive to produce manually, taking teams of experts months to years to complete, and go out of date quickly @xcite ; (semi-)automation may facilitate faster evidence synthesis without sacrificing rigor. toward this end, we initiated the mslr2022 shared task to investigate challenges in multi-document summarization and synthesis for medical literature review. in addition to soliciting direct submissions towards the task, we encouraged work extending our task/datasets, e.g., proposing scaffolding tasks, methods for model interpretability, and improved automated evaluation methods. we organized the task into two subtasks based on two datasets we provided: msˆ2 @xcite and cochrane @xcite . we received submissions and/or system reports from six participating groups. a selection of generated summaries from the final submissions will be sampled and subject to human annotation for quality and consistency against the gold summaries. the human annotations produced following the shared task will be released as a public dataset to encourage further work on this task and its associated automated evaluation metrics. in the rest of this overview, we provide descriptions of the shared task (section 2), the baseline models (section 3), submitted systems (section 4), and a summary of insights and directions for future work (section 5). ## task description we give a brief description of the datasets, task, evaluation metrics, and submission protocol for the shared task. ## baselines we provide several baseline models for comparison. baseline models from deyoung et al. ( 2021 ) are based on the bart @xcite and longformer @xcite architectures. for both subtasks, we report results of the two baseline models finetuned on the subtask dataset and evaluated on the corresponding subtask test set, as well as on the opposing test set (e.g. trained on msˆ2 and tested on cochrane and vice versa). for msˆ2, we also evaluate the condition of simply providing the background section as the generated summary. this baseline performs relatively well, indicating potential limitations of the chosen automated evaluated metrics as alluded to in @xcite . ## participating systems we provide brief descriptions of all participating systems. system performance as assessed using automated evaluation metrics are given in ittc @xcite the team adapted primera @xcite , a model based on longformer encoder-decoder @xcite ) that has been designed for multidocument summarization, resulting in strong performance on the mslr cochrane subtask. in addition to fine-tuning on the entire training sets of the mslr shared task, the team also experimented with zero-and few-shot learning scenarios. the authors found that rouge did not adequately capture the performance drops observed in the zeroand 10-shot settings, where factuality of the generated summaries was poor. the team also experiment with using global attention to highlight pico elements in the input and target texts. though rouge did not vary significantly between these two settings, the authors found that when pico spans are given global attention, the resulting summaries tended to be more abstractive. extract+bart-base @xcite the team explored how input selection strategies can improve the performance of a bart-base mode. the authors fined bart-base on the summarization dataset introduced by @xcite . they considered several extractive techniques to reduce the size of the input sequence, comparing text-rank, lexrank, and models for results extraction to select salient sentences from input documents. their results suggest that input sampling strategies are promising, though performance gains are inconsistent across the two mslr subtasks. puneict @xcite the team experimented with finetuning bart-large, distillbart, and t5-base for both the msˆ2 and cochrane subtasks. on the msˆ2 subtask, finetuned bart-large had the highest performance of the three models based on rouge score; on the cochrane subtask, distillbart performed best. scispace @xcite the team combined a bert-based extractive method with a big-bird pegasus-based abstractive summarization model @xcite , leading to strong performance on the mslr cochrane subtask. for the extractive step, the authors use a lecture summarizer model to identify the most important sentences from the input documents; this method encodes input sentences using bert, then clusters the contextual representations and selects the sentences closest to the cluster centroids. the resulting sentences are used as input into a bigbird pega-sus model pretrained on pubmed, which is finetuned on the mslr training data. in analysis, the authors observed that a common error is duplication of statements in the generated summary. the model submitted by the team to the cochrane subtask leaderboard performs best among submissions based on rouge-l, though the authors report that the same training strategy does not lead to good performance on the msˆ2 subtask due to the much longer input sequences in msˆ2. led-base-16k @xcite the team fine-tuned longformer encoder-decoder following a similar protocol to primera @xcite , improving performance over baselines in both subtasks. their input sequence included the titles and abstracts of up to 25 studies, separated by special tokens. no system description was submitted. ## insights & future directions though we observe modest overall improvements to task performance based on automated summarization evaluation metrics such as rouge and bertscore, results are inconsistent across evaluation metrics. this is especially the case when considering the evidence inference divergence metrics introduced to measure and bolster inference direction alignments between generated and gold summaries. further, several participant groups discovered problems with factuality, consistency, duplication, and more with generated summaries upon qualitative examination of their results @xcite . based on the observations of submitting teams, we summarize two key directions for future research. multidocument representation strategies several submissions explored methods for input extraction and filtering to reduce the size of the input sequence and increase the saliency of the input texts. for both subtasks, a large portion of input instances extend beyond even the token limits of long-sequence transformer language models, and this is especially the case for msˆ2 (the median number of input documents for msˆ2 is 17, nearly twice the number for the cochrane dataset). @xcite explored several strategies for sentence selection, including results extraction models, and found promising but inconsistent performance gains over a base model. @xcite employed a sentence embedding clustering and selection approach, which led to top performance on the cochrane subtask when combined with a powerful long-sequence trained summariza-tion model. however, @xcite noted that their methods did not extend well to msˆ2 due to the larger number of input documents. extension of such methods would be a promising future direction. beyond salient sentence selection, a strategy based on pico alignment and results extraction may be more pertinent for the specific task. for example, one may only want to include the results sentence from an input document if it studies the same population and research question described in the review. compression-based methods yielding less computationally intensive representations may also allow for full information retention, enabling salience determinations at the model-level, depending on other input studies and the review question at hand. evaluation metrics that better capture summary quality unsurprisingly, our defined automated evaluation metrics are lacking, in many cases failing to capture summary quality issues identified during qualitative analysis @xcite . both of our task datasets are highly compressive, e.g. the average compression ratio for the cochrane dev set is around 33 while that of the msˆ2 dev set is over 100! yet, a baseline such as copying the background section of msˆ2 leads to fairly good performance when assessed using (fuzzy-)token overlap metrics such as rouge and bertscore. this indicates that the task is perhaps less about summarizing and more about synthesizing relevant results, and hence, n-gram and token similarity-based metrics would be insufficient for capturing content similarity. these are similar concerns to those raised in single-document summarization evaluation @xcite . we included evidence inference metrics in evaluation to offer a counterpoint to more traditional metrics, yet they bring their own challenges. the values of these metrics are not particularly comparable between the two subtask datasets, nor are the numbers easy to interpret, e.g., how much worse is a model that scores 0.4 to 0.3 ∆ei-f1 at a system level? additionally, we currently perform evidence inference scoring for all possible pico tuples, regardless of whether a relationship occurs between members of each tuple, which can lead to degradation in performance (where most tuples are classified as "no effect," washing out actual differences between documents; see discussion in @xcite . improvements on pico tuple detection and alignment between documents could dramatically improve the value of evidence inference for mslr evaluation. in addition to evidence inference-based metrics, we anticipate investigating how entailment or question-answering-based evaluation metrics for single-document summarization evaluation @xcite could be extended into the multi-document space for this task (and how well existing approaches fare on this specialized data and task). further data is needed to iterate upon modelbased evaluation metrics. towards this, we intend to collect and release a dataset of human annotations of summary quality for a sample of generations submitted to this shared task, as described in section 2: evaluation. initial results will be presented at the sdp 2022 workshop. ## conclusion the mslr2022 shared task initiated further investigation into the challenging task of automatically synthesizing study results into a literature review summary. the task received submissions from six teams, leading to modest improvements on task performance and significant insights into the remaining challenges for this task. a primary challenge involves the insufficiency of automated evaluation metrics for assessing performance improvements on this task, towards which we intend to provide new datasets and methods to support and incentivize further research on this problem.