MS ˆ2: Multi-Document Summarization of Medical Studies (2021)

Abstract: To assess the effectiveness of any medical intervention, researchers must conduct a time-intensive and manual literature review. NLP systems can help to automate or assist in parts of this expensive process. In support of this goal, we release MSˆ2 (Multi-Document Summarization of Medical Studies), a dataset of over 470k documents and 20K summaries derived from the scientific literature. This dataset facilitates the development of systems that can assess and aggregate contradictory evidence across multiple studies, and is the first large-scale, publicly available multi-document summarization dataset in the biomedical domain. We experiment with a summarization system based on BART, with promising early results, though significant work remains to achieve higher summarization quality. We formulate our summarization inputs and targets in both free text and structured forms and modify a recently proposed metric to assess the quality of our system’s generated summaries. Data and models are available at https://github.com/allenai/ms2 .

## introduction multi-document summarization (mds) is a challenging task, with relatively limited resources and modeling techniques. existing datasets are either in the general domain, such as wikisum @xcite and multi-news @xcite , or very small such as duc 1 or tac 2011 @xcite . in this work, we add to this burgeoning area by developing a dataset for summarizing biomedical findings. we derive documents and summaries from systematic literature reviews, a type of biomedical paper that synthesizes results across many other studies. our aim in introducing msˆ2 is to: (1) expand mds to the biomedical domain, (2) investigate fundamentally challenging issues in nlp over scientific text, such as summarization over contradictory information and assessing summary quality via a structured intermediate form, and (3) aid in distilling large amounts of biomedical literature by supporting automated generation of literature review summaries. systematic reviews synthesize knowledge across many studies @xcite , and they are so called for the systematic (and expensive) process of creating a review; each taking 1-2 years to complete @xcite . 2 as we note in fig. 2 , a delay of around 8 years is observed between reviews and the studies they cite! the time and cost of creating and updating reviews has inspired efforts at automation @xcite @xcite , and the constant deluge of studies foot_1 has only increased this need. to move the needle on these challenges and support further work on literature review automation, we present msˆ2, a multi-document summarization dataset in the biomedical domain. our contributions in this paper are as follows: ## background systematic reviews aim to synthesize results over all relevant studies on a topic, providing high quality evidence for biomedical and public health decisions. they are a fixture in the biomedical literature, with many established protocol around their registration, production, and publication @xcite @xcite . each systematic review addresses one or several research questions, and results are extracted from relevant studies and summarized. for example, a review investigating the effectiveness of vitamin b12 supplementation in older adults @xcite synthesizes results from 9 studies. the research questions in systematic reviews can be described using the pico framework @xcite . pico (which stands for population: who is studied? intervention: what intervention was studied? comparator: what was the intervention compared against? outcome: what was measured?) defines the main facets of biomedical research questions, and allows the person(s) conducting a review to identify relevant studies (studies included in a review generally have the same or similar pico elements as the review). a medical systematic review is one which reports results for applying any kind of medical or social intervention to a group of people. interventions are wideranging, including yoga, vaccination, team training, education, vitamins, mobile reminders, and more. recent work on evidence inference @xcite goes beyond identifying pico elements, and aims to group and identify overall findings in reviews. msˆ2 is a natural extension of these paths: we create a dataset and build a system with both natural summarization targets from input studies, while also incorporating the inherent structure studied in previous work. in this work, we use the term review when describing literature review papers, which provide our summary targets. we use the term study to describe the documents that are cited and summarized by each review. there are various study designs which offer differing levels of evidence, e.g. clinical trials, cohort studies, observational studies, case studies, and more @xcite . of these study types, randomized controlled trials (rcts) offer the highest quality of evidence @xcite . ## dataset we construct msˆ2 from papers in the semantic scholar literature corpus @xcite . first, we create a corpus of reviews and studies based on the suitability criteria defined in §3.1. for each review, we classify individual sentences in the abstract to identify summarization targets ( §3.2). we augment all reviews and studies with pico span labels and evidence inference classes as described in §3.3.1 and §3.3.2. as a final step in data preparation, we cluster reviews by topic and form train, development, and test sets from these clusters ( §3.4). ## experiments we experiment with a texts-to-text task formulation (fig. 1 ). the model input consists of the background statement and study abstracts; the output is the target statement. we also investigate the use of the structured form described in §3.3.2 for a supplementary table-to-table task, where given inputs of i/o pairs from the review; the model tries to predict the evidence direction. we provide initial results for the table-to-table task, although we consider this an area in need of active research. ## discussion though mds has been explored in the general domain, biomedical text poses unique challenges such as the need for domain-specific vocabulary and background knowledge. to support development of biomedical mds systems, we release the msˆ2 dataset. msˆ2 contains summaries and documents derived from biomedical literature, and can be used to study literature review automation, a pressing real-world application of mds. we define a seq2seq modeling task over this dataset, as well as a structured task that incorporates prior work on modeling biomedical text @xcite . we show that although generated summaries tend to be fluent and on-topic, they only agree with the evidence direction in gold summaries around half the time, leaving plenty of room for improvement. this observation holds both through our ∆ei metric and through human evaluation of a small sample of generated summaries. given that only 54% of study evidence directions agree with the evidence directions of their review, modeling contradiction in source documents may be key to improving upon existing summarization methods. limitations challenges in co-reference resolution and pico extraction limit our ability to generate accurate pico labels at the document level. errors compound at each stage: pico tagging, taking the product of is and os at the document level, and predicting ei direction. pipeline improvements are needed to bolster overall system performance and increase our ability to automatically assess performance via automated metrics like ∆ei. relatedly, automated metrics for summarization evaluation can be difficult to interpret, as the intuition for each metric must be built up through experience. though we attempt to facilitate understanding of ∆ei by offering a degenerate example, more exploration is needed to understand how a practically useful system would perform on such a metric. future work though we demonstrate that seq2seq approaches are capable of producing fluent and on-topic review summaries, there are significant opportunities for improvement. data improvements include improving the quality of summary targets and intermediate structured representations (pico tags and ei direction). another opportunity lies in linking to structured data in external sources such as various clinical trial databases 7,8,9 rather than relying solely on pico tagging. for modeling, we are interested in pursuing joint retrieval and summarization approaches @xcite . we also hope to explicitly model the types of contradictions observed in tab. 8, such that generated summaries can capture nuanced claims made by individual studies. ## conclusion given increasing rates of publication, multidocument summarization, or the creation of literature reviews, has emerged as an important nlp task in science. the urgency for automation technologies has been magnified by the covid-19 pandemic, which has led to both an accelerated speed of publication @xcite as well as proliferation of non-peer-reviewed preprints which may be of lower quality @xcite . by releasing msˆ2, we provide a mds dataset that can help to address these challenges. though we demonstrate that our mds models can produce fluent text, our results show that there are significant outstanding challenges that remain unsolved, such as pico tuple extraction, co-reference resolution, and evaluation of summary quality and faithfulness in the multi-document setting. we encourage others to use this dataset to better understand the challenges specific to mds in the domain of biomedical text, and to push the boundaries on the real world task of systematic review automation. sentence class p r f background 0.972 0.952 0.958 recommendation 0.418 0.296 0.338 evidence_quality 0.580 0.528 0.550 effect 0.752 0.800 0.774 methods 0.938 0.944 0.940 goal 0.916 0.936 0.924 detailed_findings 0.856 0.860 0.858 etc 0.406 0.322 0.338 further_study 0.756 0.864 0.804