M 3: Multi-level dataset for Multi-document summarisation of Medical studies (2022)

Abstract: We present M3 (Multi-level dataset for Multi-document summarisation of Medical studies), a benchmark dataset for evaluating the quality of summarisation systems in the biomedical domain. The dataset contains sets of multiple input documents and target summaries of three levels of complexity: documents, sentences, and propositions. The dataset also includes several levels of annotation, including biomedical entities, direction, and strength of relations between them, and the discourse relationships between the input documents (“contradiction” or “agreement”). We showcase usage scenarios of the dataset by testing 10 generic and domain-specific summarisation models in a zero-shot setting, and introduce a probing task based on counterfactuals to test if models are aware of the direction and strength of the conclusions generated from input studies.

## introduction increased interest in summarisation has led to the development of many datasets and models, including in the biomedical domain, where summaries are required to support clinical decision making. nevertheless, we still lack deep understanding of what these models should be capable of to support user-relevant summarisation. the approach to summarisation is largely based on the newswire domain, extracting the most salient and diverse details about particular entities, and recombining them. yet multi-document summarisation in scientific domains poses unique challenges. for example, when summarising a set of clinical trials about drugs of a particular class, the general efficacy of that class and the certainty of that evidence is relevant, while a text full of potentially conflicting details about individual drugs is less helpful @xcite . this requires the model to aggregate entities, decide on the relationships between them, and determine the strength of conclusions, which remains a distant goal @xcite . existing summarisation systems are not up to such complex tasks, yet limited tools exist to determine where and why they are failing. the m3 (multi-level dataset for multi-document summarisation of medical studies) dataset aims to enable this, as follows: (1) we include inputs and targets of different levels of granularity such as documents, sentences and propositions, to help understand if a model is attending to correct text spans. ## related works current datasets for biomedical multi-document summarisation (mds) can broadly be divided into two groups: large datasets based on abstracts of systematic reviews and their underlying primary studies @xcite , and smaller datasets based on less complex inputs, targeting expert-written summaries aimed at a general audience @xcite . while systematic reviews are natural targets for summarisation of biomedical evidence and provide sufficient data to train neural summarisation models, they can be problematic to use. source texts are long, complex scientific documents containing significant extraneous information. for example, @xcite use all clinical trials cited by a review as inputs for summarisation, though many citations are for related studies or background information rather than for primary studies directly relevant to the review scope. though @xcite avoid this by using only primary studies as inputs, the problem is not completely solved. as we show below, the conclusions of systematic reviews often cannot be directly summarised from abstracts of underlying studies. the abstracts of primary studies may contain details not directly relevant to the conclusions of systematic reviews, such as details on methods, that need to be ignored. in contrast, the inputs of smaller-scale mds datasets such as @xcite and @xcite are shorter and more narrowly scoped, making them more appropriate sources for summarisation. however, as summaries are targeted at a more general audience and refer to evidence hand-picked by experts rather than a large set of primary studies, they are less relevant for the practical purpose of summarising and updating evidence for medical professionals. our dataset m3 tries to combine the advantages of both approaches: though it is based on systematic reviews and primary studies, we include only relevant studies as input and extract highly informative pieces of evidence from both sources and targets for summarisation. it is also not limited to cochrane reviews like the data of @xcite , making it more varied in style. ## dataset structure the benchmark dataset we present is based on biomedical literature for major eye diseases such as glaucoma and macular degeneration. we chose to focus the dataset around a particular topic to improve reliability by making it easier for the annotators and evaluators to understand the subject matter. this approach also helps to verify if models, which are usually trained on multi-topic datasets, can perform well in a specific domain with more limited vocabulary, testing whether they can learn to perform summarisation operations rather than rely on vocabulary distribution @xcite . on the other hand, our corpus is very diverse in terms of types of documents, comprising diagnostics, management, prognosis, cost effectiveness, genetics, risk factors, etc. studies in addition to the treatment trials which are usually the focus of research aimed at the synthesis of biomedical evidence. ## conclusions in this paper we presented a new dataset for multidocument biomedical summarisation which is designed to be used as a benchmark for evaluation and comparison of summarisation models. to assist this task, we provide inputs and targets of different granularity, human annotations in terms of several aspects, and also sets of counterfactual target summaries. we show some ways to use m3 by evaluating 10 summarisation models, and introduce a baseline probing task for determining awareness of polarity and modality of findings. we hope that the abundance of different annotations we provide will help the community to develop new ways to test and evaluate summarisation systems in the biomedical and scientific domain.