The patient is more dead than alive: exploring the current state of the multi-document summarisation of the biomedical literature (2022)

Abstract: Although multi-document summarisation (MDS) of the biomedical literature is a highly valuable task that has recently attracted substantial interest, evaluation of the quality of biomedical summaries lacks consistency and transparency. In this paper, we examine the summaries generated by two current models in order to understand the deficiencies of existing evaluation approaches in the context of the challenges that arise in the MDS task. Based on this analysis, we propose a new approach to human evaluation and identify several challenges that must be overcome to develop effective biomedical MDS systems.

## introduction with the number of biomedical publications doubling every two years @xcite , it is difficult for medical professionals to incorporate new, often contradictory, evidence into their daily work, as it would require appraising, comparing and synthesising the outcomes of multiple primary studies @xcite . systematic reviews, which aggregate such evidence from multiple clinical trials, provide only a partial solution, as they are very time-consuming to write and thus can be unavailable for newer clinical questions or quickly become outdated. in this context, the ability to automatically summarize evidence from multiple studies is of high practical importance. the task, however, is more challenging than general multidocument summarization (mds), as the summaries must correctly draw conclusions based on often contradictory studies, and aggregate details such as groups of patients or names and doses of treatments, in addition to dealing with often-cited difficulties posed by biomedical text such as complex lexical and semantic relationships between concepts @xcite . though recent approaches to biomedical summarization acknowledge the additional challenges of the task and try to incorporate some domain-specific knowledge to deal with them @xcite @xcite , we still lack a solid understanding of how well current models capture such knowledge, how useful the generated summaries are, or how to measure progress. in this paper, we propose a systematic approach to human evaluation of biomedical summaries, and apply it to analyse the summaries generated by two state-of-art systems. we examine the common errors in generated summaries and the correlation of automatic metrics such as rouge @xcite with our evaluation results. we choose summarization models proposed by @xcite , as they not only demonstrate the abilities of endto-end neural models, but also incorporate domainspecific knowledge such as entity prompts. the contributions of this paper are as follows: ## related studies and motivation although the importance of mds in the biomedical domain was recognized around 20 years ago with studies such as @xcite and @xcite defining some requirements and operations specific to biomedical summarization (e.g. the ability to resolve contradicting statements), until recently there have been few end-to-end systems (e.g. persival @xcite ) due to the complexity of the task. in the last few years, apart from several shared tasks and challenges dedicated to multi-answer biomedical summarization -including mediqa 2021 @xcite and bioasq ( nentidis et al., 2021) 2021 ) revived the idea of symbolic mds @xcite ) by combining a deterministic content plan with a pre-trained language model. here, we are particularly interested in the model by @xcite as it reflects the setting of summarization systems "in the wild": their input is all clinical trials cited by a systematic review rather than a sample of trials which the review was based on @xcite or a curated list of trials relevant to the summary @xcite . in terms of evaluation metrics, there has been a growing awareness of the inability of rouge to reflect the factual accuracy of summaries, so some other automatic metrics, including inference-based @xcite and question-answeringbased methods @xcite ) have been proposed. there have also been attempts to make the human evaluation more objective and systematic by defining linguistically grounded error categories and evaluation criteria @xcite . in the biomedical domain, although there are some new automatic measures proposed, such as aggregation cognisance @xcite -which measures the ability of the model to recognize if the input texts are in agreement or contradiction -and ∆ei (deyoung et al., 2021) -which reflects the alignment of summaries in terms of direction of their findings -human evaluation has primarily been based on the likert scale @xcite , making it difficult to reproduce and interpret. in this work we aim to close this gap by establishing a more reliable, grounded and objective human evaluation framework, and applying it by assessing the summaries generated by the state-of-the-art mds system of @xcite . ## summarization models the models we evaluate were trained on a largescale dataset comprising 20k systematic reviews and 470k primary studies @xcite . the conclusions, taken from the abstract of the review, are the target for the summarization. the input consists of a prompt in form of the background section of the systematic review, and the abstracts of up to 25 studies cited in the review. 1 as the prompt (background) describes the review's objective, the task is similar to query-based summarization, but with a highly detailed prompt. we use the two summarization models explored in deyoung et al. (2021): bart @xcite and longformer @xcite . both models are similar in architecture but differ in their approach to handling long input sequences: for longformer ("led" henceforth) background is concatenated with all studies and encoded together before feeding to the decoder, while for bart each study is concatenated with background and encoded separately; then their encodings are concatenated together and fed to the decoder. to adapt the models to the biomedical domain, the authors decorate the inputs by adding special tags around @xcite elements, @xmath0, <int>, <@xmath1, and also by marking the different sections such as background. ## evaluation process and criteria we sampled 100 reviews each from test summaries generated by bart-and led-based models. to evaluate them in a more systematic manner, we define the following quality dimensions which capture both factuality and fluency. ## discussion in this section we point out some issues which could explain the poor performance of the summarization systems in terms of generating conclusions in the manner of systematic reviews, and show how they relate to the principles underlying the aggregation of medical evidence. we present these as challenges to be tackled in mds system development. ## conclusions in this research, we attempted to bring the importance of factuality in biomedical mds into attention, and demonstrated that the current models are still unreliable in this respect. moreover, we showed that they fail to pick up and aggregate important details from multiple documents, excessively relying on the prompt. to support our analysis, we established a simple and reproducible human evaluation benchmark which reflects aspects of quality important for biomedical mds but can be translated into other domains. finally, we showed that the progress in biomedical mds will be limited unless we acknowledge the domainspecific challenges of the task and work towards overcoming them. though we focused our efforts on a particular domain, we hope that this work prompts taking a closer look at the summarization results in other areas, as only objective evaluation of what the models are capable of and prone to do will allow us to improve them.