Exploring the limits of a base BART for multi-document summarization in the medical domain (2022)

Abstract: This paper is a description of our participation in the Multi-document Summarization for Literature Review (MSLR) Shared Task, in which we explore summarization models to create an automatic review of scientific results. Rather than maximizing the metrics using expensive computational models, we placed ourselves in a situation of scarce computational resources and explore the limits of a base sequence to sequence models (thus with a limited input length) to the task. Although we explore methods to feed the abstractive model with salient sentences only (using a first extractive step), we find the results still need some improvements.

## introduction to summarize medical knowledge on specific issues, researchers undertake systematic reviews of the available literature. the process is usually long and expensive; it requires identifying appropriate studies, critically interpreting their findings, and finally synthesizing the results. recently, natural language processing (nlp) researchers have explored the use of automatic text summarization models and tools to assist researchers with the process. previous works by @xcite ; @xcite have tried to model the problem as a multi-document summarization task, where several input papers (or abstracts) are summarized to generate review conclusions. summarizing several documents is challenging, and few resources exist @xcite compared to single-document summarization tasks. the shared task of multi-document summarization for literature review (mslr) adopted a similar approach and challenged participants to explore the state-of-the-art systems with two large-scale multi-document summarization datasets for literature review. to this end, instead of aiming at using very complex models to maximize the target metrics, we place ourselves in a situation of scarce computational resources and explore the limits of a base sequence-to-sequence model, bart, to the task. our contributions to this shared task, therefore, are as follows: after analyzing the datasets (section 2), we first experiment with baseline models (section 3.1); since the model can only deal with a limited number of input tokens, we explore various strategies to reduce the input size (section 3.2). ## datasets and metrics we evaluated the models on two datasets: cochrane (wallace et al., 2021): the dataset consists of 4,692 systematic reviews from the cochrane collaboration foot_0 . the target is the "authors' conclusions" of the systematic review abstracts, while the input is a set of titles and abstracts of the related clinical trials. @xcite : is built from papers in the semantic scholar literature corpus @xcite . it consists of 17,876 reviews. the dataset also contains some background text derived from the reviews. the dataset creation was semi-automatic: for each review, each sentence is classified as background, target or other and sentences are then aggregated. ## experiments and results in this work, we explore the use of a simple bart base model @xcite ) -that we leave unchanged -for the task of multi-document summarization. the bart model is limited to input size of 1024 sub-token. however, as figure 1 shows above, concatenating the abstracts leads to very long input sentences, that cannot be dealt with by the model. to this end, we explore if performing a previous extractive step improves performance. since the target text summarizes the findings of previous work, we also explore the use of a classifier to extract results only from the input. ## conclusions we have explored a number of base bart models for the task of generating systematic reviews in the medical domain. given the limited number of tokens bart can handle, we adopted several simple extractive strategies to retrieve salient sentences to the abstractive model; we also trained a model from the abstract results sentences only. generally, we found results on the cochrane datasets are much more encouraging than those on the ms^2 and we believe that using the background info might improve performance. we found that the results obtained from the salient sentences only show mixed results. for ms^s, extracting the results sentences only seems to be the most promising method. for the cochrane dataset, all extractive methods show small improvements over the baseline. lexrank seems to be the most promising, as it slightly improves the results, both in terms of rouge and factuality metrics. in addition to ours, other strategies could be explored to sort the input abstract: deyoung et al. (2021), for example, sorts abstracts by some measures of quality; it would be interesting to see how this compares to our proposed strategies. we also plan to explore different input representations that go beyond the simple concatenation of abstract and data augmentation techniques. another possible route could be that of extracting domain-specific concepts, through, e.g., pubtator @xcite , to enrich abstracts.