LED down the rabbit hole: exploring the potential of global attention for biomedical multi-document summarisation (2022)

Abstract: In this paper we report the experiments performed for the submission to the Multidocument summarisation for Literature Review (MSLR) Shared Task. In particular, we adopt Primera model to the biomedical domain by placing global attention on important biomedical entities in several ways. We analyse the outputs of 23 resulting models and report some patterns related to the presence of additional global attention, number of training steps and the inputs configuration.

## introduction in this paper we describe our experiments and results on the multidocument summarisation for literature review (mslr) shared task. 1 in particular, we attempt to improve on previous multi-document summarisation models in the biomedical domain, which have tried to integrate domain knowledge by marking important biomedical entities @xcite . we hypothesise that highlighting such entities by placing global attention on them will enable better aggregation and normalisation of related entities across documents, and thus improve the factuality of the generated summaries. to explore this idea, we experiment with four different ways of modifying the global attention mechanism of primera @xcite , a recent state-of-the-art model designed for multi-document summarisation (mds). in particular, while by default the global attention tokens in primera are used to separate documents in the input and capture their relationships, we assign global attention to important biomedical entities in input documents to create links between them. moreover, to examine the effect of content selection on the quality of summaries produced by this underlying model, we compare results where we use the whole abstract as input vs. only the concluding sentences (which we expect to be more informative). we train and analyse models in zero-shot, few-shot (10 and 100), as well as fully fine-tuned scenarios. overall we evaluate (using both automatic metrics and human evaluation) a total of 23 models, two of which formed our official submissions to the leaderboard. 2 both submitted models substantially outperform the baseline approaches @xcite in terms of automatic metrics, and one achieves the best performance in terms of bertscore and rouge-2 among all submissions. overall, our contributions in comparison to the previously published domain-specific models for mds are the following: ## conclusion we tackle the problem of biomedical multidocument summarisation by incorporating pico information into a strong summarisation model, and using global attention to enhance the representation of this information. through automatic and human evaluations on an extensive set of experiments, we find that adding global attention to pico spans would help in (1) generating more abstractive summaries, and (2) improving summarization quality in few-shot settings, which is especially important in the biomedical domain.