Improving Biomedical Abstractive Summarisation with Knowledge Aggregation from Citation Papers (2023)

Abstract: Abstracts derived from biomedical literature possess distinct domain-specific characteristics, including specialised writing styles and biomedical terminologies, which necessitate a deep understanding of the related literature. As a result, existing language models struggle to generate technical summaries that are on par with those produced by biomedical experts, given the absence of domain-specific background knowledge. This paper aims to enhance the performance of language models in biomedical abstractive summarisation by aggregating knowledge from external papers cited within the source article. We propose a novel attention-based citation aggregation model that integrates domain-specific knowledge from citation papers, allowing neural networks to generate summaries by leveraging both the paper content and relevant knowledge from citation papers. Furthermore, we construct and release a large-scale biomedical summarisation dataset that serves as a foundation for our research. Extensive experiments demonstrate that our model outperforms state-of-the-art approaches and achieves substantial improvements in abstractive biomedical text summarisation.

## introduction biomedical text summarisation plays a pivotal role in facilitating the comprehension of the vast and constantly expanding body of biomedical literature @xcite , which poses a significant challenge for clinicians and domain experts who strive to remain well-informed in their respective fields. to address this challenge, the generation of high-quality summaries from the extensive corpus of biomedical literature holds immense potential in supporting research and advancements within the biomedical domain @xcite . one of the key challenges in biomedical natural language generation (nlg) lies in effectively figure 1 : the overview of our proposed citation knowledge aggregation framework. in this framework, language models are trained to incorporate features from both the content of the main paper and the abstracts of its cited papers. the rationale behind this approach is that the cited papers often share relevant research backgrounds, terminologies and writing styles, which can be used as a good template for the summary generation. to address this gap in the existing biomedical summarisation dataset, we construct a novel biomedical summarisation dataset utilising an opensource biomedical literature corpus provided by the allen institute foot_0 . during the dataset construction process, we applied rigorous filtering criteria to eliminate low-quality samples. specifically, we discarded samples with an insufficient number of citations (less than three distinct citations), as well as unqualified papers whose unique identifiers (uids) or citation uids were inaccessible within the corpus. additionally, we designed heuristic rules to select and transform the unstructured raw data corpus into a structured dataset in jsonl format. the final dataset comprises over 10,000 instances, with each instance having an average of 16 citations. to the best of our knowledge, this is the largest biomedical literature dataset foot_1 specifically tailored for citation paper-enhanced biomedical text summarisation. furthermore, we provide the corresponding methods for collecting the citation network, including cited papers and their associations. facilitated by our biomedical summarisation dataset, we further propose a novel approach to biomedical document summarisation whereby we enhance neural models with external domainspecific knowledge in the form of the abstracts of cited papers. accordingly, we introduce an attention-based network @xcite that dynamically aggregates features extracted from the citation abstracts with the encoded content features of the main paper. this aggregation is achieved by applying attention mechanisms to the associated abstracts of all cited papers, which provides the subsequent summary decoding process with addi-tional features derived from abstracts of the citation papers. within this framework, the base language model can effectively leverage both the features of the main paper and the additional domain-specific knowledge obtained from cited papers. consequently, this integration leads to enhanced performance in text summarisation. extensive experiments demonstrate that our model outperforms state-of-the-art baselines in abstractive biomedical text summarisation. we also conducted an in-depth quantitative analysis to verify the performance gain obtained by our attention-based citation knowledge enhancement framework foot_3 . our contributions are summarised as follows: ## related work in recent years, a variety of large-scale pre-trained models (plms), such as bart @xcite ; t5 @xcite ; gpt-2 @xcite , have demonstrated remarkable performance improvements across various tasks @xcite @xcite in the natural language generation (nlg) domain. these plms have also been widely applied to biomedical text summarisation. these models, e.g. biobert @xcite and biobart @xcite , have achieved remarkable performance by training on extensive biomedical literature corpora, such as pubmed 4 and mimic-iii 5 . however, certain high-level knowledge, e.g., the understanding of medical terminologies, cannot be adequately captured solely through the implicit modeling of word probabilities. to address this limitation, the improvement of biomedical background knowledge understanding is able to necessitate the integration of additional knowledge systems, such as conceptual ontologies. these ontologies explicitly model representations of domain-specific knowledge learned by neural networks. recent studies have proposed incorporating biomedical knowledge, including terminologies @xcite ) and concepts @xcite , to enhance the performance of these language models and bridge the gap between language understanding and specialized biomedical knowledge. indeed, several notable works have focused on enhancing summarisation through citations in the open domain, such as @xcite and @xcite . however, it is important to highlight that the progress of language models in the biomedical domain has been hindered by the limited availability of datasets and resources. this scarcity has impeded the further advancement and improvement of pre-trained language models (plms) specifically tailored for biomedical applications. in this study, we require a dataset that contains retrievable citation papers, making traditional raw data corpora such as pubmed and mimic-iii inadequate. to date, the sole public dataset we could find is the text analysis conference (tac) 2014 biomedical summarization track @xcite . however, this dataset is limited in size comprising merely 313 instances, and is somewhat outdated. therefore, we construct a novel dataset for investigating biomedical citation-enhanced summarisation. ## conclusion in conclusion, we proposed a novel attentionbased citation aggregation model that incorporates domain-specific knowledge from citation papers. by integrating this additional information, our model enables neural networks to generate summaries that benefit from both the paper content and the associated knowledge extracted from citation papers. furthermore, we introduced a specialized biomedical summarisation dataset, which served as a valuable resource for evaluating and advancing our research. the effectiveness of our approach was demonstrated through extensive experiments, where our model consistently outperformed stateof-the-art methods in biomedical text summarisation. the results highlight the significant improvements achieved by leveraging knowledge from citation papers and the potential for our model to enhance the understanding of biomedical literature through natural language generation techniques.