Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies (2021)

Abstract: Biomedical question-answering (QA) has gained increased attention for its capability to provide users with high-quality information from a vast scientific literature. Although an increasing number of biomedical QA datasets has been recently made available, those resources are still rather limited and expensive to produce; thus, transfer learning via pre-trained language models (LMs) has been shown as a promising approach to leverage existing general-purpose knowledge. However, fine-tuning these large models can be costly and time consuming and often yields limited benefits when adapting to specific themes of specialised domains, such as the COVID-19 literature. Therefore, to bootstrap further their domain adaptation, we propose a simple yet unexplored approach, which we call biomedical entity-aware masking (BEM) strategy, encouraging masked language models to learn entity-centric knowledge based on the pivotal entities characterizing the domain at hand, and employ those entities to drive the LM fine-tuning. The resulting strategy is a downstream process applicable to a wide variety of masked LMs, not requiring additional memory or components in the neural architectures. Experimental results show performance on par with the state-of-the-art models on several biomedical QA datasets.

## introduction biomedical question-answering (qa) aims to provide users with succinct answers given their queries by analysing a large-scale scientific literature. it enables clinicians, public health officials and endusers to quickly access the rapid flow of specialised knowledge continuously produced. this has led the research community's effort towards developing specialised models and tools for biomedical qa and assessing their performance on benchmark datasets such as bioasq @xcite . producing such data is time-consuming and there have been attempts to fine-tune pre-trained large-scale language models for general-purpose qa tasks @xcite @xcite and then use them directly for biomedical qa. furthermore, there has also been increasing interest in developing domain-specific language models, such as biobert @xcite or roberta-biomed @xcite , leveraging the vast medical literature available. while achieving state-of-the-art results on the qa task, these models come with a high computational cost: biobert needs ten days on eight gpus to train @xcite , making it prohibitive for researchers with no access to massive computing resources. an alternative approach to incorporating external knowledge into pre-trained language models is to drive the lm to focus on pivotal entities characterising the domain at hand during the fine- bert roberta biobert … patients diabetes hr composite endpoints ... covid-19 coronary pneumonia ... tuning stage. similar ideas were explored in works by zhang et al. (2019), sun et al. ( 2020 ), which proposed the ernie model. however, their adaptation strategy was designed to generally improve the lm representations rather than adapting it to a particular domain, requiring additional objective functions and memory. in this work we aim to enrich existing general-purpose lm models (e.g. bert @xcite ) with the knowledge related to key medical concepts. in addition, we want domain-specific lms (e.g. biobert) to reencode the already acquired information around the medical entities of interests for a particular topic or theme (e.g. literature relating to covid-19). therefore, to facilitate further domain adaptation, we propose a simple yet unexplored approach based on a novel masking strategy to finetune a lm. our approach introduces a biomedical entity-aware masking (bem) strategy encouraging masked language models (mlms) to learn entitycentric knowledge ( §2). we first identify a set of entities characterising the domain at hand using a domain-specific entity recogniser (scispacy (neumann et al., 2019)), and then employ a subset of those entities to drive the masking strategy while fine-tuning (figure 1 ). the resulting bem strategy is applicable to a vast variety of mlms and does not require additional memory or components in the neural architectures. experimental results show performance on a par with the state-of-the-art models for biomedical qa tasks ( §4) on several biomedical qa datasets. a further qualitative assessment provides an insight into how qa pairs benefit from the proposed approach. ## bem: a biomedical entity-aware masking strategy the fundamental principle of a masked language model (mlm) is to generate word representations that can be used to predict the missing tokens of an input text. while this general principle is adopted in the vast majority of mlms, the particular way in which the tokens to be masked are chosen can vary considerably. we thus proceed analysing the random masking strategy adopted in bert @xcite which has inspired most of the existing approaches, and we then introduce the biomedical entity-aware masking strategy used to fine-tune mlms in the biomedical domain. ## evaluation design biomedical reading comprehension. we represent a document as d i := (s i 0 , . . , s i j-1 ) , a sequence of sentences, in turn defined as s j := (w j 0 , . . , w j k-1 ), with w k a word occurring in s j . given a question q, the task is to retrieve the span w j s , . . , w j s+t from a document d j that can answer the question. we assume the extractive qa setting where the answer span to be extracted lies entirely within one, or more than one document d i . in addition, for consistency with the covidqa dataset and to compare with results in @xcite , we consider a further and sightly modified setting in which the task consists of retrieving the sentence s i j that most likely contains the exact answer. this sentence level qa task mitigates the non-trivial ambiguities intrinsic to the definition of the exact span for an answer, an issue particularly relevant in the medical domain and well-know in the literature @xcite foot_0 . datasets. we assess the performance of the proposed masking strategies on two biomedical datasets: covidqa and bioasq. ## experimental results and discussion we report the results on the qa tasks in among the unsupervised models, bert achieves slightly better performance than roberta on covidqa, yet the situation is reversed on bioasq (rows 1,5). the low precision of the two models (especially on the bioasq dataset) confirms the difficulties in generalising to the biomedical domain. specialised language models such as roberta-biomed and biobert show a significant improvement on the covidqa dataset, but a rather limited one on bioasq (rows 9,13), highlighting the importance of having larger medical corpora to assess the model's effectiveness. a general boost in performance is shared across models fine-tuned on the qa tasks, with a large benefit from the bioasq qa. the performance gains obtained by the specialised models (biobert and roberta-biomed) suggest the importance of transferring not only the domain knowledge but also the ability to perform the qa task itself (rows 9,10; 13,14). a further fine-tuning step before the training over the qa pairs has been proven beneficial for all of the models. the bem masking strategy has significantly amplified the model's generalisability, with an increased adaptation to the biomedical themes shown by the notable improvement in r@3 and mrr; with the r@3 outperforming the stateof-the-art results of t5 fine-tuned on ms-marco @xcite and proving the effectiveness of the bem strategy. ## related work our work is closely related to two lines of research: the design of masking strategies for lms and the development of specialized models for the biomedical domain. masking strategies. building on top of the bert's masking strategy @xcite , a wide variety of approaches has been proposed @xcite @xcite . a family of masking approaches aimed at leveraging entity and phrase occurrences in text. span-bert, @xcite proposed to mask and predict whole spans rather than standalone tokens and to make use of an auxiliary objective function. ernie @xcite is instead developed to mask well-known named entities and phrases to improve the external knowledge encoded. similarly, knowbert @xcite explicitly model entity spans and use an entity linker to an external knowledge base to form knowledge enhanced entity-span representations. however, despite the analogies with the bem approach, the above masking strategies were designed to generally improve the lm representations rather than adapting them to particular domains, requiring additional objective functions and memory. biomedical lms. particular attention has been devoted to the adaptation of lms to the medical domain, with different corpora and tasks requiring tailored methodologies. biobert @xcite is a biomedical language model based on bert-base with additional pre-training on biomedical documents from the pubmed and pmc collections using the same training settings adopted in bert. biomed-roberta @xcite is instead based on roberta-base @xcite using a corpus of 2.27m articles from the semantic scholar dataset @xcite . scibert @xcite follows the bert's masking strategy to pre-train the model from scratch using a scientific corpus composed of papers from semantic scholar @xcite . out of the 1.14m papers used, more than 80% belong to the biomedical domain. ## conclusion we presented bem, a biomedical entity-aware masking strategy to boost lm adaptation to lowresource biomedical qa. it uses an entity-driven masking strategy to fine-tune lms and effectively lead them in learning entity-centric knowledge based on the pivotal entities characterizing the domain at hand. experimental results have shown the benefits of such an approach on several metrics for biomedical qa tasks.