BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks (2021)

Abstract: Biomedical entity linking is the task of linking entity mentions in a biomedical document to referent entities in a knowledge base. Recently, many BERT-based models have been introduced for the task. While these models achieve competitive results on many datasets, they are computationally expensive and contain about 110M parameters. Little is known about the factors contributing to their impressive performance and whether the over-parameterization is needed. In this work, we shed some light on the inner workings of these large BERT-based models. Through a set of probing experiments, we have found that the entity linking performance only changes slightly when the input word order is shuffled or when the attention scope is limited to a fixed window size. From these observations, we propose an efficient convolutional neural network with residual connections for biomedical entity linking. Because of the sparse connectivity and weight sharing properties, our model has a small number of parameters and is highly efficient. On five public datasets, our model achieves comparable or even better linking accuracy than the state-of-the-art BERT-based models while having about 60 times fewer parameters.

## introduction biomedical entity linking (el) @xcite is the task of linking biomedical mentions (e.g., diseases and drugs) to standard referent entities in a curated knowledge base (kb). for example, given the sentence "the average nh3 concentrations were low.", the mention nh3 should be linked to the entity kb:ammonia. biomedical el is an important research problem, with applications in many downstream tasks, such as biomedical question answering @xcite , information retrieval, and information extraction @xcite @xcite . in general, two main challenges of the el task are: recently, many bert-based models have been introduced for biomedical el @xcite @xcite . while these models can achieve state-of-the-art results on many biomedical el datasets, they are computationally expensive and contain about 110m parameters. even though there are scientific labs that have a lot of computing resources, many researchers still have minimal access to large-scale computational power @xcite . therefore, it is of practical importance to provide a more scalable solution for biomedical entity linking. furthermore, the factors contributing to the success of these large bert-based models remain unclear. and thus, it is not known whether the over-parameterization is needed to achieve competitive performance. in this work, through a set of probing experiments, we shed some light on the inner workings of existing bert models for biomedical el. surprisingly, the performance only changes slightly when the input word order is shuffled or when the attention scope is restricted. based on these observations, we propose an effective convolutional neural network with residual connections (rescnn) for the task. because of the sparse connectivity and weight sharing properties, rescnn has a small number of parameters and is highly efficient. experiments on five datasets show that the performance of rescnn is comparable to the state-of-the-art (sota) bert-based models while having about figure 1 : an illustration of the adopted approach to el. in this example, the closest neighbor to the source mention is the entity name anilines. therefore, this mention should be linked to the entity c0003038. ## methods in the following sections, we will first describe some preliminaries relating to the formulation of the el problem and a general approach for the task (sec. 2.1). we will then go into details about our probing experiments in sec. 2.2. we will describe the design of our rescnn model in sec. 2.3. ## experiments data and experimental setup we experiment across five different datasets: ncbi @xcite , bc5cdr-c and bc5cdr-d @xcite , medmentions @xcite , and cometa @xcite . for each dataset, we follow the data split by @xcite . it is worth highlighting that even though the five datasets can all be categorized as "biomedical datasets", they have very different characteristics. for example, while medmentions was constructed by annotating scientific papers, cometa was built by crawling reddit (a social media forum). we report results in terms of top-1 accuracy. details about the hyperparameters are in the appendix. ## conclusions and future work our work has shown that while bert has been widely used for many nlp tasks, it is sometimes an overkill for some tasks, in which case, a simpler model can be as effective as bert and is often much more efficient. an interesting future direction is to study further how to systematically simplify/compress bert based on the insights obtained using probing experiments to increase efficiency while maintaining effectiveness. we plan to extend our work to other domains as well as other information extraction tasks @xcite @xcite @xcite .