A Comprehensive Evaluation of Biomedical Entity Linking Models (2023)

Abstract: Biomedical entity linking (BioEL) is the process of connecting entities referenced in documents to entries in biomedical databases such as the Unified Medical Language System (UMLS) or Medical Subject Headings (MeSH). The study objective was to comprehensively evaluate nine recent state-of-the-art biomedical entity linking models under a unified framework. We compare these models along axes of (1) accuracy, (2) speed, (3) ease of use, (4) generalization, and (5) adaptability to new ontologies and datasets. We additionally quantify the impact of various preprocessing choices such as abbreviation detection. Systematic evaluation reveals several notable gaps in current methods. In particular, current methods struggle to correctly link genes and proteins and often have difficulty effectively incorporating context into linking decisions. To expedite future development and baseline testing, we release our unified evaluation framework and all included models on GitHub at https://github.com/davidkartchner/biomedical-entity-linking

## introduction biomedical entity linking (bioel) is the process of identifying biomedical concepts (e.g. diseases, chemicals, cell types, etc.) in text and connecting them to a unique identifier in a knowledge base (kb). entity linking (el) is critical in text mining, as it allows concepts to be connected across disparate literature. this "harmonization" enables quick access to connected information in the knowledge base and allows for unified reasoning regarding diverse surface forms and mentions. second, annotating a biomedical corpus is a timeconsuming task that requires specialized domain expertise, which have limited availability to label data. concretely, the largest labeled bioel dataset, medmentions @xcite , covers approximately 1% of the candidate entities in its reference ontology while annotating 0.17% of the abstracts in pubmed. third, though dozens of ontologies and terminologies have been curated in recent years, concepts are often not cross-referenced, leading to a lack of interoperability. furthermore, even carefully unified collections such as umls lack synonyms and definitions for the vast majority of concepts. most biomedical concepts are not labeled in any gold-standard el corpus. thus, robust zero-shot performance is critical for effectively performing el at scale. however, lack of labelled data by specialized domain experts simultaneously makes it difficult to accurately assess the capacity of current models to generalize to unseen data. while some bioel surveys have been published @xcite , they do not evaluate models in a consistent way or on a uniform collection of datasets. rather than a traditional survey, we contend a systematic evaluation of current bioel models is needed to: 1) accurately compare current models; 2) identify strengths and weaknesses; 3) prioritize directions for future research; 4) provide a framework to expedite future bioel development. to address these needs, this paper contributes the following: ## problem definition we assume that we are given a corpus @xmath0, where each d i is a document in the cor- symbol definition d corpus of documents d i individual document in corpus m ij an entity mention in document i c -(+) ij left (right) context of entity mention m ij m collection of all entities in context e database of entities e k individual entity we assume that a database of entities is provided @xmath3. each entity is identified by a unique identifier and may also contain informational metadata such as entity type(s), definition, aliases, etc. most entity-linkers assume access to ground truth entity mention spans. however, these can be determined programmatically via a named entity recognition algorithm. the task of entity linking is to learn a function f : @xmath4. most entity linkers use a two-stage approach to find the correct entity link for a given mention span. the first stage is candidate generation (cg), which defines a function f cg : @xmath5. once a set of entity candidates have been generated, they are passed into a named entity disambiguation * (ned) module f n ed : e n × @xmath6, which chooses the best candidate for a final entity link. in practice, f cg is chosen to be a computationally inexpensive algorithm with high recall, while f n ed is more costly and precise. the final entity linker is defined as @xmath7. ## datasets we evaluate included bioel methods on a variety of biomedical datasets ( ## models a wide variety of methods have been used for bioel. here we describe families of models used for bioel and list included models from each category. more detailed descriptions of each individual model are found in appendix b. we summarize the different models evaluated in models evaluated were those with near state-ofthe-art performance at time of publication when evaluated on at least one included bioel entity linking dataset. from this pool, we excluded models with no open-source implementation or whose implementation was rendered unusable due to lack of documentation or software updates. with the exception of metamap, all models were published in the past 5 years. ## evaluation strategy as noted in @xcite , evaluation strategies between different entity linking papers are inconsistent, leading to wide disparities in reported results. differences primarily revolve around how to score predictions where multiple normalizations are given for a named entity, e.g. because all predicted entities share the same alias. we identified three main strategies for this in the literature. for each dataset, we generate ranked entity candidates from each model in sec. 4. for models that only natively link to umls, links to other kbs are computed by predicting entities in umls @xcite and mapping these predictions to other kbs using cross references provided by umls and obofoundary @xcite . predictions are ranked and evaluated using recall @ k for @xmath8, 2, . . . , 10} (note that recall@1 is equivalent to accuracy). we perform our main evaluations using the basic evaluation strategy unless otherwise specified. ## results our main result in ## discussion of the models evaluated, there was no model that clearly performed "best" for all datasets or evaluation metrics. however, arboel showed consistently high performance and was always among the highest-performing models on each dataset. sapbert was arguably the best-performing alias matching method, sometimes surpassing arboel in recall@5 for various datasets. one noteworthy result is the relatively poor performance of all models in