Exploring Partial Knowledge Base Inference in Biomedical Entity Linking (2023)

Abstract: Biomedical entity linking (EL) consists of named entity recognition (NER) and named entity disambiguation (NED). EL models are trained on corpora labeled by a predefined KB. However, it is a common scenario that only entities within a subset of the KB are precious to stakeholders. We name this scenario partial knowledge base inference; training an EL model with one KB and inferring on the part of it without further training. In this work, we give a detailed definition and evaluation procedures for this practically valuable but significantly understudied scenario and evaluate methods from three representative EL paradigms. We construct partial KB inference benchmarks and witness a catastrophic degradation in EL performance due to dramatically precision drop. Our findings reveal these EL paradigms can not correctly handle unlinkable mentions (NIL), so they are not robust to partial KB inference. We also propose two simple-and-effective redemption methods to combat the NIL issue with little computational overhead.

## introduction biomedical entity linking (el) aims to identify entity mentions from biomedical free texts and link them to the pre-defined knowledge base (kb, e.g. umls @xcite ), which is an essential step for various tasks in biomedical language understanding including relation extraction @xcite @xcite and question answering @xcite . el naturally contains two subtasks: named entity recognition (ner) and named entity disambiguation (ned). ner is designed for mention detection, while ned aims to find the best match entities from kb. one direct way for el is executing ner and ned sequentially @xcite @xcite . neural ner and ned models are usually trained by corpora labeled with a kb. however, potential users of biomedical el, including doctors, patients, and developers of knowledge graphs (kgs) may only be interested in entities inside a subset of kb such as snomed-ct @xcite , one semantic type of entities in umls, or kb customized by medical institutions. besides, doctors from different medical institutions have different terminology sets. some hospitals are using icd-10, while some hospitals are still using icd-9 and even custom terminology set. patients are only interested in specific diseases, symptoms, and drugs. as for developers of kgs, they may need to build a kg for special diseases like diabetes @xcite and covid-19 @xcite , or particular relation types like drug-drug interaction @xcite . all scenarios above need to infer el using a partial kb. off-the-shelf models trained on a comprehensive kb will extract mentions linked to entities outside the users' kb. although retraining models based on users' kbs can obtain satisfactory performances, it is not feasible under most scenarios because users can have significantly different kbs and may have difficulties with computational resources in finetuning large-scale models. therefore, we propose a scenario focusing on inference on the partial kb. we name this scenario partial knowledge base inference: train an el model with one kb and infer on partial of this kb without further training. fig. 1 provides a case of this scenario. this scenario is widely faced in the medical industry but remains understudied. this work reviews and evaluates current stateof-the-art el methods under the partial kb inference scenario. to be specific, we evaluate three paradigms: (1) ner-ned @xcite , (2) ned-ner @xcite , (3) simultaneous generation @xcite . the first two paradigms are pipeline methods, whose difference is the order of ner and ned. the last paradigm is an end-to-end method that generates mention and corresponding concepts by language models. we construct partial kb inference datasets based on two widely used biomedical el datasets: bc5cdr @xcite and medmentions (mohan and li, 2019). our experimental findings reveal the different implicit mechanisms and performance bottlenecks within each paradigm which shows partial kb inference is challenging. we also propose two redemption methods based on our findings, post-pruning and thresholding, to help models improve partial kb inference performance effortlessly. post-pruning infers with a large kb and removes entities in the large kb but not in partial kb. post-pruning is effective but memoryunfriendly for storing embeddings of entities in the large kb. thresholding removes entities with scores below a threshold. these two redemption methods are all designed to reduce the impact of nil entities and boost el performances. to our best knowledge, this is the first work that researches partial kb inference in biomedical el. our main contributions are the following: angell et al. (2021) and agarwal et al. (2021) first rerank the disambiguation target to boost performance. to overcome the limitation of labeled ned corpus, liu et al. (2021); yuan et al. (2022c,a) leverage synonyms from huge biomedical kb for zero-shot ned. varma et al. (2021); zhang et al. (2021b) use weakly supervised data generated from wikipedia and pubmed for data augmentation. ner and ned are both essential components of el. in this work, we further explore partial kb inference by analyzing performance in these two steps and reveal how the design and order of ner and ned infer el performance in partial kb inference. entity linking although el can be handled by a direct pipeline of ner and ned, there is limited research focusing on the task as a whole in biomedical. as el may enjoy the mutual benefits from supervision of both subtasks, zhao et al. (2019) deal with biomedical el in a multi-task setting of ner and ned. medlinker (loureiro and jorge, 2020) and ujiie et al. (2021b) approach biomedical el by sequentially dealing with ner and ned in the general domain, @xcite is proposed and formulated el as a seq2seq task. they detect and disambiguate mentions with constrained language generation in an end-to-end fashion. we categorize genre as simultaneous-generate el. entqa @xcite provides a novel framework by first finding probable concepts in texts and then treating each extracted concept as queries to detect corresponding mentions in a question-answering fashion which is categorized as ned-ner in our framework. simultaneous-generate and ned-ner fashion are not widely examined in biomedical el, and they interest us to examine their performances for biomedical el and partial kb inferences. partial kb inference in el in the biomedical domain, there is no prior work considering this setting to the best of our knowledge. nilinker @xcite is the most related work which focuses on linking nil entities out of the training kb, while ours aim to infer el on part of the training kb and discard nil entities. ## problem definition entity linking let e denote a target kb comprises of a set of biomedical concepts. given a text s with length n, an el model aims to find the mentions m and corresponding concepts @xmath0. concretely, the model can be regarded as a mapping f : @xmath1, where p @xmath2, j, e)|0 ≤ @xmath3, @xmath4, and i, j mark the start and end positions of the mention spans in s. ## experiments in this section, we introduce our experimental setup, which includes implementation details of el methods we investigated ( §4.1) and datasets we create for investigating partial kb inference ( §4.2). ## results in this section, we present the main results of partial kb inference ( §5.1) then, we provided two redemption methods for enhancing model performance in partial kb inference ( §5.2). in the end, we discuss the factors related to difficulties hindering partial kb inference performance ( §5.3). ## conclusion in this work, we propose a practical scenario, namely partial kb inference, in biomedical el and give a detailed definition and evaluation procedures for it. we review and categorize current state-ofthe-art entity linking models into three paradigms. through experiments, we show ner-ned and simultaneous generation paradigms have vulnerable performance toward partial kb inference which is mainly caused by mention detection precision drop. the ned-ner paradigm is more robust due to well-modeled mention-concept reliance. we also propose two methods to redeem the performance drop in partial kb inference and discover out-kb annotations may enhance the in-kb performance. post-pruning and thresholding can both improve the performance of ner-ned and simultaneous generation paradigms. although post-pruning is easy-to-use, it needs to store the large kb e 1 (with their embeddings or trie) which has large memory consumption. thresholding does not rely on large kb e 1 which also has better performance on the ner-ned paradigm. our findings illustrate the importance of partial kb inference in el which shed light on the future research direction.