Generative Biomedical Entity Linking via Knowledge Base-Guided Pre-training and Synonyms-Aware Fine-tuning (2022)

Abstract: Entities lie in the heart of biomedical natural language understanding, and the biomedical entity linking (EL) task remains challenging due to the fine-grained and diversiform concept names. Generative methods achieve remarkable performances in general domain EL with less memory usage while requiring expensive pre-training. Previous biomedical EL methods leverage synonyms from knowledge bases (KB) which is not trivial to inject into a generative method. In this work, we use a generative approach to model biomedical EL and propose to inject synonyms knowledge in it. We propose KB-guided pre-training by constructing synthetic samples with synonyms and definitions from KB and require the model to recover concept names. We also propose synonyms-aware fine-tuning to select concept names for training, and propose decoder prompt and multi-synonyms constrained prefix tree for inference. Our method achieves state-of-the-art results on several biomedical EL tasks without candidate selection which displays the effectiveness of proposed pre-training and fine-tuning strategies. The source code is available at https://github.com/Yuanhy1997/GenBioEL .

## introduction biomedical entity linking (el) refers to mapping a biomedical mention (i.e., entity) in free texts to its concept in a biomedical knowledge base (kb) (e.g., umls @xcite ). this is one of the most concerned tasks of research in medical natural language processing which is highly related to high-throughput phenotyping @xcite , relation extraction @xcite , and automatic diagnosis @xcite . recent methods in biomedical el mainly used neural networks to encode mentions and each concept name into the same dense space, then linked mentions to corresponding concepts depending on embedding similarities @xcite @xcite @xcite . synonyms knowledge has been injected into these similarity-based methods by contrastive learning @xcite . for example in umls, concept c0085435 has synonyms: reiter syndrome, reactive arthritis and rea which help models to learn different names of a concept entity. however, similarity-based methods requires large memory footprints to store representation for each concept which are hard to deploy. in general domain, @xcite ) viewed el as a seq2seq task which inputs mentions with contexts and outputs concept names tokenby-token. mentions, contexts and concepts can be mingled due to the power of transformers, and the model does not need to store representations for each concept during inference. genre pretrained on wikipedia el datasets to boost performances. however, directly implementing genre on biomedical el cannot harvest satisfying results. the gap occurs in two aspects: (1) there are no such large-scale human-labeled biomedical el datasets for pre-training. (2) biomedical concepts may have multiple synonyms. we find the results are sensitive to the synonyms selection for training, and simply using a 1-to-1 mapping between names and concepts as @xcite may hurt performances. to address the above issues, we propose kbguided pre-training and synonyms-aware finetuning to improve generative el. for pre-training, we construct pre-training samples using synonyms and definitions collected from kbs and sentence templates. kb-guided pre-training has the same format as seq2seq el, which fills the gap of loss of pre-training corpus. compared to the method introduced in @xcite , ours performs better in biomedical el with fewer resources. for fine-tuning, we propose decoder prompts to highlight mentions. we find the model tends to generate textually similar names to mentions. hence textual similar criterion is proposed for selecting concept names during fine-tuning. during inference, we propose a multi-synonyms constrained prefix tree, which results in significantly improved performance. the overview of our approach is illustrated in figure 1 . we conduct experiments on various biomedical el datasets and achieve sota results on cometa, bc5cdr, and askapatient (aap) even without candidate selection. extensive studies show the effectiveness of our proposed pre-training and fine-tuning schemes. ## approach define a set of concepts e as target concepts (i.e. concepts from target kbs). for each concept @xmath0, we have a set of synonyms names f (e) = {s i e |@xmath1, . . . , n e }}. all the synonyms forms a name set @xmath2, . . . , n e }}. names-toconcept mappings can be defined by:σ(s i e ) = e where σ = f - foot_0 . for mention m with left and right contexts c l and c r which gold label is e @xmath3, we need to find the target concept êm ∈ e. m, c and s comprise a sequence of tokens. ## conclusion to the best of our knowledge, our work is the first to explore generative el in the biomedical domain. we inject synonyms and definition knowledge into the generative language model by kgguided pre-training. we emphasize the synonym selection issue and propose synonyms-aware finetuning by considering the textual similarity. decoding prompts are also introduced to improve the model's performance. our model sets new stateof-the-art on different biomedical el benchmarks. genre shows that well-selected candidate sets can improve seq2seq el, and we believe this will further boost our performances.