Cross-Lingual UMLS Named Entity Linking using UMLS Dictionary Fine-Tuning (2022)

Abstract: We study cross-lingual UMLS named entity linking, where mentions in a given source language are mapped to UMLS concepts, most of which are labeled in English. Our cross-lingual framework includes an offline unsupervised construction of a translated UMLS dictionary and a per-document pipeline which identifies UMLS candidate mentions and uses a fine-tuned pretrained transformer language model to filter candidates according to context. Our method exploits a small dataset of manually annotated UMLS mentions in the source language and uses this supervised data in two ways: to extend the unsupervised UMLS dictionary and to fine-tune the contextual filtering of candidate mentions in full documents. We demonstrate results of our approach on both Hebrew and English. We achieve new state-of-the-art (SOTA) results on the Hebrew Camoni corpus, +8.9 F1 on average across three communities in the dataset. We also achieve new SOTA on the English dataset MedMentions with +7.3 F1.

## introduction public health practices are becoming increasingly digital, with tools to explore scientific sources of information such as medical literature and online health communities rising in popularity. such tools are essential in offering insights to researchers, providing information to patients and to their caregivers. reliable identification of mentions of biomedical concepts in free text is a key technique to enable robust mining of such textual resources. named-entity recognition (ner) is the task of classifying entities in text to high level classes @xcite . named-entity linking (nel) seeks to additionally classify entity mentions in text into specific concepts according to an existing reference list or knowledge base. we focus in this work on biomedical nel, i.e., identifying mentions referring to biomedical concepts such as disorders and drugs and linking them to normalized concepts, for example, concept unique identifiers (cuis) listed in the unified medical language system (umls) controlled vocabulary. biomedical nel has been mostly studied in english. other languages present additional challenges because terms in the ontology are described in english. we address crosslingual nel which consists of mapping mentions in a source language to concepts labeled and described in a different target language. we focus on cross-lingual umls nel, where mentions in the source language (we specifically test hebrew, see figure 1 for a hebrew tagging example) are mapped to umls concepts. we aim for a general solution that can be adapted to any source language. we operate in a low resource setting, where the ontology is large, text describing most entities is not available, and labeled data can only cover a small portion of the ontology. we also consider different genres of text to be annotated, ranging from consumer health medical articles in popular web sites to scientific biomedical articles. our main contributions are: (1) we provide a general framework for cross-lingual umls nel that can be adapted to source languages with few pre-requisites; (2) our method exploits a small annotated corpus of documents in the source language and genre annotated manually for umls mentions (a few thousands annotated mentions). this training data is split to support (a) the extension of the unsupervised umls dictionary with corpus-salient entity names and (b) fine-tune the contextual ranking and filtering of (candidate mentions, concept) pairs. we find that the step of umls dictionary fine-tuning boosts nel performance and identify a clear tradeoff in allocating training data between lexicon extension and contextual fine-tuning; (3) we demonstrate results of our approach on both hebrew and english. we achieve new sota on the hebrew camoni corpus @xcite with +8.87 f1 and on the english dataset medmentions @xcite with +7.3 f1 foot_0 . ## previous work biomedical nel is challenging because the underlying ontology (most often umls) is extremely large and the acquisition of annotated training data requires rare and expensive expertise. @xcite presented medlinker, a tool for improving biomedical nel by predicting the semantic type of a medical concept mention and filtering out candidates of the wrong type. medlinker was tested on the medmentions task of concept linking @xcite , improving above taggerone @xcite , the baseline model for medmentions which did not use deep learning. medlinker splits the end to end task of entity linking into two stages -candidate recognition and linking. for candidate matching, it combines a bilstm-crf model for contextual matching with an approximate dictionary matching method to increase recall. in the cross-lingual setting, dictionary matching is not applicable. we report our results on the same medmentions dataset in section 5.2. past work has shown that using in-domain text can provide additional gains over general-domain language models @xcite . therefore, recent work (biobert @xcite , scib-ert @xcite ) addressed biomedical nel, focusing on pre-training models on scientific/medical text. @xcite developed sap-bert, a pre-training scheme which exploits the graph structure of the umls controlled vocabulary and aims at learning an encoding of medical mentions that can align with synonym relations in the umls graph. combining the sapbert objective with pre-training on biomedical text of pubmed-bert @xcite boosts results on nel. experimental results demonstrated that sapbert outperforms many domain-specific bert-based variants (biobert and scibert) on the bc5cdr (biocreative v cdr) corpus. although our model focuses on cross-lingual nel, it also applies to english documents. we compare our results to these approaches on bc5cdr and medmentions (tables 4 and 3 ). indexing of the abundant biomedical scientific literature requires precise detection of medical concepts. @xcite developed a lowresource recognition and linking model of biomedical concepts (henceforth referred to as lrr) aimed at generalizing to entities unseen at training time, and incorporating linking predictions into the mention segmentation decisions. this bert-based model achieved sota results on the medmentions task. in our work, we adopt the lrr bottom-up candidate generation approach (see section 4.2). we address the main drawback of the approach by incorporating a umls dictionary fine-tuning technique which extends the list of candidate pairs (source expression, cui) on a portion of the training data. we elaborate on the motivation for the technique in section 4.5 and demonstrate its contribution in ablation experiments (see section 5.4). cross-lingual nel, the problem of grounding mentions of entities in a source language text into a different target language knowledge base (typically english), has been addressed in recent years, with a range of promising techniques. when the source and target languages operate over different alphabets and sound systems, both translation and transliteration of terms (which is a noisy process even when done by people) must be handled. @xcite curated the camoni corpus, an annotated resource of hebrew posts from online health communities (ohcs), where noisy text (as opposed to scientific text) introduces additional challenges. many user queries mention medical terms, which are very likely to include noisy transliterations. for example, the hebrew query equivalent to "how do i know i have fibromyalgia?" does not return any results in the search engine of the camoni online community when 'fibromialgia' is transliterated. @xcite introduced mdtel (medical deep transliteration entity linking) for hebrew-english nel on noisy text in ohcs, and tested it on the camoni corpus. mdtel adopts a fourstep approach -consisting of an offline unsupervised hebrew umls dictionary learning, candidate mention generation, high-recall matching and filtering of matching mentions. we adopt mdtel's unsupervised umls dictionary matching, which uses an attention-based recurrent neural network encoder-decoder that maps umls from english to hebrew (either a hebrew translation or translit-figure 1 : a forum post from the camoni sclerosis community that translates to: "hello, recently, my gait has deteriorated and i was suggested to begin botox treatment to release the muscles and prevent spasticity. has anyone here undergone such treatment? does it help? is there a risk that such treatment will greatly weaken the muscle, causing the exact opposite action?". the post contains 37 words and 6 spans that link to 4 different cuis of umls concepts. notice that a span can consist of more than 1 word (such as the term matched with "gait abnormality") and a single cui can be referenced from several places in the same post (such as the cui of "general treatment"). ## task formulation given input language l and target language l t , a database of medical concepts c lt : l * @xmath0. using c lt , we want to learn a function f from a span in input language l and its context to a cui. we identify dictionary c l : l * → cu i. c l is the translated version of the medical concepts database c lt . we learn c l by mapping the medical terms in l t to terms in l. given mapping c l , we aim to learn: ## model architecture our end-to-end cross-lingual umls nel model (figure 2 ) consists of four consecutive stages: (1) multilingual umls mapping: generate umls dictionary c l (see section 4.1) based on the method of @xcite , and fine-tune it using our umls dictionary fine-tuning technique (see section 4.5); (2) candidate generation: consider all spans of up to k words as candidate mentions and compute vector representations for both mentions and concepts (see section 4.2); (3) high recall matching: use a semantic similarity based score function to generate the top matching entities with high recall (see section 4.3) and ( 4 ) contextual relevance modeling: encode each candidate into a context-dependent vector representation using a pre-trained transformer-based language model fine tuning process (see section 4.4). our approach attempts to avoid three types of mistakes: (1) morphological and transliteration noise, where candidate terms in the source language might be extracted due to a transliteration or morphological error and matched with umls entities, (2) contextual errors, where candidate terms which are not medical terms when considering the context might be matched with umls entities, and (3) partial umls tagging, where candidate terms which are not the full medical terms in the text but rather more general umls mentions might be tagged instead of the full term (e.g., in the mention "flu vaccine", "flu" should not be tagged). the first challenge is addressed by learning a high-recall c l dictionary with generalization capabilities, trained both on translation and transliteration data; the second, is addressed by an mbert-based contextual language model; the third, by systematic consideration of all spans up to size k as candidates as part of the candidate generation and contextual relevance components. ## experiments we test our approach both on cross-lingual umls linking using the camoni dataset of hebrew consumer health data and on english umls linking using medmentions and bc5cdr, which include scientific papers in the bio-medical field. ## conclusion in this work we explored the task of cross lingual named entity linking in the biomedical field. we describe a pipeline to detect and link mentions of umls concepts in documents in hebrew or in english, which improves upon existing methods. the model dataset f1 % biobert @xcite bc5cdr 88.6 scibert ( beltagy et al., 2019) bc5cdr 90.0 sapbert @xcite bc5cdr-d 93.5 our model bc5cdr 73.0 we compared our performance to baseline approaches on the camoni dataset in hebrew @xcite , and the medmentions @xcite and bc5cdr english datasets. our end-toend approach achieves sota results on camoni in hebrew and medmentions in english with significant improvements. for bc5cdr, we observe that the small size of the dataset prevents our dictionary augmentation technique from reaching its potential and models trained on specialized biomedical text (pubmedbert with sapbert training objective) obtain better coverage. such specialized training is, however, not available in a multi-lingual setting. dataset umls dft added concepts recall % medmentions subset ✗ 0 62.7 medmentions subset ✓ 227 64.4 for future work, we intend to test whether utilizing language-specific bert models instead of multilingual bert (e.g., swapping m-bert with the recently released alephbert @xcite , a hebrew version of bert) could improve results on the hebrew camoni corpus. in addition, taking into account the sapbert objective which exploits the umls graph structure as part of either fine-tuning or pre-training in hebrew could lead to improved generalization capabilities. finally, exploring datasets with additional source languages will help understand the capabilities of our multilingual pipeline. the clef ehealth challenges @xcite are good candidates for such analysis. figure 6: distribution of medmentions mention and umls lengths (in words) figure 7: distribution of bc5cdr mention and mesh id lengths (in words) hp description value m top matches parameter of the high recall matcher (section 4.3) 50 th threshold of selecting possible matched concepts for the spans (section 4.3) 0.4 w s window size per side of the candidate mention (section 4.4) 2 r umls dictionary fine-tuning percentage (section 4.5) 20 -the model's learning rate 2e -5 -train epochs 3 -batch size 32