Evaluating Biomedical Word Embeddings for Vocabulary Alignment at Scale in the UMLS M etathesaurus Using S iamese Networks (2022)

Abstract: Recent work uses a Siamese Network, initialized with BioWordVec embeddings (distributed word embeddings), for predicting synonymy among biomedical terms to automate a part of the UMLS (Unified Medical Language System) Metathesaurus construction process. We evaluate the use of contextualized word embeddings extracted from nine different biomedical BERT-based models for synonym prediction in the UMLS by replacing BioWordVec embeddings with embeddings extracted from each biomedical BERT model using different feature extraction methods. Finally, we conduct a thorough grid search, which prior work lacks, to find the best set of hyperparameters. Surprisingly, we find that Siamese Networks initialized with BioWordVec embeddings still out perform the Siamese Networks initialized with embedding extracted from biomedical BERT model.

## introduction the umls @xcite is a biomedical terminology integration system that includes over 200 source vocabularies 1 . the umls metathesaurus construction process organizes synonymous terms from these source vocabularies into concepts. the current metathesaurus construction process uses a lexical similarity model and semantic preprocessing to determine synonymy, followed by a human review. the large scale and diversity of the metathesaurus make the construction process very challenging, tedious, and error-prone. therefore, to assist the umls metathesaurus construction process, nguyen et al. introduced the umls vocabulary alignment (uva) task, or synonymy prediction task @xcite . they designed and train a siamese network to predict if two umls atoms are synonymous. the siamese network is initialized using biowordvec embeddings, learned using fasttext @xcite . given the recent successful use of contextualized word embeddings, extracted from transformer models, for different downstream nlp tasks @xcite @xcite , we explore the use of contextualized 1 https://uts.nlm.nih.gov/uts/ embeddings extracted from several distinct biomedical bert-based language models. objectives. 1) find which type of word embeddings, including contextualized embeddings, achieves the best performance when used with the siamese network for the synonymy prediction (or uva) task. 2) find which feature extraction method works best to extract word embeddings from the biomedical bert models for optimal performance. 3) find the best hyperparameters and optimization of the prediction task to train the siamese networks for the uva task. approach. 1) we analyze the performance of the siamese networks initialized with embeddings from nine different biomedical bert models for synonymy prediction. 2) we explore different feature extraction techniques to extract bert embeddings. 3) we conduct a grid search and optimization of the prediction task to train the siamese networks. contributions. 1) we conduct an extensive analysis to extract embeddings from nine different biomedical bert models using four feature extraction techniques. 2) somewhat surprisingly, we find that siamese networks still achieve the highest performance for synonymy prediction when initialized with biowordvec embeddings. 3) we find that no single feature extraction method works well across the different biomedical bert models. 4) with a thorough grid search, we find substantial increases in f1-score (e.g., 2.43%), when compared to the default hyperparameters. 5) overall, our work contributes to defining best practices for the use of embeddings in siamese networks. see https: //arxiv.org/abs/2109.13348 for an extension of this paper as it presents an extended analysis of the experiments and additional results. ## umls -knowledge representation the umls metathesaurus links terms and codes between health records, pharmacy documents, and insurance documents @xcite . the metathesaurus consists of several building blocks, including atoms and concepts. all atoms in the umls metathesaurus are assigned a unique identifier (aui). atoms that are synonymous are grouped into a single concept identified with a concept unique identifier (cui). tuple atom string source aui cui t 1 headache msh a0066000 c0018681 t 2 headaches msh a0066008 c0018681 t 3 cephalodynia msh a26628141 c0018681 t 4 cephalodynia snomedct us a2957278 c0018681 we use the 2020aa version of the umls, which contains 15.5 million atoms from 214 source vocabularies grouped into 4.28 million concepts. ## problem formulation an essential part of the umls construction process is identifying similar atoms across source vocabularies to integrate knowledge from different sources accurately. the umls vocabulary alignment (uva)or synonymy prediction -task is to identify synonymous atoms by measuring the similarity among pairs of atoms. a machine learning model should be able to identify the synonymous atoms are that lexically: similar but are not synonymous and dissimilar but are synonymous. let (t i , t j ) be a pair of input tuples, where i ̸ = j. each tuple is initialized from a different source vocabulary in the form of (str, src, aui), where str is the atom string, src is the source vocabulary, and aui is the atom unique identifier (aui). let f : t × @xmath0, 1 be a prediction function that maps a pair of input tuples to either 0 or 1. if f (t i , t j ) = 1, then the atom strings (str i , str j ) from t i and t j are synonymous and belong to the same concept (and hence, share same the cui). ## dataset we thank nguyen et al. for sharing the dataset used in their work @xcite . the dataset is created using the 2020aa release of the umls metathesaurus. we use the all dataset for our study. the training and validation dataset contains a total of 192,400,462 examples, where 88.4% of the examples are negative examples. the testing dataset set contains a total of 173,035,862 examples, where 96.8% of the examples are negative examples. we refer the readers to section 4.2 of @xcite for a detailed description. we first describe the siamese networks for the uva then describe the biomedical bert variants. siamese networks for the uva task nguyen et al. assess the similarity of atoms using lexical features of the atom strings (str). the authors design a siamese network that inputs a pair of atom strings, and outputs a similarity score between 0 and 1, sim(str i , str j ) ∈ [0, 1] (see figure 1 ). the inputs are preprocessed, tokenized, and then sent through an initial embedding layer initialized with biowordvec embeddings @xcite . the word embeddings are then fed into bidirectional long short term memory (bi-lstm) layers, followed by two dense layers. all atom pairs with a @xmath1.5 are considered synonyms (using the manhattan distance). their deep learning model has a precision of 94.64%, recall of 94.96% and an f1-score of 94.8% and outperforms a rule-based approach for synonymy prediction by 23% in recall, 2.4% in precision, and 14.1% in f1-score. in their follow-up work, nguyen et al. add an attention layer after the bi-lstm layers that improves the precision by +3.63% but decreases recall by 1.42%. ## approach to analyze the performance of the different embeddings extracted from the various bert models, we train the siamese network end to end, similar to @xcite . we investigate the use of the nine biomedical bert models (mentioned in section 5) as a source of word embeddings. our experimental setup of consists of two primary steps for each of the siamese networks (with and without attention): 1) feature extraction of word embeddings from biomedical bert models. 2) grid search of optimal hyperparameters and optimization. our code will be available at https://anonymous.4open.science/r/uva embedding benchmarking-8124/. for the training and testing data, we recommend reaching out to nguyen et al. @xcite . ## conclusion we investigate if contextualized embeddings extracted from biomedical bert-based language models can improve the performance of siamese networks, introduced by @xcite , to predict synonymy in the umls metathesaurus. despite the excellent performance of bert models on biomedical nlp tasks, biowordvec embeddings still remain competitive for the uva task. this confirms the importance of investigating the use of traditional distributed word embeddings. among the biomedical bert models, sapbert trained on umls data performs best, suggesting the importance of using a model trained on datasets directly relevant to the task at hand. finally, we demonstrate the importance of exploring different feature extraction methods and hyperparameter tuning for deep learning models.