Biomedical Entity Representation with Graph-Augmented Multi-Objective Transformer (2024)

Abstract: Modern biomedical concept representations are mostly trained on synonymous concept names from a biomedical knowledge base, ignoring the inter-concept interactions and a concept’s local neighborhood in a knowledge base graph. In this paper, we introduce Biomedical Entity Representation with a Graph-Augmented Multi-Objective Transformer (BERGAMOT), which adopts the power of pre-trained language models (LMs) and graph neural networks to capture both inter-concept and intra-concept interactions from the multilingual UMLS graph. To obtain fine-grained graph representations, we introduce two additional graph-based objectives: (i) a node-level contrastive objective and (ii) the Deep Graph Infomax (DGI) loss, which maximizes the mutual information between a local subgraph and a high-level graph summary. We apply contrastive loss on textual and graph representations to make them less sensitive to surface forms and enable intermodal knowledge exchange. BERGAMOT achieves state-of-the-art results in zero-shot entity linking without task-specific supervision on 4 of 5 languages of the Mantra corpus and on 8 of 10 languages of the XL-BEL benchmark.

## introduction biomedical concepts, such as diseases, symptoms, drugs, genes, and proteins, are critical for many biomedical applications, including drug discovery @xcite @xcite , clinical decision making @xcite , and biomedical research @xcite @xcite @xcite . the same biomedical concept may have multiple nonstandard names, abbreviations, and misspellings. medical concept normalization (mcn), also known as medical concept linking, is a task of mapping entity mentions to a large set of medical concept names and their unique identifiers (cuis) from a knowledge base (kb). the biomedical domain is characterized by extensive kbs such as the unified medical language system (umls) @xcite , which includes over 166 lexicons/thesauri with over 4m concepts and 15m concept names in 27 languages. the development of meaningful and robust biomedical entity representations continues to be a challenging task for language models (lms). recent studies have probed lms trained on biomedical texts in english and discovered that domainspecific pre-trained language models (plms), such as biobert @xcite and @xcite , exhibit high levels of bias and lack synonym awareness @xcite . for spanish, the second language by number of concept names in umls, plms pre-trained on clinical data fall short compared to the simplistic sparse baseline in the mcn task @xcite . textual triples from a kb are commonly used to incorporate knowledge into neural networks with metric learning and contrastive learning framework @xcite @xcite @xcite . positive and negative pairs are created using head and tail terms of the same or different concepts, as illustrated in fig. 1 , where maux de tête is a french synonym of headache but differs from sharp headache (headache is a broader concept for sharp headache). graph-based representations are another way to represent biomedical knowledge with concepts as nodes and relationships as edges. inspired by semantic matching methods like transe @xcite and dist-mult @xcite , @xcite proposed a method to integrate term-relation-term similarities into backbone lm. however, this approach doesn't fully utilize inter-concept interactions since it learns from individual relation triplets rather than performing an aggregation over the whole concept's local neighborhood in knowledge graph (kg) described by umls kb. in this paper, we present biomedical entity representation with graph-augmented multi-objective transformer (bergamot) which utilizes plms and graph neural networks (gnns) to capture inter-concept and intra-concept interactions from the multilingual umls graph. as shown in fig. 1 , the bergamot architecture includes four losses: (i) a textual term-term contrastive loss that learns from positive and negative concept name pairs; (ii) a node-node contrastive loss that learns on concept nodes to distinguish between nodes based on their local subgraphs in larger kg; (iii) dgi loss that lets a gnn distinguish between factually accurate (present in the knowledge graph) concept subgraphs and corrupted ones; (iv) an intermodal contrastive loss that enables mutual information exchange between textual and graph encoders. the source code and pre-trained models are freely available at: https: //github.com/andoree/bergamot . ## related work mcn and entity representations there are several conventional approaches to address an mcn problem. the most common is the entity classification @xcite into a small number of target concepts. however, umls and similar kbs may include millions of different concepts arranged in a hierarchical structure. another popular approach is ranking mentions or con-cepts by the mutual similarity term learnt from positive and negative pairs on some corpora @xcite . aside from mcn methods, a plethora of works are focused on features and representations of entity mentions based on syntax, morphology, and synonyms @xcite @xcite . @xcite chose a straightforward convolutional layer as an encoder. the network was trained with the triplets of a disease mention, as well as positive and negative concept candidates. the supervised biosyn model @xcite maximizes the likelihood of synonym appearance among the most similar 20 terms. morphology was encoded with a character-level tf-idf representation to obtain a sparse similarity score. the distance between biobert @xcite ) cls tokens was utilized as a high-level dense similarity. the final similarity score is a weighted sum of both sparse and dense similarities. dilbert @xcite introduces novel negative sampling strategies for a triplet loss, utilizing the hierarchical structure of the umls. both biosyn and dil-bert are limited to a single language (english) and a small concept subset within a specific terminology. entity representation learning may be augmented with external knowledge from domainspecific kbs @xcite @xcite . @xcite proposes encoding contextual meaning, conceptual meaning, and the similarity between synonyms during the representation learn-ing process. two novel training objectives are forcing the similarity between the representation of a named entity and its synonym or target concept, respectively. despite the promising results, the ranking based on representations from this model performs worse than a plain dictionary-based baseline. umlsbert @xcite discussed a novel knowledge augmentation strategy to utilize domain-specific knowledge from umls during a model's pretraining phase. sapbert @xcite model takes advantage of a self-alignment pretraining (sap) on the umls synonymous pairs. the resulting bert-like model outperforms its predecessors (biobert, scibert, umlsbert). coder, using the umls graph and a relational loss with sap, outperforms sapbert on the mcn task @xcite . ## experimental evaluation to train bergamot, we use the umls 2020ab release which contains approximately 4.4 million concepts and 15.9 million unique concept names from 215 source vocabularies. the full statistics on pre-training multilingual and monolingual data as well as the number of concept names for languages are shown in appendix a. we remove all duplicated edges (i.e., edges with matching source and target concepts, and relation type). to ensure each batch includes a sufficient amount of positive samples, we pre-generate synonymous concept name pairs following @xcite . since random sampling could result in the underrepresentation of languages other than english, we explicitly add (i) monolingual non-english and (ii) cross-lingual positive concept name pairs to each batch. however, the quadratic growth in the number of possible language pairs with respect to the number of languages limits our ability to enrich the training dataset with multilingual data. we discard all cross-lingual pairs which consist of two non-english terms thus forcing all languages to benefit from extensive english knowledge. to explore how the bergamot's performance is affected by a monolingual low-resource setting, we additionally subsample a monolingual subgraph and generate a monolingual set of positive pairs for spanish, dutch, french, and german. we create each dataset by removing all terms that came from a non-target language and all nodes that have no concept name in a target language. for statistics on positive textual pairs count, number of edges and nodes, please see tab. 8 in appx b. for evaluation on the mcn task, we use two cross-lingual benchmarks: mantra corpus covers mentions in english, french, german, spanish, and dutch while xl-bel covers 10 languages. both benchmarks allow zeroshot evaluation only, i.e., there are no training sets. additionally, we employ the french quaero corpus @xcite and two datasets in spanish: (i) @xcite ) and (ii) cantemist @xcite . @xcite introduced a novel filtering procedure which drops test set mentions that are identical to a term from the dictionary. for dataset descriptions and statistics, refer to appendix b. to explore the ability of our model to solve diverse non-mcn nlp tasks, we additionally evaluate our model on biomedical question answering (qa) and textual entailment (te) tasks. we conducted qa experiments on two datasets: (i) pub-medqa @xcite and bioasq @xcite . the goal of te task is to determine a logical relationship between two pieces of text: a premise and a hypothesis. for te experiments, we utilize two corpora: (i) mednli @xcite and (ii) scitail @xcite ## conclusion and future work we presented bergamot, a graph-augmented architecture with backbone lm designed to learn inter-concept and intra-concept interactions from the multilingual knowledge graph. bergamot outperforms existing language models pre-trained on knowledge triples from umls on two concept normalization benchmarks with a diverse set of languages. since bergamot is currently an encoder model and it is not able to generate texts, an important future research would be to extend bergamot to include language generation capabilities and advance kg-enhanced language generation. the work was supported by a grant for research centers in the field of artificial intelligence, provided by the analytical center in accordance with the subsidy agreement (agreement identifier 000000d730321p5q0002) and the agreement with the ivannikov institute for system programming of dated november 2, 2021 no. 70-2021-00142. we specially thank zulfat miftahutdinov for useful discussions and feedback pointing out issues in a previous iteration of the paper. we thank anonymous reviewers for valuable feedback.