B io LORD : Learning Ontological Representations from Definitions for Biomedical Concepts and their Textual Descriptions (2022)

Abstract: This work introduces BioLORD, a new pre-training strategy for producing meaningful representations for clinical sentences and biomedical concepts. State-of-the-art methodologies operate by maximizing the similarity in representation of names referring to the same concept, and preventing collapse through contrastive learning. However, because biomedical names are not always self-explanatory, it sometimes results in non-semantic representations. BioLORD overcomes this issue by grounding its concept representations using definitions, as well as short descriptions derived from a multi-relational knowledge graph consisting of biomedical ontologies. Thanks to this grounding, our model produces more semantic concept representations that match more closely the hierarchical structure of ontologies. BioLORD establishes a new state of the art for text similarity on both clinical sentences (MedSTS) and biomedical concepts (MayoSRS).

## introduction natural language processing models are well positioned to support healthcare providers by automatically extracting and synthesizing relevant information from clinical notes. for this, we introduce biolord, a pre-training strategy for end-to-end biomedical information extraction, capable of producing meaningful representations for biomedical terms and clinical sentences simultaneously. this is achieved through the continued pretraining of an existing sentence embedding model, using contrastive learning and pairs consisting of the names and definitions of a given biomedical concept (see fig. 1 ). this design choice proved crucial for the effectiveness of biolord, as it enables the transfer of knowledge from the definitions to the representation of biomedical names, thereby overcoming limitations of existing works (see §2.3) through a more effective usage of the knowledge contained in biomedical ontologies (see §2.1). indeed, to improve coverage and diversity, we supplemented definitions with textual descriptions generated from the numerous concept-to-concept relationships contained in biomedical ontologies. our key contributions are 1 a versatile training strategy using dictionaries and knowledge graphs to create highly semantic representations for the key phrases of a domain, 2 an associated biolord model trained on the biomedical domain, 3 an extensive evaluation ( §4) demonstrating its ability to provide semantic representations usable in a broad range of information extraction scenarios, including a new state of the art for biomedical concept representation and clinical sentence similarity, and 4 an in-depth analysis of the strengths and weaknesses of our proposed approach ( §5). ## related work let us first consider how prior works attempted to address the biomedical domain's usage of a large, specialized, and often opaque vocabulary (e.g., papa syndrome foot_0 or cat scratch disease foot_1 ). ## pre-training methodology to produce representations of biomedical concepts that overcome the limitations described above, we modified the way the positive pairs are constructed. like the prior works cited in §2.2, we start by establishing a list of names for each umls concept. however, unlike previous works, we do not use these names directly to form positive pairs. instead, we construct pairs formed with, on the one side, a randomly selected name for a given concept and, on the other side, a definition or description for that concept (see fig. 1 ). we hypothesize that a definition or description of a given concept provides a more robust semantic anchor for this concept than another of its names. as mentioned before, names in the medical domain can be quite opaque, and do not always offer useful insights into what exactly is being referred to. by inducing representational similarity between a concept name and its known definitions, we aim to distill their respective knowledge into the representations of the concept names themselves. this key idea influenced some design choices for our experimental setup, including the choice of the data curation process, model initialization, and training procedure (as described in this section). ## huggingface.co/datasets/fremycompany/biolord-ds 4 experimental evaluation to demonstrate the effectiveness of our definitionbased pre-training strategy, we subsequently train and evaluate our model on multiple semantic text similarity (sts) tasks. for each benchmark, we compare our results with the state of the art. we also report results for the models biosyn @xcite and sapbert @xcite , since these share their encoder architecture (bert-base) and parameter count (110m) with our model, enabling fair comparison. we also provide results for the base models of biosyn, sapbert, and bio-lord: respectively biobert @xcite , pubmedbert @xcite and stamb2 @xcite . when applicable, we use the same finetuning strategy for all six models. our key evaluation tasks are biomedical concept similarity, biomedical concept normalization, and sentence similarity (both in and out of the biomedical domain). in the following paragraphs, we describe the datasets and the experimental setup for these tasks, and already refer to the respective results tables. we however reserve our thoughts and insights for §5 to enable cross-task comparisons. ## discussion thanks to the experiments described in the previous section, the suitability of definition-based contrastive learning can now be demonstrated, which we will do in this section. we also discuss patterns appearing in our results, which might be useful in designing further improvements to our technique. our pre-training strategy significantly improves the alignment with human judgement of biomedical entity representations (+27.6pts on mayosrs compared to stamb2, our initialization model; see unlike both sapbert and biosyn, biolord's performance on general-purpose sentence representations remains competitive. it is however worth noting that the aforementioned pre-training strategies seem to impair general-purpose sentence representation, even after fine-tuning (see in the specific case of biomedical sentences, the biolord pre-training nonetheless remains a netpositive. thanks to this, we achieve state-of-theart results for bi-encoders on medsts (86.3pts, +0.4pts over initialization), and we can report improvements over initialization on mednli-s as well (89.9pts, +0.5pts). ## limitations while biolord models perform better on generalpurpose text than other state-of-the-art biomedical models, we would like to recognize that no guarantee can be made regarding their ability to make sense of non-ontological modalities of languages in the biomedical domain (spoken language, social media, online forums, etc. . . ). this might cause issues in circumstances where the content being processed is not originating from clinicians. biolord models are not well suited to tasks requiring the latest scientific knowledge, for example found in pubmed papers, because the origin of most of their biomedical knowledge is ontologies. this type of knowledge might be better encoded in pubmedbert-based models. hybrid approaches might therefore be worthwhile for natural language inference and scientific paper analysis scenarios. another limitation worth mentioning is that, unlike graph neural networks, biolord models do not provide a single and unique embedding for each concept of an ontology, because concepts might have more than one known name. as representations are produced for names rather than concepts, each name will produce a different (but hopefully close) representation for a concept. this might amper rule-based decision processes. despite these limitations, we envision that most use cases traditionally considered in the biomedical domain are suitable for the usage of biolord models, in an end-to-end setup. ## conclusion the results detailed in this paper demonstrate that our name-to-definition contrastive pre-training strategy (biolord) is able to produce bi-encoder models with a state-of-the-art performance for both biomedical concept similarity and clinical sentence similarity, using one set of weights. as part of our investigation, we were also able to show that appropriate finetuning strategies can enable general-purpose models (like stamb2) to fully overcome their initial disadvantage at biomedical text understanding compared to models of the same size trained on biomedical data only (such as pubmedbert). this can be achieved while preserving most of their pre-existing general-purpose text understanding capabilities. by releasing a bi-encoder model able to embed both clinical sentences and biomedical concepts in the same latent space, we immensely simplify some clinical language processing scenarios, like clinical note retrieval and diffuse information extraction. for this reason, we hope that future works in the biomedical domain will continue to combine clinical sentences and biomedical concept benchmarks to produce directly usable end-to-end models.