Integrating Higher-Level Semantics into Robust Biomedical Name Representations (2021)

Abstract: Neural encoders of biomedical names are typically considered robust if representations can be effectively exploited for various downstream NLP tasks. To achieve this, encoders need to model domain-specific biomedical semantics while rivaling the universal applicability of pretrained self-supervised representations. Previous work on robust representations has focused on learning low-level distinctions between names of fine-grained biomedical concepts. These fine-grained concepts can also be clustered together to reflect higher-level, more general semantic distinctions, such as grouping the names nettle sting and tick-borne fever together under the description puncture wound of skin. It has not yet been empirically confirmed that training biomedical name encoders on fine-grained distinctions automatically leads to bottom-up encoding of such higher-level semantics. In this paper, we show that this bottom-up effect exists, but that it is still relatively limited. As a solution, we propose a scalable multi-task training regime for biomedical name encoders which can also learn robust representations using only higher-level semantic classes. These representations can generalise both bottom-up as well as top-down among various semantic hierarchies. Moreover, we show how they can be used out-of-the-box for improved unsupervised detection of hypernyms, while retaining robust performance on various semantic relatedness benchmarks.

## introduction recent work on representation learning for biomedical names has mainly involved the training of neu-ral encoder architectures such as lstms @xcite or transformers @xcite to finetune name representations for biomedical normalization tasks. such representations are often tailored towards normalization tasks (e.g. linking names to corresponding concept identifiers), without providing explicit guarantees about their transferability to other use contexts and applications. as a solution for this issue, the biomedical name encoder (bne) model @xcite has been proposed as a comprehensive framework for robust and transferable representations. according to this framework, the robustness of biomedical name representations is characterized along three dimensions. firstly, semantic similarity between names should be reflected by their closeness in the embedding space. secondly, the variety of textual contexts in which a name appears should be somehow represented in the encoding. lastly, a name embedding should be sufficiently close to a pretrained prototypical representation of its conceptual meaning, e.g. a representation of its corresponding concept identifier from a biomedical ontology. such a multi-task model can be effectively trained using synonym sets extracted from ontologies such as the umls or snomed-ct. however, these synonym sets typically reflect only finegrained distinctions between the lowest-level concepts from ontologies. if robust name representations should truly reflect semantic similarity in general, then the assumption is being made that training on such fine-grained synonym sets learns biomedical semantics in a bottom-up way, expecting names of lower-level concepts to spontaneously form relevant higher-level clusters. level 1 c0564444 wound of skin level 2 c0561369 puncture wound of skin level 3 c0561546 bite wound c0576723 sting of skin level 4 c1302713 animal bite wound c0275134 poisoning due to lizard venom c0576722 animal sting c0576724 plant sting example name tick-borne fever poisoning caused by gila monster venom poisoning by bombus nettle sting however, such assumptions have not yet been empirically validated, for instance by showing that an encoder not only learns the differences between names such as nettle sting and tick-borne fever, but also simultaneously learns that they can be grouped together under the more general description puncture wound of skin. moreover, research on representation learning and hierarchical classification for e.g. computer vision has indicated that neural models can leverage substantially different discriminative information for higher, more general levels of categorization than for more fine-grained lower levels @xcite . such hierarchical differences can be exploited to generalize from higher to lower levels @xcite , but they can also be difficult to integrate consistently into a single neural model @xcite . ## related work while context-dependent self-supervised representations usually outperform other text representations on a variety of bionlp problems, such as semantic similarity and question answering, there is no single embedding model for biomedical and clinical texts that is consistently superior and thus can serve as a generally suitable bio-encoder (tawfik and spruit, 2020). to this date, the bne model by @xcite is the most prominent attempt at developing a supervised resource for encoding biomedical names. it uses a multi-task training regime in which it combines objectives from different aspects of deep representation learning, such as a contrastive loss (le-khac et al., 2020), conceptual grounding (see e.g. @xcite ), and explicit regularization of the learned representations (e.g. used by @xcite ). our modifications to the original bne model are informed by such literature. our application of a deep averaging network (dan) @xcite is inspired by a recent subfield of nlp research which has emphasized the effectiveness of random encoders @xcite and simple pooling mechanisms of word embeddings. the fasttext encoder which we use as a baseline and as input for the dan is an example of a simple word-embedding-based model (swem) with average pooling @xcite . ## conclusion and future work in this paper, we have introduced the challenge of integrating higher-level semantics into robust biomedical name representations. we provide a framework to both train and evaluate encoders for