Conceptual Grounding Constraints for Truly Robust Biomedical Name Representations (2021)

Abstract: Effective representation of biomedical names for downstream NLP tasks requires the encoding of both lexical as well as domain-specific semantic information. Ideally, the synonymy and semantic relatedness of names should be consistently reflected by their closeness in an embedding space. To achieve such robustness, prior research has considered multi-task objectives when training neural encoders. In this paper, we take a next step towards truly robust representations, which capture more domain-specific semantics while remaining universally applicable across different biomedical corpora and domains. To this end, we use conceptual grounding constraints which more effectively align encoded names to pretrained embeddings of their concept identifiers. These constraints are effective even when using a Deep Averaging Network, a simple feedforward encoding architecture that allows for scaling to large corpora while remaining sufficiently expressive. We empirically validate our approach using multiple tasks and benchmarks, which assess both literal synonymy as well as more general semantic relatedness.

## introduction biomedical and clinical free-text contain mentions of biomedical terms which can provide valuable information for text mining applications. such textual mentions, as well as their corresponding reference names in biomedical ontologies, can often be expressed in various synonymous surface forms (e.g. pleuritic pain vs. pain breathing), which is challenging for downstream applications. effective dense representation of these biomedical names icd-10 snomed-ct c0564504 schizoid fantasy schizoid fantasy -mental defense mechanism f60.1 c0338969 introverted personality disorder introverted personality c0036339 schizoid personality disorder unspecified schizoid personality disorder recent research has focused more explicitly on encoding domain-specific biomedical semantics by training biomedical name representations that are robust, i.e., reflecting the synonymy and semantic relatedness of names by their closeness in the embedding space, preferably in a consistent way that generalizes across different biomedical subdomains and corpora. to date, the most effective approaches have applied some form of conceptual grounding: minimizing the distance between on the one hand representations of names, and on the other hand pretrained embeddings of their concept identifiers. these concept embeddings are supposed to reflect domain-specific semantics, and are constructed using a variety of different techniques, including distributional similarity of graph relations and distributional similarity of textual occurrences in large-scale free-text, as well as combinations thereof @xcite . while knowledge graph embeddings of biomedical concepts can encode a variety of semantic relations, @xcite show that such graph embeddings need to incorporate textual features to make them effective targets for conceptual grounding. such features help to translate textual representations of names to the topology of the concept embedding space, which otherwise reflects only ontological information. in other words, concept embeddings are mostly useful targets for grounding to the extent that name representations can be efficiently mapped to them by the encoder architecture. this raises the question whether we can increase the effectiveness of conceptual grounding by better aligning the topology of the created name embedding space and the pretrained concept embedding space. in this paper, we investigate how to maximally exploit low-cost concept embeddings, which can be constructed using only pretrained word embeddings and sets of biomedical synonyms or semantically related names. to this end, we enrich a siamese neural network encoder for biomedical names with 2 novel constraints which are meant to effectively map encoded names to pretrained concept embeddings. the first constraint, which we call the linear constraint, applies canonical correlation analysis (cca) to pretrained embeddings of names and their concepts to project them into a space which improves their linear mapping. these transformed embeddings are then used as input representations for the neural encoder. the second constraint adds a training objective which we call prototypical grounding: minimizing the distance between a pretrained concept embedding and the average of all the encoded names belonging to that concept. this average is an approximation of the prototypical representation of a concept in the name embedding space. while the linear constraint involves a simple preprocessing step, the prototypical grounding constraint can be computationally expensive for largescale corpora. therefore, we use a simple deep averaging network (dan) @xcite as encoder to prove the effectiveness and scalability of our approach, even for a neural architecture that has no access to word order like lstms have or cannot apply attention over specific word combinations like transformers can. we train and evaluate our encoder on different categorizations of biomedical names. for instance, ## related work biomedical name encoders a variety of neural architectures have been proposed for encoding biomedical names. @xcite use a multi-sense lstm with attention over different word senses. this attention is conditioned on the context of the biomedical name. @xcite include a character-level bidirectional lstm in a word-level bidirectional lstm which extracts a fixed-size representation using max pooling over all dimensions, followed by a linear transformation. @xcite finetunes pretrained contextsensitive biobert @xcite representations and uses them in tandem with lexical tf-idf representations. while past research has explicitly investigated the role of various training objectives, even jointly in multi-task training regimes, the specific impact of encoder architectures has not received much attention or comparison. averaging networks research on sentence embeddings and paraphrasing has consistently found that simple encoding procedures such as averaging of word embeddings can rival or even outperform complex neural architectures on tasks for which those are finetuned @xcite @xcite . moreover, research on deep averaging networks @xcite has found that feedforward neural networks that use averaged word embeddings as input can be tuned to textual classification tasks such as sentiment analysis if the network is sufficiently large and/or deep. this way, small differences in the input can be magnified by the network where relevant. prototypical networks while successful approaches to few-shot learning such as matching networks @xcite optimize representation models on the level of single instances, follow-up work has shown the benefits of simultaneously learning class representations using those same models. for instance, prototypical networks @xcite train a neural encoder with objectives that involve class prototypes, which are created by averaging the encodings of all instances that belong to a single class. in this paper, we include a training objective for our encoder which forces synonymous or semantically related biomedical names to form class prototypes that approximate the pretrained embedding of their concept identifier. ## conclusion and future work in this paper, we have shown how two conceptual grounding constraints for biomedical name encoders can infuse name representations with more domain-specific semantics without losing robustness. these representations can help with retrieving literal synonyms as well as semantically related terms, and can be sufficiently expressed by a deep averaging network, which is a feedforward neural network that only takes averaged word embeddings as input. we believe future work can include a comparison of neural encoding architectures with a wider range of complexity. decreasing the complexity of neural architectures can allow for including more comprehensive training objectives which target more effective encoding of domain-specific semantics.