Scalable Few-Shot Learning of Robust Biomedical Name Representations (2021)

Abstract: Recent research on robust representations of biomedical names has focused on modeling large amounts of fine-grained conceptual distinctions using complex neural encoders. In this paper, we explore the opposite paradigm: training a simple encoder architecture using only small sets of names sampled from high-level biomedical concepts. Our encoder post-processes pretrained representations of biomedical names, and is effective for various types of input representations, both domain-specific or unsupervised. We validate our proposed few-shot learning approach on multiple biomedical relatedness benchmarks, and show that it allows for continual learning, where we accumulate information from various conceptual hierarchies to consistently improve encoder performance. Given these findings, we propose our approach as a low-cost alternative for exploring the impact of conceptual distinctions on robust biomedical name representations.

## introduction recent research in biomedical nlp has focused on learning robust representations of biomedical names. to achieve robustness, an encoder should represent the semantic similarity and relatedness between different names (e.g. by their closeness in the embedding space), while its embeddings should also remain as transferable and generally applicable as self-supervised pretrained representations. prior research into robust representations has shown three distinct tendencies. firstly, research typically focuses on encoders with complex neural architectures and a large amount of parameters. as chapter v: mental and behavioural disorders f34 f63 persistent mood disorders habit and impulse disorders f34.0 f63.0 cyclothymia pathological gambling f34.1 f63.1 dysthymia pyromania lastly, as a result of using fine-grained concepts, vast amounts of biomedical names are needed to model the large collection of fine-grained distinctions present in ontologies. for instance, @xcite train their encoder on 156k disorder names. these three tendencies share an underlying assumption: complex neural encoder architectures can learn biomedical semantics by generalizing in a bottom-up fashion from large amounts of finegrained semantic distinctions, if provided with sufficient quantities of training data. however, it is not self-evident that such an approach is the most effective way to achieve generalpurpose biomedical name representations. for instance, it does not directly address what conceptual distinctions are actually relevant to improve representations for downstream nlp applications. finding and exploiting relevant distinctions can be an empirical question, and as such require lowcost exploration of various conceptual hierarchies. such a heuristic search is expensive in the current paradigm. in this paper, we explore a scalable few-shot learning approach for robust biomedical name representations which is orthogonal to this paradigm. we investigate to what extent we can fit a simple encoder architecture using only a small selection of data, with a limited amount of concepts containing only a few samples each (i.e., few-shot learning). to this end, we don't use fine-grained concepts for training, but more general higher-level concepts which span a large range of fine-grained concepts. this paper offers two main contributions. firstly, our proposed approach offers an alternative for training biomedical name encoders with much lower computational cost, both for training and inference at test time. it is applicable to largescale hierarchies containing at least ten thousands of names and is equally effective for different types of pretrained representations when tested on various biomedical relatedness benchmarks. secondly, we show that this approach allows for low-cost continual learning from multiple concept hierarchies, and as such can help with the accumulation of relevant domain-specific information for downstream biomedical nlp tasks. ## approach our approach is similar to supervised postprocessing techniques of word embeddings such as retrofitting and counterfitting @xcite , but instead post-processes pretrained representations of biomedical names. ## conclusion and future work we have proposed a novel approach for scalable few-shot learning of robust biomedical name representations, which trains a simple encoder architecture using only small subsamples of names from higher-level concepts of large-scale hierarchies. our model works for various pretrained input embeddings, including already specialized name representations, and can accumulate information over various hierarchies to systematically improve performance on biomedical relatedness benchmarks. future work will investigate whether such improvements trickle down properly to downstream biomedical nlp tasks.