Triplet-Trained Vector Space and Sieve-Based Search Improve Biomedical Concept Normalization (2021)

Abstract: Concept normalization, the task of linking textual mentions of concepts to concepts in an ontology, is critical for mining and analyzing biomedical texts. We propose a vector-space model for concept normalization, where mentions and concepts are encoded via transformer networks that are trained via a triplet objective with online hard triplet mining. The transformer networks refine existing pre-trained models, and the online triplet mining makes training efficient even with hundreds of thousands of concepts by sampling training triples within each mini-batch. We introduce a variety of strategies for searching with the trained vector-space model, including approaches that incorporate domain-specific synonyms at search time with no model retraining. Across five datasets, our models that are trained only once on their corresponding ontologies are within 3 points of state-of-the-art models that are retrained for each new domain. Our models can also be trained for each domain, achieving new state-of-the-art on multiple datasets.

## introduction concept normalization (aka. entity linking or entity normalization) is a fundamental task of information extraction which aims to map concept mentions in text to standard concepts in a knowledge base or ontology. this task is important for mining and analyzing unstructured text in the biomedical domain as the texts describing biomedical concepts have many morphological and orthographical variations, and utilize different word orderings or equivalent words. for instance, heart attack, coronary attack, mi, myocardial infarction, cardiac infarction, and cardiovascular stroke all refer to the same concept. linking such terms with their corresponding concepts in an ontology or knowledge base is critical for data interoperability and the development of natural language processing (nlp) techniques. research on concept normalization has grown thanks to shared tasks such as disorder normalization in the 2013 share/clef @xcite , chemical and disease normalization in biocreative v chemical disease relation (cdr) task @xcite , and medical concept normalization in 2019 n2c2 shared task @xcite , and to the availability of annotated data @xcite . existing approaches can be divided into three categories: rule-based approaches using string-matching or dictionary look-up @xcite @xcite , which rely heavily on handcrafted rules and domain knowledge; supervised multi-class classifiers @xcite @xcite @xcite , which cannot generalize to concept types not present in their training data; and two-step frameworks based on a nontrained candidate generator and a supervised candidate ranker @xcite @xcite @xcite @xcite , which require complex pipelines and fail if the candidate generator does not find the gold truth concept. we propose a vector space model for concept normalization, where mentions and concepts are encoded as vectors -via transformer networks trained via a triplet objective with online hard triplet mining -and mentions are matched to concepts by vector similarity. the online hard triplet mining strategy selects the hard positive/negative exemplars from within a mini-batch during training, which ensures consistently increasing difficulty of triplets as the network trains for fast convergence. there are two advantages of applying the vector space model for concept normalization: 1) it is computationally cheap compared with other supervised classification approaches as we only compute the representations for all concepts in ontology once after training the network; 2) it allows concepts and synonyms to be added or deleted after the network is trained, a flexibility that is important for the biomedical domain where frequent updates to ontologies like the unified medical language system (umls) metathesaurus foot_0 are common. unlike prior work, our simple and efficient model requires neither negative sampling before the training nor a candidate generator during inference. our work makes the following contributions: the code for our proposed framework is available at https://github.com/dongfang91/ triplet-search-connorm . ## related work earlier work on concept normalization focuses on how to use morphological information to conduct lexical look-up and string matching @xcite @xcite @xcite @xcite . they rely heavily on handcrafted rules and domain knowledge, e.g., d'souza and ng (2015) define 10 types of rules at different priority levels to measure morphological similarity between mentions and candidate concepts in the ontologies. the lack of lexical overlap between concept mention and concept in domains like social media, makes rule-based approaches that rely on lexical matching less applicable. supervised approaches for concept normalization have improved with the availability of annotated data and deep learning techniques. when the number of concepts to be predicted is small, classification-based approaches @xcite @xcite @xcite are often adopted, with the size of the classifier's output space equal to the number of concepts. approaches differ in neural architectures, such as character-level convolution neural networks (cnn) with multi-task learning @xcite and pre-trained transformer networks @xcite . however, classification approaches struggle when the annotated training data does not contain examples of all concepts -common when there are many concepts in the ontology -since the output space of the classifier will not include concepts absent from the training data. to alleviate the problems of classification-based approaches, researchers apply learning to rank in concept normalization, a two-step framework including a non-trained candidate generator and a supervised candidate ranker that takes both mention and candidate concept as input. previous candidate rankers have used point-wise learning to rank @xcite , pair-wise learning to rank @xcite @xcite , and list-wise learning to rank @xcite @xcite . these learning to rank approaches also have drawbacks. firstly, if the candidate generator fails to produce the gold truth concept, the candidate ranker will also fail. secondly, the training of candidate ranker requires negative sampling beforehand, and it is unclear if these pre-selected negative samples are informative for the whole training process @xcite . inspired by @xcite , we propose a triplet network with online hard triplet mining for concept normalization. our framework sets up concept normalization as a one-step process, calculating similarity between vector representations of the mention and of all concepts in the ontology. online hard triplet mining allows such a vector space model to generate triplets of (mention, true concept, false concept) within a mini-batch, leading to efficient training and fast convergence @xcite . in contrast with previous vector space models where mention and candidate concepts are mapped to vectors via tf-idf @xcite , treelstms @xcite , cnns @xcite or elmo @xcite , we generate vector representations with bert @xcite , since it can encode both surface and semantic information @xcite . there are a few similar works to our vector space model, cnn-triplet @xcite , biosyn @xcite , roberta-node2vec @xcite , and tti @xcite . cnn-triplet is a two-step approach, requiring a generator to generate candidates for training the triplet network, and requiring various embedding resources as input to cnn-based encoder. biosyn, roberta-node2vec, and tti are onestep approaches. biosyn requires an iterative candidate retrieval over the entire training data during each training step, requires both bert-based and tf-idf-based representations, and performs a variety of pre-processing such as acronym expansion. both roberta-node2vec and tti use a bertbased encoder to encode the mention texts into a vector space, but they differ in how to generate vector representations for medical concepts. specifically, roberta-node2vec uses a node2vec graph embedding approach to generate concept representations, and fixes such representations during training, while tti randomly initializes vector representations for concepts, and keeps such representations learnable during training. note that none of these works explore search strategies that allow domainspecific synonyms to be added without retraining the model, while we do. ## proposed methods we define a concept mention m as a text string in a corpus d, and a concept c as a unique identifier in an ontology o. the goal of concept normalization is to find a mapping function f that maps each textual mention to its correct concept, i.e., @xmath0. we define concept text t as a text string denoting the concept c, and @xmath1, where t (c) is all the concept texts denoting concept c. concept text may come from an ontology, @xmath2, where o(c) is the synonyms of the concept c from the ontology o, or from an annotated corpus, @xmath3, where d(c) is the mentions of the concept c in an annotated corpus d. t (c) will allow the generation of tuples (t, c) such as (mi,c0027051) and (myocardial infarction,c0027051). note that, for a figure 1 : example of loss calculation for a single instance of triplet-based training. the same bert model is used for encoding t i , t p , and the triplet network architecture has been adopted in learning representations for images @xcite ) and text @xcite . it consists of three instances of the same sub-network (with shared parameters). when fed a (t i , t ip , t in ) triplet of texts, the sub-network outputs vector representations for each text, which are then fed into a triplet loss. we adopt pubmed-bert @xcite as the sub-network, where the representation for the concept text is an average pooling of the representations for all sub-word tokens foot_1 . this architecture is shown in figure 1 . the inputs to our model are only the mentions or synonyms. we leave the resolution of ambiguous mentions, which will require exploration of contextual information, for future work. ## model selection we use the development data to choose whether to train the triplet network on just the ontology or also the training data, and to choose which among the similarity search strategies described in section 3.2. approach ncbi bc5cdr-d bc5cdr-c share/clef mcn sieve-based (d'souza and ng, 2015) 84.65 --90.75 -sieve-based (luo et al., 2019) ----76.35 taggerone @xcite 88.80 88.9 94.1 --cnn-based ranking @xcite 86.10 --90.30 -bert-based ranking @xcite 89.06 --91.10 -bert-based ranking @xcite ----83.56 biosyn @xcite 91.1 93.2 96.6 --tti ( henry et al., 2020 outperform their non-sieve-based counterparts. for example, d-t + o-t has better average ranks than o-t, d-t, or od-t (rows 7 vs. 1, 3, and 5; and rows 21 vs. 15, 17, and 19). from this analysis on the dev set, we select the following models to evaluate on the test set: train:o + search:o-t this is the best approach that requires only the ontology; no annotated training data is used. train:o + search:d-t+od-t this is the best approach that only needs to be trained once (on the ontology), as the training data is only used to add extra concept text during search time. this is similar to a real-world scenario where a user manually adds some extra domain-specific synonyms for concepts they care about. train:od + search:d-t+od-t this is the best approach that can be created from any combination of ontology and training data. the triplet network must be retrained for each new domain. train:od + search:tuned this is the bold models in the second half of ## results the pubmed-bert rows in perhaps most interestingly, our triplet network trained only on the ontology and no annotated training data, train:o+search:d-t+od-t, achieves within 3 points of state-of-the-art on all datasets. we believe this represents a more realistic scenario: unlike prior work, our triplet network does not need to be retrained for each new dataset/domain if their concepts are from the same ontology. instead, the model can be adapted to a new dataset/domain by simply pointing out any extra domain-specific synonyms for concepts, and the search can integrate these directly. domain-specific synonyms do seem to be necessary for all datasets; without them (i.e., train:o+search:o-t), performance is about 10 points below state-of-the-art. as a small qualitative analysis of the models, ## limitations and future research our ability to normalize polysemous concept mentions is limited by their context-independent representations. although our pubmed-bert encoder is a pre-trained contextual model, we feed in only the mention text, not any context, when producing a representation vector. this is not ideal for mentions with multiple meanings, e.g., potassium in clinical notes may refer to the substance (c0032821) or the measurement (c0202194), and only the context will reveal which one. a better strategy to generate the contextualized representation for the concept mention, e.g., @xcite , may yield improvements for such mentions. we currently train a separate triplet network for each ontology (one for medic, one for ctd, one for snomed-ct, etc.) but in the future we would like to train on a comprehensive ontology like the umls metathesaurus @xcite , which includes nearly 200 different vocabularies (snomed-ct, meddra, rxnorm, etc.), and more than 3.5 million concepts. we expect such a general vector space model would be more broadly useful to the biomedical nlp community. we explored one type of triplet training network, but in the future we would like to explore other variants, such as semi-hard triplet mining @xcite for generating samples, cosine similarity for measuring the similarity during training and inference, and multi-similarity loss @xcite for calculating the loss. ## conclusions we presented a vector-space framework for concept normalization, based on pre-trained transformers, a triplet objective with online hard triplet mining, and a new approach to vector similarity search. across five datasets, our models that require only an ontology to train are competitive with state-of-the-art models that require domain-specific training.