Clinical Concept Linking with Contextualized Neural Representations (2020)

Abstract: In traditional approaches to entity linking, linking decisions are based on three sources of information – the similarity of the mention string to an entity’s name, the similarity of the context of the document to the entity, and broader information about the knowledge base (KB). In some domains, there is little contextual information present in the KB and thus we rely more heavily on mention string similarity. We consider one example of this, concept linking, which seeks to link mentions of medical concepts to a medical concept ontology. We propose an approach to concept linking that leverages recent work in contextualized neural models, such as ELMo (Peters et al. 2018), which create a token representation that integrates the surrounding context of the mention and concept name. We find a neural ranking approach paired with contextualized embeddings provides gains over a competitive baseline (Leaman et al. 2013). Additionally, we find that a pre-training step using synonyms from the ontology offers a useful initialization for the ranker.

## introduction medical concept linking produces structured topical content from clinical free text @xcite . healthcare providers often refer to medical concepts in clinical text notes that are absent from associated health record metadata despite their importance to understanding a patient's medical status. for example, in the patient reports a history of seizure disorder..., the phrase seizure disorder refers to the concept epilepsy contained within the unified medical language system (umls) ontology @xcite . however, this may be absent from metadata as it is not part of the current diagnosis. concept mentions can use non-standard terms (e.g. epilepsy), thus concept linking requires non-lexical methods. additionally, some terms (cancer) are ambiguous and could refer to multiple concepts (breast cancer, colon cancer, etc.) the related task of entity linking -linking named entities (people, places, and organizations) to a knowledge base -has been explored in nonmedical domains @xcite @xcite . entity linking systems consider three sources of information: 1) similarity between mention strings and names for the kb entity; 2) comparison of the document context to information about the kb entity (e.g. entity description); 3) information contained in the kb, such as entity popularity or inter-entity relations. in contrast to the dense kbs in entity linking, concept linking uses sparse ontologies, which contain a unique identifier (cui), title, and links to synonyms and related concepts, but rarely longform text. for example, while the concept epilepsy has many synonyms in umls, it has no definition or other long description. furthermore, umls concept names are more formal than clinical notes, making mention matching challenging. therefore, we need an approach that can use local context from the mention (surrounding sentence), and whatever information may be present in the ontology to build a contextualized non-lexical representation for matching. additionally, entity linking systems are often able to leverage greater amounts of annotated data, which are not available in the clinical space. text that does not have restrictive privacy protections can be annotated more easily through crowdsourcing, or other sources of non-gold standard data collected (e.g., wikipedia cross-links). as the annotation of clinical notes is expensive due to the knowledge required of annotators and the protected status of clinical records, any effort in clinical concept linking must focus on leveraging a small amount of annotations, and using larger amounts of related or unannotated data when possible. we propose learning contextualized representations that leverage both free text and information from knowledge bases. we train a contextualized language model @xcite on unannotated clinical text, leveraging sentence context to construct a mention. we explore several methods of building representations of the mention span and concept, including pooling and attention, and pre-training our linker with additional data from the ontology to augment the small amount of annotated data present. the resulting ranker outperforms a non-contextualized version of our model, and beats the previous best performing system @xcite in most metrics. ## concept linking concept linking (alternatively: named entity recognition, entity normalization), has a long history @xcite in the clinical nlp community, with common approaches including generating lexical variations to increase matches (metamap) @xcite , dictionary matching algorithms @xcite , rule based systems (d'souza and ng, 2015), and mention/ontology context overlap @xcite . learned ensembles can also be effective @xcite . concept linking has also been applied to bio-medical literature @xcite @xcite and is most similar to the task of entity linking @xcite @xcite . similar to our approach, @xcite learn representations of concepts in umls. while we cannot make a direct comparison since they do not cover all of our kb (snomed-ct), initial experiments with their embeddings performed worse than our method. while some jointly consider the task of mention finding and linking @xcite , we follow the more common convention of separating the two and assuming gold mention spans @xcite . formally, we are given a mention m in a document and must select the best cui (concept) c from an ontology/kb, or cui-less if no relevant concept exists. many systems utilize a rule-based approachoften as a pre-processing step -that uses the training data to augment a dictionary (d' @xcite . while this approach does quite well, it poorly generalizes to unseen mentions or new domains. 1 therefore, our work will focus on a learned system and compare it to similar baselines. while related to concept linking, entity linking requires a different solution due to several factors. many entity linking systems @xcite leverage context from a large document, such as wikipedia, to make linking decisions, while a similar source is not present in umls. further, earlier work @xcite showed that standard entity linking systems don't work well on the related domain of biomedical journal literature, which suggests that separate solutions are required. ## methods our concept linking system is based on a pairwise neural network ranker ( §3.1) using contextualized representations ( §3.2) for both the mention and concept. we leverage the context present in clinical notes for our representations and synonyms present within the umls to train our linker. ## experimental setup we train and evaluate our system on the share/clef ehealth evaluation lab 2013 task 1b dataset @xcite , which consists of span-level annotations for disorder concepts taken from the mimic 2.5 clinical note dataset @xcite . the publicly available training set includes 200 clinical notes, which we split into a 100 note training set, and development and testing sets of 50 documents each -the shared task test set was not available. the data is annotated against snomed-ct @xcite , one of the ontologies within umls. we choose to focus on this smaller dataset as leveraging small amounts of annotated data is critical to building useful tools in the clinical domain. we only included mention annotations for concepts that occur in the selected subset of the ontology noted in the annotation guidelines for the respective datasets or are marked as cui-less 5 . in we compare our system to dnorm @xcite for the share/clef 2013 dataset, the best performing system in the share/clef 2013 shared task. 6 unlike many other concept linking systems, dnorm scores each mention against all concepts and does not use a triage system, allowing a fair comparison to our system. dnorm builds term frequency-inverse document frequency (tf-idf) representations of both the mention and concept and learns a weighted similarity to rank concepts for each mention. it is unable to return concept candidates for mentions that are out-ofvocabulary as it uses a word-level measure. the authors add a specific cui-less representation, which is made of entries occurring more than four times in training. we report results on our recreated test set, as the evaluation set provided for the shared task was not available to us. we also compare with using word2vec @xcite representations instead of elmo representations in the same linking architecture to test the effect of contextualized embeddings. we trained the word2vec model on the mimic dataset. we created single embeddings (@xmath0, ignoring all out-of-vocabulary words. we explored several parameter configurations for our model suggested in @xcite , reporting the best performing models on development. these include hidden layers of size @xcite @xcite and number of layers in [1,2,3], with a tanh activation function for final layer and relu @xcite for all others. we optimize using the adam optimizer @xcite , and a dropout rate of 0.2. parameter values and development metrics are available in appendix a. for the elmo models, we trained for 10 epochs using the default configuration. for cui-less mentions, we select a threshold score based on the development set, equal to the mean score of all cui-less entries. if an entry does not have a scored concept above that threshold, we consider it cui-less, adding cui-less at that position in the list for mrr. we use the pytorch framework and code from the spotlight library @xcite . ## results ## discussion our neural ranking models with attention outperform all other models, except for cui-only accuracy. in the case of entities with cuis, we find that pre-training the model does provide a gain in ranking accuracy (mrr). in the case of all entities, we find that the attention models provide a sizable gain in both accuracy and mrr. we conducted an error analysis of the best performing mrr model (att. + pre.) on the development data, looking at errors where the gold standard concept was not highly ranked (assigned a rank of 10 or above). of those errors (@xmath1, we find that 26% are mentions that contain only acronyms (e.g. lbp for lower back pain), and 14% are mentions containing some other abbreviation (a shorted word, e.g. post nasal drip for posterior rhinorrhoea, or a partial acronym, seizure d / o for epilepsy). comparing to similar errors from attention model (@xmath2, we find that the number of acronym errors is nearly the same (24) as the better performing model (26). in contrast, the number of non-abbreviation errors drops significantly. this suggests that pre-training provides useful signal for mentions that consist of variations appearing in the ontology. however, it does not help with acronyms or other abbreviations that are less likely to appear in the ontology or are shorter and more ambiguous (e.g., 'r' for rhonchus). while the linker often predicted unrelated concepts (40% of errors) for concepts where the correct concept was ranked above 10, many incorrect concept predictions were somewhat related to the gold concept (e.g., for mention atherosclerotic plaque with gold concept atherosclerotic fibrous plaque our model predicted the concept atherosclerosis). we further noticed that in 21% of cases the linker predicted a relevant concept (e.g., mention thrombosed and thrombosis), but is not counted as correct due to annotation decisions. this could be due to multiple possible concepts in the ontology or the presence of closely-related concepts. deploying our system in a large-volume clinical setting would likely require several alterations. the main computational barrier to labeling a large amount of data, the speed of prediction, can be addressed by using an accurate candidate selection system to prune the number of concepts considered. considering a smaller subset (e.g., 20) of concepts instead of all would significantly improve the speed. further, if using a consistent portion of the ontology, caching the concept embeddings c as opposed to building them in-model also enhances efficiency. depending on the application, a less accurate but faster linker might be a better choice (e.g. for all clinical notes at a medical institution). in contrast, a more complex linker, such as ours, maybe a better option for specific subsets of notes that require better accuracy (e.g., the results of specific clinical studies). our results demonstrate the advantages of using contextualized embeddings for ranking tasks, and that using information from the knowledge base for training is an essential direction for learning concept representations for sparse kb domains. future work will consider additional methods for integrating ontology structure into representation learning.