A Generate-and-Rank Framework with Semantic Type Regularization for Biomedical Concept Normalization (2020)

Abstract: Concept normalization, the task of linking textual mentions of concepts to concepts in an ontology, is challenging because ontologies are large. In most cases, annotated datasets cover only a small sample of the concepts, yet concept normalizers are expected to predict all concepts in the ontology. In this paper, we propose an architecture consisting of a candidate generator and a list-wise ranker based on BERT. The ranker considers pairings of concept mentions and candidate concepts, allowing it to make predictions for any concept, not just those seen during training. We further enhance this list-wise approach with a semantic type regularizer that allows the model to incorporate semantic type information from the ontology during training. Our proposed concept normalization framework achieves state-of-the-art performance on multiple datasets.

## introduction mining and analyzing the constantly-growing unstructured text in the bio-medical domain offers great opportunities to advance scientific discovery @xcite and improve the clinical care @xcite . however, lexical and grammatical variations are pervasive in such text, posing key challenges for data interoperability and the development of natural language processing (nlp) techniques. for instance, heart attack, mi, myocardial infarction, and cardiovascular stroke all refer to the same concept. it is critical to disambiguate these terms by linking them with their corresponding concepts in an ontology or knowledge base. such linking allows downstream tasks (relation extraction, information retrieval, text classification, etc.) to access the ontology's rich knowledge about biomedical entities, their synonyms, semantic types and mutual relationships. concept normalization is a task that maps concept mentions, the in-text natural-language mentions of ontological concepts, to concept entries in a standardized ontology or knowledge base. techniques for concept normalization have been advancing, thanks in part to recent shared tasks including clinical disorder normalization in 2013 share/clef @xcite and 2014 semeval task 7 analysis of clinical text @xcite , and adverse drug event normalization in social media mining for health (smm4h) @xcite . most existing systems use a string-matching or dictionary look-up approach @xcite @xcite , which are limited to matching morphologically similar terms, or supervised multi-class classifiers @xcite @xcite , which may not generalize well when there are many concepts in the ontology and the concept types that must be predicted do not all appear in the training data. we propose an architecture (shown in figure 1 ) that is able to consider both morphological and semantic information. we first apply a candidate generator to generate a list of candidate concepts, and then use a bert-based list-wise classifier to rank the candidate concepts. this two-step architecture allows unlikely concept candidates to be filtered out prior to the final classification, a necessary step when dealing with ontologies with millions of concepts. in contrast to previous list-wise classifiers @xcite which only take the concept mention as input, our bert-based list-wise classifier takes both the concept mention and the candidate concept name as input, and is thus able to handle concepts that never appear in the training data. we further enhance this list-wise approach with a semantic type regularizer that allows our ranker to leverage semantic type information from our work makes the following contributions: the code for our proposed generate-and-rank framework is available at https://github.com/ dongfang91/generate-and-rank-connorm . ## related work traditional approaches for concept normalization involve string match and dictionary look-up. these approaches differ in how they construct dictionaries, such as collecting concept mentions from the labeled data as extra synonyms @xcite , and in different string matching techniques, such as string overlap and edit distance @xcite . two of the most commonly used knowledge-intensive concept normalization tools, metamap @xcite and ctakes @xcite both employ rules to first generate lexical variants for each noun phrase and then conduct dictionary look-up for each variant. several systems (d' @xcite have demonstrated that rule-based concept normalization systems achieve performance competitive with other approaches in a sieve-based approach that carefully selects combinations and orders of dictionaries, exact and partial matching, and heuristic rules. however, such rule-based approaches struggle when there are great variations between concept mention and concept, which is common, for example, when comparing social media text to medical ontologies. due to the availability of shared tasks and annotated data, the field has shifted toward machine learning techniques. we divide the machine learning approaches into two categories, classification @xcite @xcite @xcite @xcite and learning to rank @xcite @xcite @xcite . most classification-based approaches using deep neural networks have shown strong performance. they differ in using different architectures, such as gated recurrent units (gru) with attention mechanisms @xcite , multi-task learning with auxiliary tasks to generate attention weights @xcite , or pre-trained transformer networks @xcite ; different sources for training word embeddings, such as google news @xcite or concept definitions from the unified medical language system (umls) metathesaurus @xcite ; and different input representations, such as using character embeddings @xcite . all classification approaches share the disadvantage that the output space must be the same size as the number of concepts to be predicted, and thus the output space tends to be small such as 2,200 concepts in @xcite and around 22,500 concepts in @xcite . classification approaches also struggle with concepts that have only a few example mentions in the training data. researchers have applied point-wise learning to rank @xcite , pairwise learning to rank @xcite , and list-wise learning to rank @xcite on concept normalization. generally, the learning-to-rank approach has the advantage of reducing the output space by first obtaining a smaller list of possible candidate concepts via a candidate generator and then ranking them. dnorm @xcite , based on a pair-wise learning-to-rank model where both mentions and concept names were represented as tf-idf vectors, was the first to use learning-to-rank for concept normalization and achieved the best performance in the share/clef ehealth 2013 shared task. list-wise learning-to-rank approaches are both computationally more efficient than pairwise learning-to-rank @xcite and empirically outperform both point-wise and pair-wise approaches @xcite . there are two implementations of list-wise classifiers using neural networks for concept normalization: @xcite treat the selection of the best candidate concept as a flat classification problem, losing the ability to handle concepts not seen during training; @xcite take a generate-and-rank approach similar to ours, but they do not leverage resources such as synonyms or semantic type information from umls in their bert-based ranker. ## results in the other datasets, performance of our complete model is much higher: 87.46% for askapatient, 88.24% for smm4h-17 7 . on the twadr-l, smm4h-17, and mcn test sets, adding the bert-based ranker improves performance over the candidate generator alone, and adding the semantic type regularization further improves performance. for example, lucene alone achieves 79.25% accuracy on the mcn data, adding the bert ranker increases this to 82.75%, and adding the semantic type regularizer increases this to 83.56%. on askapatient, performance of the full model is similar to just the bert multiclass classifier, perhaps because in this case bert alone already successfully improves the state-ofthe-art from 85.71% to 87.52%. the +gold setting allows us to answer how well our ranker would perform if our candidate generator made no mistakes. first, we can see that if the correct concept is always in the candidate list, our list-based ranker (+bert-rank) outperforms the multi-class classifier (bert) on all test sets. we also see in this setting that the benefits of the semantic type regularizer are amplified, with test sets of twadr-l and mcn showing more than 1.00% gain in accuracy from using the regularizer. these findings suggest that improving the quality of the candidate generator should be a fruitful future direction. overall, we see the biggest performance gains from our proposed generate-and-rank architecture in the mcn dataset. this is the most realistic setting, where the number of candidate concepts is large and many test concepts were never seen during training. in such cases, we cannot use a multiclass classifier as a candidate generator since it would never generate unseen concepts. thus, our ranker shines in its ability to sort through the long list of possible concepts. ## qualitative analysis ## limitations and future research the available concept normalization datasets are somewhat limited. @xcite notes that askapatient and twadr-l have issues including duplicate instances, which can lead to bias in the system; many phrases have multiple valid mappings to concepts but the context necessary to disambiguate is not part of the dataset; and the 10-fold cross-validation makes training complex models unnecessarily expensive. these datasets are also unrealistic in that all concepts in the test data are seen during training. future research should focus on more realistic datasets that follow the approach of mcn in annotating mentions of concepts from a large ontology and including the full context. our ability to explore the size of the candidate list was limited by our available computational resources. as the size of the candidate list increases, the true concept is more likely to be included, but the number of training instances also increases, making the computational cost larger, especially for the datasets using 10-fold cross-validation. we chose candidate list sizes as large as we could afford, but there are likely further gains possible with larger candidate lists. our semantic type regularizer is limited to exact matching: it checks only whether the semantic type of a candidate exactly matches the semantic type of the true concept. the umls ontology includes many other relations, such as is-a and part-of relations, and extending our regularizer to encode such rich semantic knowledge may yield further improvements in the bert-based ranker. ## conclusion we propose a concept normalization framework consisting of a candidate generator and a list-wise classifier based on bert. because the candidate ranker makes predictions over pairs of concept mentions and candidate concepts, it is able to predict concepts never seen during training. our proposed semantic type regularizer allows the ranker to incorporate semantic type information into its predictions without requiring semantic types at prediction time. this generate-and-rank framework achieves state-of-theart performance on multiple concept normalization datasets.