Social Media Medical Concept Normalization using R o BERT a in Ontology Enriched Text Similarity Framework (2020)

Abstract: Pattisapu et al. (2020) formulate medical concept normalization (MCN) as text similarity problem and propose a model based on RoBERTa and graph embedding based target concept vectors. However, graph embedding techniques ignore valuable information available in the clinical ontology like concept description and synonyms. In this work, we enhance the model of Pattisapu et al. (2020) with two novel changes. First, we use retrofitted target concept vectors instead of graph embedding based vectors. It is the first work to leverage both concept description and synonyms to represent concepts in the form of retrofitted target concept vectors in text similarity framework based social media MCN. Second, we generate both concept and concept mention vectors with same size which eliminates the need of dense layers to project concept mention vectors into the target concept embedding space. Our model outperforms existing methods with improvements up to 3.75% on two standard datasets. Further when trained only on mapping lexicon synonyms, our model outperforms existing methods with significant improvements up to 14.61%. We attribute these significant improvements to the two novel changes introduced.

## introduction with the rise of internet and easy accessibility, social media has become primary choice to share information. social media includes generic platforms like twitter, facebook and health related platforms like askapatient.com and patient.info. most of the common public express their health related issues in a descriptive way using informal language. for example, a person suffering from diarrhoea expresses it as "bathroom with runs". some of the colloquial health mentions along with standard concepts is given in research in medical concept normalization in social media text started with phrase-based machine translation model of @xcite . previously, most of the existing work approach medical concept normalization in social media text as supervised multi-class text classification @xcite @xcite @xcite . in this approach, initially concept mention representation vector is learned using any of the deep learning models and then it is given to fully connected softmax layer to get the predicted concept. some of the models are based on shallow neural networks like cnn or rnn with word embeddings as input @xcite @xcite and the rest of concept mention standard concept feel so off depersonalization (id: 79499004) rapid heart beat tachycardia (id: 3424008) unable to relax restlessness (id:162221009) limsopatham and collier (2016) proposed models based on cnn and rnn with outof-domain word embeddings as input, tutubalina et al. (2018) experimented with attention based rnn model on the top of in-domain word embeddings and subramanyam and sivanesan (2020) experimented with bidirectional rnns and elmo embeddings. miftahutdinov and tutubalina (2019) experimented with bert and tf-idf based semantic features while kalyan and sangeetha (2020) proposed model based on bert (devlin et al., 2019) and highway networks (srivastava et al., 2015). the main drawbacks in classification based mcn systems are a) completely ignoring target concept information by representing target concepts as meaningless one hot vectors. b) with the addition of new concepts every year, the number of concepts in clinical knowledge base is increasing. to accommodate new concepts, these models have to be re-trained from scratch which is timetaking and computationally expensive process (pattisapu et al., 2020). to overcome the drawbacks in multi-class classification framework to normalize medical concepts, pattisapu et al. (2020) formulate mcn as a text similarity problem and propose a model based on roberta @xcite and graph embedding based target concept vectors. initially, all the target concept vectors are generated using graph embedding techniques. then they finetune a roberta based model which learns concept mention vector and then projects it into target concepts embedding space using two dense fully connected layers. finally, the closest target concept to the concept mention in the embedding space is chosen. the main drawbacks in the model of @xcite al. (2020) are • graph embedding techniques leverage only the network structure and completely ignore other valuable information associated with concepts like concept description and synonyms. moreover it is time and resource consuming process to generate target concept vectors using graph embedding techniques. like @xcite , we approach mcn as text similarity problem. the contribution of this paper is the two novel changes we introduce in the original model of @xcite to overcome the drawbacks and further improve the performance. following @xcite , we conduct experiments on two publicly available mcn datasets and achieve improvements up to 3.75%. further when trained only on mapping lexicon synonyms, our model outperforms existing methods with improvements up to 14.61%. we attribute these improvements to the two novel changes introduced. ## proposed method initially, all the target concept vectors are generated using sroberta @xcite and retrofitting algorithm @xcite . then we fine-tune a roberta model which learns representation vector of concept mention. finally, concept-id concept description synonyms 278040002 loss of hair falling hair, thinning hair 60862001 tinnitus ringing in ears, noise in ears 77692006 hypersomnia sleeps too much, excessive sleep hypersomnia, excessive sleepiness 60119000 exhaustion washed out, worn out the target concept which is closest (based on cosine similarity) to the concept mention in the embedding space is chosen. we refer to our model as "ontology enriched text similarity framework based roberta (oetsr)" . each concept has a description and set of synonyms as shown in learning concept mention representation is a key step in medical concept normalization @xcite @xcite . like pattisapu et al. (2020), we use roberta to learn concept mention representations. roberta is a variant of bert model trained on 160gb text data with better training strategies. we train the model using adamw optimizer which minimizes cosine embedding loss (l) between concept mention vector, @xmath0, @xmath1. here, h is the hidden vector size in roberta. during training, we freeze the target concept vectors. during inference, we encode concept mention using our fine-tuned roberta model. the concept mention is mapped to the closest target con-cept (based on cosine similarity) in the embedding space. psytar : @xcite gathered psychiatric medicines related reviews from askapatient.com and created this dataset. it consists of 6556 colloquial health related mentions and 618 unique snomed-ct codes. @xcite created five fold dataset foot_1 from these annotations and released it publicly. snomed-ct synonyms: snomed-ct is one of the commonly used medical lexicons which includes around 0.35m concepts. each medical concept has unique id (code), concept description (fully specified name) and set of synonyms. each synonym can be treated as a health mention. to show the performance of our model in the absence of manually annotated instances, we train our model on the dataset created from these synonyms and then evaluate on cadec and psytar datasets. all the results are reported in ## results following @xcite , we conduct experiments on cadec and psytar datasets. the ## conclusion in this work, we come up with a text similarity based model to normalize colloquial health related mentions in user-generated posts. @xcite formulate mcn as text similarity problem and propose a model based on roberta and graph based target concept vectors. our model is an enhancement of the original model of @xcite with two simple and novel changes which improve the performance up to 3.75%. we use retrofitted target concept vectors to represent concepts which leverage both concept description and synonyms, unlike graph embedding techniques. moreover, it is easy and faster to compute retrofitted target concept vectors compared to graph embedding based target concept vectors. in future, we would like to explore options like distant supervision to generate additional training examples.