Normalization of Long-tail Adverse Drug Reactions in Social Media (2020)

Abstract: The automatic mapping of Adverse Drug Reaction (ADR) reports from user-generated content to concepts in a controlled medical vocabulary provides valuable insights for monitoring public health. While state-of-the-art deep learning-based sequence classification techniques achieve impressive performance for medical concepts with large amounts of training data, they show their limit with long-tail concepts that have a low number of training samples. The above hinders their adaptability to the changes of layman’s terminology and the constant emergence of new informal medical terms. Our objective in this paper is to tackle the problem of normalizing long-tail ADR mentions in user-generated content. In this paper, we exploit the implicit semantics of rare ADRs for which we have few training samples, in order to detect the most similar class for the given ADR. The evaluation results demonstrate that our proposed approach addresses the limitations of the existing techniques when the amount of training data is limited.

## introduction discovering adverse drug reactions (adrs) is a critical component of drug safety. in addition to controlled clinical trials, continuous monitoring of adverse effects after market introduction provides valuable insights into adrs. studies have shown that traditional techniques (i.e., voluntary and mandatory reporting of adrs by patients) of post-market adrs are not able to fully characterize drugs' adverse effects @xcite @xcite . social media could help to obtain more information on the occurrence of adverse effects in the real world, by monitoring the information discussed and shared by users for their personal experiences with pharmaceutical drugs. such monitoring can aid in the monitoring of public health @xcite , and provide new opportunities for the identification of adverse drug reactions @xcite @xcite . however, web users report adrs using a different language style and terminology that depends on the user's medical proficiency, but also on the type of online medium (e.g. health forums vs micro-post social networks). therefore, adr reports from user generated content typically differ significantly from adr statements in professional medical text. as exemplified in a critical step in the practical use of usergenerated text for adr surveillance starts with the normalization of reported adverse events, meaning linking the user-reported event to a formal knowledge base such as the umls (unified medical language system) foot_0 or its subsets like meddra (medical dictionary for regulatory activities) 2 . there is a large body of work on adr normalization in social media such as twitter @xcite @xcite , or forums like dailystrength @xcite machine translation, supervised-learning @xcite @xcite @xcite which all showed to have limited performance compared to deep learning-based techniques @xcite @xcite @xcite . ## related work existing methods for normalizing adrs in clinical text content fall into five categories: 1) rulebased approaches @xcite @xcite , ; 2) deep learning approaches @xcite @xcite @xcite 3) machine translation techniques @xcite @xcite ; 4) other supervised approaches such as dnorm @xcite and 5) unsupervised techniques @xcite . rule-based approaches mostly rely on string matching techniques, ignoring the semantics of each text mention, which results in a relatively poor performance @xcite . the unsupervised techniques, despite the fact that they capture semantic similarity between text with minimal string similarity, barely outperform the rule-based techniques. the use of language in domains like social media or online forums is totally different than the official medical terminology used in knowledge bases and hence it is hard to find a common embedding space that is able to match semantically similar entities. the results published in @xcite @xcite show that supervised machine learning techniques such as deep learning and machine translation clearly outperform rule-based and unsupervised techniques (i.e., deep learning achieving the highest accuracy). in addition to the task specific deep learning techniques, bert @xcite a bi-directional language representation model was recently fine-tuned in order to tackle the adr normalization problem as a sequence classification task . this attempt demonstrated remarkable performance across different user-generated text datasets @xcite . however, to perform properly, supervised techniques require large collection of labeled training data for each concept and are not suitable for normalizing concepts with none or few training samples. this is clearly shown in figure 2 , were we reproduced the state of the art rnn as presented in @xcite and also fine-tuned bert, to visualize the performance of those models as a function of the available training samples that each concept (class) has on the smm4h 2017 twitter dataset. in social media posts some medical concepts like adrs or symptoms, are way more common than others. for instance, it is common to find many different expressions referring to 'headache' or 'stomach pain' caused by a drug use, rather than 'sleepwalking'. as a result, there is a high class imbalance between common and rare medical concepts that usually show up in social media. in figure 1 , we can clearly see that more than 40 % of the medical concepts (classes) present in this dataset (507) have just one training sample. in this context, it becomes obvious that there is a need for an alternative way of predicting concepts when we have insufficient training data. addressing the problem of scarcity in training data is a well-studied research topic. few-shot learning @xcite is a family of machine learning algorithms that are able to perform classification with only a few 'shots' (samples) from each class. to achieve that, most of those techniques leverage less complex models to avoid overfitting on the limited amount of training data. for instance, embedding-based few-shot learning models @xcite , try to classify unknown samples by creating a meaningful embedding (feature representation) of the labeled and unlabeled data and then classifying the unlabeled samples based on their most similar embedding from the training data. recent studies @xcite show that this family of nearest neighbor based approaches can achieve surprisingly promising results in a variety of tasks. in natural language processing, this family of techniques can profit from various word embedding models @xcite @xcite to create a semantically meaningful representation of text. those embedding models are proven to create vector representation of words that can capture their semantics, in such a way that semantically similar words will have similar vectors, in terms of cosine similarity. on the sentence and phrase level, several techniques can be used to derive a fixed-size vector representation for the whole phrase, regardless of the number of tokens the phrase includes. these techniques can vary from simple approaches like calculating the average of the individual token embeddings of a phrase, to more sophisticated techniques that use encoder-based models to derive a meaningful fixed-size vector for each multi-token phrase they receive as input. recently, researchers in sbert (reimers and gurevych, 2019) modified the bert language representation model, leveraging siamese and triplet network structures, to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity. the above was demonstrated as a primary drawback of the original bert version. the network was fine-tuned with pairs of sentences from natural language inference (nli) and semantic textual similarity (sts) datasets achieving state of the art performance in minimizing the embedding distance between semantically similar sentences compared to previous sentence embedding techniques. despite the aforementioned research in this domain, to our knowledge, none of the previous research in medical concept normalization in usergenerated text proposed a solution that would address the problem of class imbalance and scarcity in the training data. ## adr normalization in the medical domain, getting more annotated data for rare concepts would be extremely expensive and time consuming. therefore we need to focus our effort on predicting a class (medical concept) for which we have none or maybe few representatives. on a theoretical level, our basic hypothesis is to test whether creating fixed-size vector representations of adrs from the training and test data is a valuable feature in order to normalize the unlabeled adr mentions, based only on their vector similarities. in that case, we are expecting that a nearest neighbor (1-nn) approach can take advantage of its simplicity and demonstrate a better performance in medical concepts where training data is limited. as we mentioned before, this is a significant percentage of the data in a real world scenario. our method is composed of the following stages: 1) we use sentence embedding techniques to encode all adr phrases in the training and test data into a fixed-size vector so that similar adrs appear in close proximity; 2) we use cosine distance to measure the semantic similarity between the vector representation of an unlabeled adr and each representation of the training samples; 3) we classify the unlabeled adr with the label of its nearest neighbor from the training data. there are different techniques to encode words or phrases into a fixedsize vector representation. in this paper, we use two techniques, a simple averaging approach and the state-of-the-art sentence representation model @xcite described below: sbert model training details: to train the model for our task, we needed pairs of similar and dissimilar adr phrases, which is a timeconsuming and expensive process to obtain. for this reason, we used the two pre-trained models provided by the authors. the first model is trained on a combination of the snli @xcite and mnli @xcite datasets, consisting of approximately 1 million sentence pairs labeled as a contradiction, entailment, or neutral. the second one is trained on the aforementioned nli data, and then further tuned on the stsb dataset @xcite , which consists of 8,628 sentence pairs from image captions and news. those pairs are labeled with a number between 0 and 5, indicating the semantic relatedness of each sentence pair. besides, we also experimented with mednli @xcite and biosses sts @xcite datasets, which are manually annotated sentence pairs on the clinical domain. despite the different combinations of data tried, the best performing version was the provided pre-trained model on snli, mnli, and stsb data, which will be further used in our experiments. after encoding the phrases from the train and test data in the semantic vector space, the normalization of the test samples is done based on their nearest neighbor from the training data. as a similarity measure, we use cosine similarity, which is the most common similarity measure between word vector representations @xcite . we finally assign the concept label of the nearest neighbor (i.e., the sample with the highest similarity measure) from the training data to the unlabeled adr in the test set. ## experimental settings as an evaluation metric we use accuracy, which denotes the percentage of the correctly normalized adr samples in the test data. the focus of our evaluation is on the variation of performance using the complete set of training data as well as the different fractions of available training data per class. those subsets are selected based on how many training samples the true label of a test set sample has in the training data. in this way we can demonstrate the effectiveness of our technique in dealing with rare adrs. furthermore, we include the results of qualitative analysis, which will help us identify in more detail the strengths and the limitations of the proposed approach ## results and discussion the results of our experimental evaluation in the two aforementioned datasets are demonstrated in for this reason, we divided all test sets into three different subsets, based on how many training samples the true label of test set sample has in the training data. the first test subset includes adrs with a true label that has just 1 to 4 unique training samples in the corresponding training data. accordingly, the second subset includes concepts with 5-19 samples in the training set and the third 20 or more of them. the bin limits were selected so that each subset has a significant number of samples. the results of the accuracy in those subsets are shown in figures 3 and 4 . from these results, we ## qualitative analysis we perform a qualitative analysis of our 1-nn-sbert approach performance on the smm4h 2017 dataset. the purpose of the qualitative analysis is to get insights about 'where' and 'why' this approach fails or succeeds and to highlight the underlying properties which are hard to digitize without losing any meaning. while the main focus of the paper was on normalizing rare concepts that have a few training samples available, we also started investigating an additional source of knowledge for the concepts for which no training data is available. for the medical concepts with no training data, we used all different synonymous terms (i.e., available in umls) associated with this concept as its representatives. ## conclusions and future work in this work, we have presented a 1-nn-sbert approach that is robust for normalizing rare classes of adrs. our technique cannot compete with a deep neural network in concepts where the training data is available on a larger scale. yet, it easily outperforms deep learning techniques when dealing with rare classes of adrs. this approach can easily scale as the number of classes increases and can adapt to the changes and emergence of new adrs, with no training cost. finally, our sentence embedding model can benefit from the concept of transfer learning, as it does not require task-specific training data to generate high-quality sentence and phrase representations. as future work our model can be improved in the direction of separating the common and the rare medical concepts at test time in a more effective and efficient way. using multiple binary classifiers, or neural networks with multiple sigmoid functions instead of the final softmax layer have been used in similar domains where multilabel classification or open-set classification @xcite is considered.