Identification of Medication Tweets Using Domain-specific Pre-trained Language Models (2020)

Abstract: In this paper, we present our approach for task1 of SMM4H 2020. This task involves automatic classification of tweets mentioning medication or dietary supplements. For this task, we experiment with pre-trained models like Biomedical RoBERTa, Clinical BERT and Biomedical BERT. Our approach achieves F1-score of 73.56%.

## introduction in recent times, social media platforms like twitter, facebook, reddit attracted large number of internet users. the valuable information shared by internet users which also includes health related experiences is useful in many tasks including pharmacovigilance @xcite . user generated texts in social media are noisy with lots of slang words and misspelled words. we participate in task1 of smm4h2020 which aims to develop a system that can identify tweets with medication or dietary supplement mentions. example of tweet with medication or dietary supplement mention is 'it is good to take vitamin c every day after lunch'. an example of a tweet without medication or dietary supplement mention is 'vitamin c is good for health' @xcite . the main challenge in this task is that the system should be able to identify from the context of the tweet that the mention having drug or dietary supplement name is actually referring to the drug or dietary supplementary. this task is treated as binary classification and aims at training a model which can label the given tweet with 1 if it contains medication mention and 0 if there is no medication mention. the performance of models in this task is evaluated using f1-score of class 1. many research works @xcite show that models trained on medical text can better understand medical terms. so, the task is experimented with pretrained models like biomedical roberta @xcite , clinical bert @xcite and biomedical bert @xcite . biomedical bert and biomedical roberta are obtained by further pre-training bert @xcite and roberta @xcite models on biomedical corpus while clinical bert is obtained by further pre-training bert model on mimic-iii corpus @xcite . among these, the model based on biomedical roberta achieves the highest f1-score of 73.56%. ## dataset and preprocessing the organizers of this task released train, validation and test sets. the train set includes 55419 tweets ( 146 positive tweets and 55273 negative tweets), validation set includes 13853 tweets (35 positive tweets and 13818 negative tweets) and test set consists of 29687. as tweets are noisy in nature, the following basic steps are used to clean the tweets: ## model description in recent times, researchers have focused on exploiting deep pre-trained language models like bert, roberta in most of the natural language processing tasks. these models are adapted to medical domain by means of additional training on large medical text. this paper investigates how well domain specific models like biomedical roberta, biomedical bert and clinical bert identify medication tweets. first, tweet representation e @xmath0, n represents hidden state vector in pre-trained language model which is equal to 768. then, sigmoid layer with parameters @xmath1. here ptlm refers to pretrained language model and it can be biomedical roberta, biomedical bert or clinical bert. ## experiments and results to handle imbalance in the dataset, the dataset is augmented with 9622 tweets (4975 positive tweets and 4647 negative tweets) from smm4h 2018 task1 dataset @xcite . further, positive tweets are up sampled 15 times and 90% of the negative tweets are randomly chosen. we use validation set to find optimal values for various hyperparameters. we use batch size of 16, learning rate of 3e-5 and train the model for 3 epochs. all our models are implemented using transformers library in pytorch @xcite . the task organizers released precision and recall scores only for which model got the highest f1-score, the performance of our models and average score is listed in ## conclusion the medication mentions in tweets are identified using domain specific deep pre-trained models. experimental results show that the model based on biomedical roberta achieves the best f1-score of 73.56% which is significantly higher than the average f1-score of 66.28%.