Automatic Classification of Tweets Mentioning a Medication Using Pre-trained Sentence Encoders (2020)

Abstract: This paper describes our submission to the 5th edition of the Social Media Mining for Health Applications (SMM4H) shared task 1. Task 1 aims at the automatic classification of tweets that mention a medication or a dietary supplement. This task is specifically challenging due to its highly imbalanced dataset, with only 0.2% of the tweets mentioning a drug. For our submission, we particularly focused on several pretrained encoders for text classification. We achieve an F1 score of 0.75 for the positive class on the test set.

## introduction automatic drug name recognition has mostly been studied in terms of extracting drug names from medical documents and biomedical articles @xcite . however, expanding the same task to extracting drug names from tweets poses a lot more challenges. tweets are shorter and do not provide enough context compared to academic biomedical articles; they also contain ambiguity, noise, and misspellings, especially in the form of colloquially used terms for the same drugs @xcite . the shared task of the 5th social media mining for health applications specifically aims at tasks that use natural language processing for health applications. we participated in task 1, which is defined as the automatic classification of tweets that mention medications. we use several pre-trained encoders such as bert @xcite , biobert @xcite , clinical biobert @xcite , scibert @xcite , roberta @xcite , biomed-roberta @xcite , electra @xcite and ernie 2.0 @xcite for the classification of tweets. ## dataset the training and the validation dataset were provided to us by the organizers of smm4h2020. the training dataset consisted of 55419 tweets, with only 146 positive tweets and 55273 negative tweets. the validation dataset consisted of 13853 tweets, with only 35 positive tweets and 13818 negative tweets. the dataset is highly imbalanced, and the positive tweets account for only 0.2% of the whole dataset. the test set for submitting our system predictions consisted of 29687 tweets. ## experiments and system descriptions along with bert, we use several other pre-trained sentence encoders that are a result of various improvisations over bert such as biobert @xcite , clinical biobert @xcite , scibert @xcite , roberta @xcite , and biomed-roberta @xcite . for all of the above pre-trained sentence encoders, we use the pytorch implementation through the transformers library 1 fine-tuning them for 3 epochs with a learning rate of 2e-5, maximum sequence length as 128, and a batch size of 8. unlike bert, electra @xcite ) uses an alternative pretraining task, called replaced token detection. it aims to be more sample-efficient than masked language modeling used in bert. using the implementation of electra foot_1 provided by the authors we fine-tuned electra for 3 epochs with a learning rate of 1e-4, maximum sequence length as 128, and a batch size of 32. ernie 2.0 @xcite provides a continual pre-training framework to incrementally build several pre-training tasks that focus on extracting lexical, syntactic, and semantic information from the training corpora. we use the implementation in paddlepaddle foot_2 provided by the authors and fine-tune ernie 2.0 for 3 epochs with a learning rate of 3e-5, maximum sequence length as 128, and a batch size of 64. ## results and discussion due to the imbalance in the dataset, the metric used for evaluating the systems is the f1 score for the positive class, where positive class refers to the set of tweets that mention a drug or a dietary supplement.