Ensemble BERT for Classifying Medication-mentioning Tweets (2020)

Abstract: Twitter is a valuable source of patient-generated data that has been used in various population health studies. The first step in many of these studies is to identify and capture Twitter messages (tweets) containing medication mentions. In this article, we describe our submission to Task 1 of the Social Media Mining for Health Applications (SMM4H) Shared Task 2020. This task challenged participants to detect tweets that mention medications or dietary supplements in a natural, highly imbalance dataset. Our system combined a handcrafted preprocessing step with an ensemble of 20 BERT-based classifiers generated by dividing the training dataset into subsets using 10-fold cross validation and exploiting two BERT embedding models. Our system ranked first in this task, and improved the average F1 score across all participating teams by 19.07% with a precision, recall, and F1 on the test set of 83.75%, 87.01%, and 85.35% respectively.

## introduction social media platforms such as twitter have been used in various public health studies. in these studies, detecting tweets containing health-related words such as diseases, treatments and medications is a fundamental yet difficult step. these difficulties are exacerbated by the short length and informal nature of tweets, which often contain non-standard grammar, frequent misspellings, many contractions, extensive slang, and combined symbols (emojis/emoticons) to express emotion. despite these difficulties, unique insights can be gained from exploring twitter data which motivates further research on this task. task 1 of smm4h shared task 2020 @xcite challenged participants to develop an automatic classification system to identify tweets mentioning medications or dietary supplements. the task was formulated as a binary classification task, in which given a set of tweets a system should predict the label for each tweet. organizers created the upennhlp twitter pregnancy corpus @xcite and divided it into training, validation, and test sets. the training and validation datasets contained tweets labeled with either positive or negative labels indicating whether or not they contain a medication or dietary supplement mention. the test set contained tweets with labels removed, and was withheld from participants until a short evaluation period during which participants predicted labels for each tweet. participants were evaluated based on their system's performance (f1 score for the positive class) at generating labels for the test set. the class distribution of the training, validation, and test sets was highly imbalanced. only about 0.26% of the tweets mention medications, which is typical for this task and common in practice. as a baseline the shared task organizers described the kusuri classifier @xcite . kusuri was trained in a combination of the training and validation datasets and achieved good performance in detecting positive tweets in this test dataset at f1 score 78.79%. we sought to improve upon kusuri by developing a system that combines a rule-based preprocessing component, a pre-filtering component, and an ensemble of bert-based classifiers to predict the label of each tweet (figure 1 ). in our best performing system, we used 10-fold cross validation to divide the data into 10 figure 1 : an overview of the system for automatic classification of tweets mentioning medications subsets, then trained a bio+clinical bert classifier and a bert-large uncased classifier on each subset to obtain 20 classifiers. these 20 classifiers were ensembled using the average of their predicted probabilities to predict the final class labels. we further increased the performance of this system by training it on additional data from smm4h 2018 which consisted of a very similar task. ## conclusion in this work, we developed an ensemble tweet classifier based on bert combined with pre-processing and pre-filtering steps. the system was evaluated on a natural, highly imbalanced dataset, achieved the highest f1 score in task 1 of the smm4h shared task 2020. we found that the best system configuration consisted of an ensemble of 20 bert-based classifiers built using bert-large uncased and bio+clinical bert ensembled using an average method. we also found that including related training data improved performance, and that the pre-filtering step did not improve performance. for future work, we plan to experiment with different neural network classification layers in addition to bert outputs.