FBK @ SMM 4 H 2020: R o BERT a for Detecting Medications on T witter (2020)

Abstract: This paper describes a classifier for tweets that mention medications or supplements, based on a pretrained transformer. We developed such a system for our participation in Subtask 1 of the Social Media Mining for Health Application workshop, which featured an extremely unbalanced dataset. The model showed promising results, with an F1 of 0.8 (task mean: 0.66).

## introduction twitter is a valuable source of user-generated data, including health data, and might be used for monitoring drug abuse and adverse effects online. however, tweets that mention medications need to be flagged first. to do so, lists of drugs are not enough: this is due to common spelling mistakes on social networks and language ambiguity (e.g. xanax might be a drug or a belgian band). the substask 1 of the 2020 social media mining for health workshop (#smm4h) challenged participants to identify tweets containing drugs or dietary supplements' mentions. the corpus showed a natural, highly imbalanced distribution of labels, with extremely rare positive occurrences. in this paper, we describe our entries, based on a pretrained roberta model @xcite . ## datasets the main dataset (ds2020), for which participants generated predictions, is a corpus of tweets by 112 pregnant women, as described in @xcite . its label distribution is unbalanced (181+/69091-), with positive instances only constituting 0.2% of the training set. given the scarcity of positive samples, we also employed another dataset (ds2018), previously used in the 2018 #smm4h shared tasks @xcite . it consists of 9611 tweets from 7584 users, annotated for medications' and supplements' mentions; the dataset was artificially balanced (4975+/4647-). additionally, we used a large corpus of about 5 million general-domain, unlabelled english tweets. ## method we submitted three runs to the leaderboard. all models were based on a roberta pretrained transformer. roberta is an enhancement of bert @xcite , with a modified training process (objective and data) and hyperparameters. the first run (base model) is a simple pretrained roberta base uncased classifier. we first fine-tuned the original model on the ds2018 dataset to classify tweets mentioning medications and supplements. the hyperparameters were selected to maximize f 1 for the ds2020 validation set. following a transfer learning approach, we used the resulting weights to initialize the final model and trained it on the ds2020 data. the original text was not preprocessed; for both datasets, we only used the text of the tweets, discarding the date, time, and user. no list of medications or other prior knowledge was employed and no explicit training on the medical domain was performed. in the second run (mlm model), we followed a similar pipeline, but pretraining roberta on a large corpus of tweets as a first step. the training was performed with a masked language model (mlm) twitter corpus fine-tuning ds 2018 fine-tuning ds 2020 objective, virtually continuing the original transformer's training, to adapt it to the twitter language specificities. masked language modelling consists of masking out a fraction of the input tokens and training the model to predict the original tokens based on context. finally, we observed that, when trained for a varying number of epochs, the mlm models exhibited comparable f 1 but made different errors on the validation set. for this reason, we chose an ensemble of five mlm checkpoints (ensamble mlm model) as our third run; we chose all checkpoints to have good results on validation data, but varying precision/recall thresholds. at test time, majority voting was used. once the hyperparameters were fixed on the validation set, all final models were trained on both the train and validation data to predict the test set. figure 1 presents the pipeline employed for the third run. ## experiments and results in this section, we report our experimental results. given the skewness of the dataset, system performances were measured in terms of f 1 for the positive class (i.e. tweets containing medications or supplements). for all models, we used a batch size of 32, and experimentally found a constant learning rate of 10 -6 to perform best. surprisingly, a long fine-tuning seemed beneficial: training was stopped at epoch 132 (base) and 73 (mlm) after evaluating the models at each epoch for a maximum of 200. figure 2 shows the errors made by the base model. note that some tweets might be hard to classify according to the gold standard even for a human annotator. ## conclusion we presented our entries to the #smm4h shared task. the model obtained promising results albeit trained on tweets' text only. pretraining on the ds2018 set first highly improves the model performance (+0.1 f 1 ), while the mlm pretrain seems detrimental. this might be due to the quality of the unannotated dataset, which we used as obtained by the twitter api, without any filtering or preprocessing.