Adverse Drug Reaction Detection in T witter Using R o BERT a and Rules (2020)

Abstract: This paper describes the method we developed for the Task 2 English variation of the Social Media Mining for Health Applications (SMM4H) 2020 shared task. The task was to classify tweets containing adverse effects (AE) after medication intake. Our approach combined transfer learning using a RoBERTa Large Transformer model with a rule-based post-prediction correction to improve model precision. The model’s F1-Score of 0.56 on the test dataset was 10% better than the mean of the F1-Score of the best submissions in the task.

## introduction the enormous quantity of self-reporting about medication effects in social media provides an opportunity for additional surveillance of medication effects, supplementing traditional health-related reporting systems @xcite . task 2 of the smm4h shared task challenge @xcite was defined as a binary classification task to identify tweets reporting an adverse effect (ae) from taking a medication. the language differences between aes and indications (the reasons for using a medication) in the data could be quite subtle and increased the difficulty of the task. datasets were available in english, french and russian, but we only worked with the english dataset. fine tuning of state-of-the-art language models for classification has proven to be highly effective when applied to small but varied text collections @xcite . we chose a hugging face roberta large transformer model @xcite based on our previous experience in identifying vaccine adverse reaction mentions in tweets. informed by previous studies that have used lexicons to improve classification results @xcite , we developed a post-prediction rule-based correction based on a lexicon of phrases that mostly appeared in what we judged were non-ae tweets in the test data. this improved the f1-score by 2%. ## data description and preparation the shared task english data was supplied as two tab-delimited text files, one with 20,544 tweets as training data, and another of 5,134 tweets as a validation dataset. each tweet was accompanied with a tweet id, a user id, and a "positive" class label of 1 for an ae, and a "negative" class label of 0 for a non-ae. eventually an unlabeled test dataset of 4,759 tweets was provided by the challenge organizers for submission to the task. around 9.3% of the data was designated as the positive ae class in the labeled datasets. a comparison of tweets indicates how difficult this challenge is, there sometimes seemed to be little difference between an ae and something like it. for instance, these consecutive entries with their classes, both describing a side effect of pain following rivaroxaban intake: negative class: "04.40 just taken flecainide and 2 paracetamol to dull pain side effect of rivaroxaban."; positive class: "20.05 day 16 rivaroxaban diary. intense back and knee pain most of day; taken more paracetamol. had to shop, walking painful". there were several aspects of the data which had a bearing on data preparation and classifier training, and on our conclusions about possible future directions: (i) we identified 423 texts that were in both the training and validation datasets, and (ii) within these datasets there were also duplicates; (iii) the data was highly imbalanced. to fix records that existed in both datasets, we removed them from the training data. we then also removed remaining duplicates from within each dataset. after cleaning, there were 1,896 positive labels in the training data and 473 in the validation data. the training dataset then consisted of 20,110 tweets -the 1,896 positive labels with the rest being all the negative labels. the de-duplicated validation dataset consisted of 2,167 negative and 473 positive labels, a total of 4,807 tweets. we addressed the class imbalance in the training dataset by randomly sorting the data then splitting the negative labels of the training data into 7 datasets and combining each with all the positive labels. therefore, each training dataset comprised of 2,602 negative labels and 1,896 positive labels, a total of 4,498 tweets. the tweets text was not altered, and we added no features. data was exported as a tab-delimited format suitable for use by the transformer model. the hugging face libraries we used provided routines to tokenize and vectorize the text to transformer model requirements. ## model we used hugging face's roberta large pytorch transformer model @xcite for classification, specifically the class transformers.robertaforsequenceclassification, which is described in the hugging face documentation as a "roberta model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output)". roberta is an acronym for "robustly optimized bert pretraining approach" and was developed by facebook to improve on google's original bert -"bidirectional encoder representations from transformers" @xcite . a very summary explanation is that the models are based on transformer language models and utilize multiheaded encoder/decoder attention mechanisms, which dispense with recurrence and convolutions entirely @xcite . they use intentionally masked sections of text to learn to predict the most probable words in sentences. roberta improves on bert by removing its next-sentence pretraining objective; by using larger mini-batches and learning rates; and using an order of magnitude more data and for a longer time than bert was trained on. roberta large was the largest model of the available roberta models on the hugging face site, we chose it because our tests on similar twitter data have shown improvement in f1-scores compared to bert. ## generated texts when analyzing the data, we found two users in the training data who between them had 251 tweets that consisted of what looked like generated text. the tweets contained non-sensical phrases intermingled with key medication-related words, e.g.: "scuba dive ventolin hfa two brett butlers. the former"; only two of these texts were labelled as ae. we assessed the functionality of the 251 generated texts by learning classifiers on seven balanced datasets without the generated tweets, retaining the best model from each dataset. compared to the models trained on all the data we found a higher recall to precision ratio, but that f1-scores were uniformly worse by at least 0.01. in this case we found an ensemble of the top 5 models gave the best f1-score, but at 0.661 on the validation data it was 0.024 worse than the 0.685 of the best ensemble trained on all the data. although it was not performant, we wanted to know how this worked with the test data and so made a submission using the ensemble with a manual correction. its test f1-score was 0.46, which although identical with the mean best test dataset f1-score for the challenge, was 10% below our best score of 0.56 obtained when the generated text was retained. it was apparent that the generated text had a correcting effect by removing false positives. it seemed possible that this was due to their containing text with potentially ae indicative words surrounded with words and language structure that was not indicative of ae. a 10% difference was significant and so it seems likely that these texts have been introduced by the challenge organizers as a corrective measure.