Identifying Personal Experience Tweets of Medication Effects Using Pre-trained R o BERT a Language Model and Its Updating (2020)

Abstract: Post-market surveillance, the practice of monitoring the safe use of pharmaceutical drugs is an important part of pharmacovigilance. Being able to collect personal experience related to pharmaceutical product use could help us gain insight into how the human body reacts to different medications. Twitter, a popular social media service, is being considered as an important alternative data source for collecting personal experience information with medications. Identifying personal experience tweets is a challenging classification task in natural language processing. In this study, we utilized three methods based on Facebook’s Robustly Optimized BERT Pretraining Approach (RoBERTa) to predict personal experience tweets related to medication use: the first one combines the pre-trained RoBERTa model with a classifier, the second combines the updated pre-trained RoBERTa model using a corpus of unlabeled tweets with a classifier, and the third combines the RoBERTa model that was trained with our unlabeled tweets from scratch with the classifier too. Our results show that all of these approaches outperform the published methods (Word Embedding + LSTM) in classification performance (p < 0.05), and updating the pre-trained language model with tweets related to medications could even improve the performance further.

## introduction personal experience is an important piece of information for health-related surveillance activities. understanding one's health experience can help gain insight into the status of one's health, changes of one's health condition after the intervention, or the effects related to any medications one took. investigating effects related to the use of pharmaceutical products is an important activity of post-market surveillance. first-hand information related to patients' medication use most directly reflects the effects of the medication, beneficially or adversely. in that case, it is necessary to find valuable data sources and construct efficient methods for processing and analyzing this data. the widespread availability of social media has made it possible for people to share their personal experiences freely online. twitter is one of the most prevalent social media services, and studies have shown that the data from social media such as twitter has been applied to many health-related applications. examples are as follows: drug adverse events @xcite , public health @xcite , mental health @xcite , dental pain @xcite , influenza @xcite @xcite @xcite , breast cancer @xcite , and epidemic outbreak and spread detection @xcite . personal experience is about a person's encounters or observations related to his or her life. personal experience information related to the use of medication is of unique value for post-market surveillance because it is the first-hand information that reflects the health condition changes due to medication usage. personal experience tweets (pets) related to medication use are a kind of twitter post expressing one's personal experience and information after the administration of medication. the types of experiences could be undesirable feelings caused by medications' sideeffects, or beneficial effects that help improve a ## related works distinguishing pets and non-pets can be treated as a binary classification task. in the conventional machine learning field, algorithms require a set of manually engineered features extracted from the raw text and/or metadata @xcite , usually known as feature engineering, and features chosen can significantly impact the classifier's performance. however, extracting/engineering valuable yet optimal features from tweets is difficult due to the limitation of human knowledge and understanding even for the domain experts. besides, feature engineering extracts features that are typically based on the analysis of statistics regarding information gain usually with little or no direct consideration of the semantics. in other words, conventional machine learning with feature engineering methods may not be optimal for this task. efforts of performance improvement have been made in previous research endeavors in the task of predicting personal experience tweets related to medication effects. in one of the earliest efforts, personal pronouns were considered as an important feature @xcite . later, alvaro and colleagues engineered a set of features @xcite , and their features include twitterspecific features, n-grams, punctuation elements, and topics, but the group decided to discard the topic feature due to the significant efforts required and its minimum merit of improving classification performance. a set of 22 engineered features based upon both textual content and metadata of tweets was proposed in constructing a corpus of personal experience tweets @xcite . subsequently, calix and colleagues introduced the concept of deep gramulator to include a textual feature that contains expressions in one class but not in the opposite class, to improve the discriminatory ability of the classification @xcite . advancement in neural embedding, which demonstrated state-of-art results in many classification tasks on textual data, motivated the development of a new approach of combining word embedding (word2vec) and a recurrent neural network which demonstrated a significant improvement of classification performance (@xmath0.05) @xcite . thanks to the development of word embedding techniques and the long short term memory (lstm) neural network, @xcite assessed a set of different word embedding techniques: glove @xcite , fasttext @xcite ) and word2vec @xcite to build vector space models (vsm) to represent the semantics of tweets by learning from a corpus of 22 million unlabeled tweets. the vector representations of tweets were fed into an lstm neural network for classification. all of these methods achieved better performance in classification measures than the previous methods with 22 human-engineered features using conventional classification algorithms @xcite . unlike the word embedding + lstm method, which need to learn the vsm first and then train the lstm network from scratch for classification, google introduced a fine-tuning based approach by proposing the bidirectional encoder representations from transformers (bert) model @xcite , which achieved recordbreaking results in 18 downstream nlp tasks. besides, google's new method relies on contextual information rather than term co-occurrences. after that, facebook made some optimization based on bert and released a robustly optimized bert pretraining approach (roberta) model ( liu et al. 2019 ) which generated even better performance than bert in downstream tasks. one important and useful aspect of both approaches is that the pretrained models can be updated with new data, without the need to generate a new model from scratch with the added data, which generally requires a significant amount of computation resources. in this study, we set the performance of the word embedding + lstm neural network method as the baseline and investigated the performance improvements of pets prediction with the pretrained roberta language model. we also studied a procedure of updating the pre-trained roberta language model and training the roberta from scratch with the medication-related tweets and analyzing the impact on the performance change. ## method in this work, we introduced three ways to identify personal experience tweets about medication effects by using roberta language model: (1) pretrained roberta -adding a classifier to the standard pre-trained roberta model and finetuning the model for classification; (2) updated roberta -updating the pre-trained roberta language model with our dataset first, then adding a classifier to roberta and fine-tuning the model for classification; and (3) twitter robertatraining the roberta language model with our corpus of unannotated tweets from scratch, then adding a classifier for classification. finally, 10fold cross-validation was performed to gather the performance data, and statistical analysis was performed to determine if the differences in performance among different methods were due to the chance. the pipelines of data processing and analysis is illustrated in figure 1 . our process started with gathering twitter data and performing text encoding after preprocessing. afterwards, the encoded texts were used with the roberta model and the classifier for our methods. the left pipeline is for the pretrained roberta approach, the middle one for updated roberta, and the right one for twitter roberta. ## results to compare the performance differences between our methods and baseline methods, 10-fold crossvalidation was conducted for each method and the mean value of each classification measure was collected. ## discussions according to the results in a noticeable improvement between pre-trained and updated roberta models and baseline methods is the precision and recall, whereas the precision of twitter roberta model remained relatively unchanged at the same time. the recall is the sensitivity of how many true instances are predicted correctly and precision rates how many positive predictions are correct. a higher recall could help the model discover more potential positive instances and higher precision means more true positives (tp) and less false positives (fp) in the prediction. in other words, roberta models can improve the sensitivity and identify pets more precisely, resulting in more true positives in the predicted pet class. another remarkable measure could be the roc/auc score, which was also improved significantly as shown by the curves in figure 4 . roc (receiver operating characteristic) is a curve plotting true positive rate (tpr, or sensitivity) in the y-axis and false positive rate (fpr, or 1-specificity) in the x-axis, and is commonly used to show how well the model can distinguish two different objects. the area under the curve (auc) of roc is used to quantify the score of roc. the results in our methods also achieved a modest improvement in accuracy, but it could not be interpreted as that better accuracy leads to better performance. because our dataset is imbalanced (pets: non-@xmath2: 3.16, as shown in the results also show that performing lm updating before classifier fine-tuning could yield more improvement in accuracy, precision, f1, and auc. nevertheless, the p-values indicate that they are not significant if updating the lm for more steps. but as for the twitter roberta model, which was trained from scratch, the steps of training affected performances in some measures which were supported by our statistical analysis. this outcome suggests that a larger number of steps are needed for performance improvement when training from scratch, and small steps are enough for lm updating to achieve better performance than the original roberta model. the possible reason for the improvement of these roberta-based methods over baseline approaches could be attributed to the level of features. as is known, the features extracted by vsm such as word2vec, which is based upon word-level and co-occurrence. but roberta, which extracts contextual-level features, maybe more powerful in processing tweet-like text which is poisoned by misspelling and incorrect grammars. the possible explanation for the performance difference between updated roberta and twitter roberta can be the slow learning process. the updating process is based on the pre-trained roberta model, which is already pre-trained with a very large dataset by facebook. it may be easier to adapt itself to our dataset, and the larger number of updating steps did less to help improve performance. but for twitter roberta, since it was trained from scratch and only 15% of tokens were randomly masked, the model could only learn a small part of sentences for each step. therefore, it may take more time to learn the data distribution, and the larger number of training steps is recommended. ## conclusion in this study, we investigated different ways to use facebook's roberta model to improve performance in predicting personal experience tweets on medication use. our results demonstrated that using the fine-tuning method on the pre-trained roberta model achieved better classification performance than previous word embedding + lstm methods, and the original pretrained roberta could perform better than training a new roberta model from scratch. more importantly, updating the pre-trained roberta language model with our data could yield better performance. the 10-fold crossvalidation was used to test statistically the performance differences between our approaches and baseline methods. the results confirmed that the improvement does exist with statistical significance (@xmath3.05). this suggests the pretrained roberta model and lm updating method are better choices for this task and significantly boost the capability to identify personal experience tweets. it is conceivable that our method could apply to other classification tasks using twitter data related to health issues.