uestcc@ SMM 4 H ’22: R o BERT a based Adverse Drug Events Classification on Tweets (2022)

Abstract: This is a description of our participation in the ADE Mining in English Tweets shared task, organized by the Social Media Mining for Health SMM4H 2022 workshop. We participate in the subtask a of shared Task 1, and the paper introduces the system we developed for solving the task. The task requires classifying the given tweets by whether they mention the Adverse Drug Effects. We utilize RoBERTa model and apply several methods during training and finetuning period. We also try to improve the performance of our system by preprocessing the dataset but improve the precision only. The results of our system on test set are 0.601 in F1- score, 0.705 in precision, and 0.524 in recall.

## introduction adverse drug events (ades) refer to negative side effects related to the drug. in the area of social media pharmacovigilance, mining ades from social media is one of the most studied topics. in the social media mining for health 2022 (smm4h) shared task 1a @xcite , we focus on classifying tweets reporting ades and labeling them with ade or noade. in training process, we input 18000 labeled tweets provided by organizers to roberta model and finetune the model by several practical methods such as stochastic weight averaging (swa), then output the prediction results. the system is shown in figure 1 . we also attempt to preprocess the tweets by deleting some stop words and emojis, the performance of all approaches is shown in the paper. ## data the dataset @xcite provided by the organizer includes a training set of 17,385 tweets, a validation set of 915 tweets, and a test set of 10,984 tweets. it is worth noting that the sample proportion is highly unbalanced, tweets mentioning ades are only 7% of the training data. for data preprocessing, we save the original training data at first containing the redundant symbols and emojis in the tweets. then we perform following preprocessing on the original dataset to construct another dataset as a comparison: a) lowercase all words and remove whitespace. b) remove instances of '@user' followed by '_'. c) remove extranous characters and emojis. d) remove stopwords foot_0 . ## method in this section, we introduce the methodology we propose to accomplish the task of the competition. we directly use the original dataset as input of the system at first. then we pass the input into the bert-based model roberta (lee and toutanova, 2018) and get the token representations. by inputting the representations through the fully connected layers and finetuning the model, we obtain the final model and use it to predict the test set. during the period of finetuning, we add several methods to improve the performance of our model: ## experiments in the shared task 1a, the roberta model is trained for 10 epochs with a learning rate 5 × 10 -5 using adam optimizer @xcite . we set the batch size in training duration to 16 and 64 in validation and test duration, respectively. in addition, embedding size, output size, and sequence length in the experiments are set up as 768, 768, and 256. we also conduct the ablation experiments to test the performance of several methods mentioned in last section. we first set the parameter alpha to 0.2 during the process of mix-up. then we add fgm and pgd methods into the training process and use swa with a learning rate 5 × 10 -5 . as the results of ablation experiment shown in in order to verify the effectiveness of data preprocessing, we use the datasets before and after data preprocessing to train the model respectively. the results on validation set of two datasets are in as for the prediction results of test set, points of three evaluation metrics significantly drop compared with results on validation set. in our two submitted results of prediction, one uses data preprocessing and another not, both are predicted by the roberta model added with three methods. in addition, full training set (validation set + training set) is used to train the model. results of our submissions are demonstrated in ## conclusion in this work, to complete the task of binary classification on tweets, we propose a system based on roberta model applying with several methods to improve the generalization ability of our model, and get the results of 0.731 in precision, 0.524 in recall, and 0.601 in f1 score. we also validate the effectiveness of finetuning methods and data preprocessing in the experiments. the performance of our proposed system on test dataset compared with the mean results indicates the robustness and generalization ability of our system, and we will continue to improve it in the future.