BERT based Adverse Drug Effect Tweet Classification (2021)

Abstract: This paper describes models developed for the Social Media Mining for Health (SMM4H) 2021 shared tasks. Our team participated in the first subtask that classifies tweets with Adverse Drug Effect (ADE) mentions. Our best performing model utilizes BERTweet followed by a single layer of BiLSTM. The system achieves an F-score of 0.45 on the test set without the use of any auxiliary resources such as Part-of-Speech tags, dependency tags, or knowledge from medical dictionaries.

## introduction in this effort, we focus on detecting tweets that have ade mentions as a part of the social media mining for health (#smm4h) -2021 shared tasks @xcite . organizers of smm4h task 1 provided datasets of english tweets with binary annotations of 1 and 0 indicating the presence or absence of ade mentions in the tweet. we develop a robust system against the class imbalance problem in the dataset that classifies tweets containing at least one ade mention. we also validate the importance of emojis and hashtags in ade classification empirically. ## method we explore three bert-based models for classification: (i) bert @xcite , (ii) roberta @xcite , and (iii) bertweet @xcite . we pass the input through our bertbased models to get token representations. to compute the sentence representations, we consider two cases -i) [cls] token (fine-tuning) ii) we pass token representations without [cls] and [sep] through a single layer bilstm and concatenate the forward and backward context. the sentence representation is passed through a fully connected neural network layer followed by a sigmoid activation to predict probabilities. to tackle class imbalance, we experiment with oversampling, undersampling, and addition of perclass penalties in the objective function. for oversampling approach, we randomly sampled positive examples with replacement until each class contained 10,000 tweets. for the undersampling approach, we randomly sample negative examples to create a balanced training dataset. ## experiments for the classification task, each bert model is trained for 10 epochs with a learning rate of 1 * 10 2017 ). we set the batch size to 32 and the maximum sequence length to 128. to tackle class imbalance, we add weights to the standard cross-entropy loss. we set weights as 0.7 and 0.3 for ade and noade classes, respectively. we utilize pytorch foot_1 implementation of bert for training. we train roberta over , roberta under and bert base -unweighted, using standard unweighted cross-entropy loss. we conduct model selection for every 200 steps against the validation set using the f1-score of the ade class for comparison. ## discussion it is evident from bertweet outperforms bertweet raw , which uses preprocessing techniques described in @xcite . our preprocessing steps are inspired by @xcite with the only difference being that we remove all user mentions and web/url links from the tweet. we empirically validate our intuition that the user mentions, web links act as noise in the text and do not provide any valuable information needed for the classification task. ## conclusion in this work, we explore an application of bert to the task of binary classification on english tweets. we validate that use of per-class penalties in the objective function helped in overcoming the class imbalance problem. we have empirically evaluated differently tuned model versions and preprocessing methods against f1-score for the "ade" class. experiments have shown that our model has achieved an f1-score of 0.46, precision of 0.523, and recall of 0.409 on the test set. the future directions would be to evaluate the potential of supplementary resources in our model, such as part-of-speech tags, dependency tags, knowledge from medical dictionaries (such as med-dra).