UB Health Miners@ SMM 4 H ’22: Exploring Pre-processing Techniques To Classify Tweets Using Transformer Based Pipelines. (2022)

Abstract: Here we discuss our implementation of two tasks in the Social Media Mining for Health Applications (SMM4H) 2022 shared tasks – classification, detection, and normalization of Adverse Events (AE) mentioned in English tweets (Task 1) and classification of English tweets self-reporting exact age (Task 4). We have explored different methods and models for binary classification, multi-class classification and named entity recognition (NER) for these tasks. We have also processed the provided dataset for noise, imbalance, and creative language expression from data. Using diverse NLP methods we classified tweets for mentions of adverse drug effects (ADEs) and self-reporting the exact age in the tweets. Further, extracted reactions from the tweets and normalized these adverse effects to a standard concept ID in the MedDRA vocabulary.

## introduction since 2005 in the us, the percentage of american adults using social media has risen from 53% in 2012 to 72% in 2021 according to the research and tracking by pew research center. tremendous amounts of data are being generated on social media, where people express different aspects of their lives to their social circle. many people also express their thoughts on various health-related topics. all this data presents significant opportunities for studying it to monitor people's health and the factors affecting their health. our work is inspired by the current research using transformer architectures, and the results that autobots ensemble team @xcite had achieved at smm4h 2020 and the kfu nlp team @xcite had achieved at smm4h 2019 using bert @xcite as well as by the benchmark work and system configuration in re-portage @xcite . task 1 consists of 3 subtasks, classification, extraction, and normalization. for a binary classi-fication task, distinguishing tweets that report an adverse effect (ae) of medication are annotated as "1", from those that do not are annotated as "0", subtle causal language variation in the tweets needs to be addressed between aes and indications. for extractions, the beginning and end offsets of these spans and the actually extracted spans are provided in the dataset for training the model and removing the spans for the unseen data. for the normalization subtask, the extracted spans, and the normalized standard concept ids of meddra foot_0 vocabulary are provided. task 4 is a binary classification task that automatically distinguishes tweets that self-report the user's exact age, annotated as "1", from those that do not, annotated as "0". automatically identifying the actual age, rather than their age groups, would enable the large-scale use of social media data for applications that do not align with the predefined age groupings of extant models, including health applications such as identifying specific age-related risk factors for observational studies, or selecting age-based study populations. the dataset @xcite provided for this task has 17,120 annotated tweets for classifying tweets containing adverse drug events (ades) related to drugs from the ones that do not contain ades. for classification, we used a deep neural network classifier based roberta-based @xcite , pre-trained transformer model from hugging face foot_1 @xcite . for the roberta-based classifier, we preprocessed the tweets by normalizing the hashtags to words, emoticons to expressions, and removing the duplicates, and urls from the dataset. after getting the embeddings of the pre-processed tweet, we took the mean of the sequence output to be the pooled output and passed it to the hidden layer, dropout layer (drop rate of 0.5), and a linear layer that predicts the class of each tweet. for training, we used adam optimizer (kingma and ba, 2014), 10 epochs, warmup of 0.2, the learning rate of 1e-5, and weight decay of 0.001. as the data set was quite imbalanced, with approximately 10% data of "ades" class, and the remaining 90% data for the "non-ades" class, we have tried different techniques to balance the dataset by downsampling the major class, data augmentation of the positive class by back-translation @xcite method to german as well as russian, paraphrasing using the parrot paraphraser, and assigning weights to the positive class. ## error analysis we observe that the model performs poorly to classify the tweets containing drug names and other medical terms. for the named entity recogni-tion, the model using spacy fails in the cases where multiple spans must be extracted from the tweet. for the normalization, for improving the f1 score, more data is required as we have only 8,815 data points including the dataset of smm4h 2022, smm4h 2020, and cadec corpus trained to classify a total of 1,023 unique meddra ids. ## result and analysis the performance evaluation metrics for the tasks are precision (p), recall (r), and f1-score (f1) computed on the positive class which is the minority class. the above result tables report the various techniques and the scores of those respective models. different hyperparameters were tested to get the optimum f1 score of each of the models and different padding/truncating length lines 40, 32, and 16 of the embedding for the normalization task were explored to see if that affected the results for the tasks and padding/truncating to embedding length of 16 gave the best results. ## conclusion all the datasets that were provided were highly imbalanced amongst the classes and many techniques were explored to overcome these imbalances, namely random downsampling of the majority class, increasing the number of minority classes by paraphrasing the data of those classes, data augmentation by a round trip translation from english to german and back to english and this was also tried with russian to create more data points. from the whole project, it was learned that the balance of the data between classes does not guarantee the performance of the model. it can also be seen that the preprocessing step is also very important to preserve the context of the tokens with respect to each other. for future scope, a medical domain-based pre-trained language model, with the present ones in an ensemble architecture might work to improve the score of the system.