IIITN NLP at SMM 4 H 2021 Tasks: Transformer Models for Classification on Health-Related Imbalanced T witter Datasets (2021)

Abstract: With increasing users sharing health-related information on social media, there has been a rise in using social media for health monitoring and surveillance. In this paper, we present a system that addresses classic health-related binary classification problems presented in Tasks 1a, 4, and 8 of the 6th edition of Social Media Mining for Health Applications (SMM4H) shared tasks. We developed a system based on RoBERTa (for Task 1a & 4) and BioBERT (for Task 8). Furthermore, we address the challenge of the imbalanced dataset and propose techniques such as undersampling, oversampling, and data augmentation to overcome the imbalanced nature of a given health-related dataset.

## introduction twitter has gained a huge popularity among all the social media platforms, especially to share and discuss information related to various aspects of life, including health-related problems. analysing these health related tweets and extracting the meaningful information from them is an important task for offering better health related services. with the advancements in sequential deep models, natural language processing (nlp) and underlying processes got benefited from it and effective automation is introduced for the various nlp processes to a great extent. healthcare research community has developed a keen interest in processing these health related information efficiently using advancements of deep learning. the sixth social media mining for health applications (smm4h) shared tasks focus on addressing such classic health related problems applied to twitter micro-corpus (tweets) @xcite . our team participated in three different shared binary classification tasks viz. task 1a, task 4, and task 8. task 1a focuses on distinguishing tweets mentioning adverse drug effects (ade) from other tweets (noade). @xcite focused on the identification of tweets mentioning drugs having potential signals for ades. task 4 focuses on distinguishing tweets mentioning adverse potential outcomes (apo) from other tweets (noapo). task 8 focuses on segregating the tweets containing self-reports (s) of breast cancer from other tweets (nr). the datasets provided for the shared tasks 1a and 8 are highly imbalanced. however, dataset for the shared task 4 is comparative balanced. due to the scarcity of users tweeting on health topics, most of the datasets on these topics are highly imbalanced in nature. @xcite gives a broad overview on the various balancing techniques applied on various medical datasets. @xcite demonstrates the effect of strategies such as oversampling and cost-sensitivity on various health-related datasets. @xcite presents extension of this work on costsensitivity to allow models such as biobert and bert to generalize well on imbalanced datasets. @xcite @xcite also present strategies such as text generation techniques, embedded feature extraction methods to generalize the classifier on an imbalanced dataset. we propose transformer based classification models for the binary classification for all the aforementioned tasks. we especially address the class imbalance in the datasets, for task 1a and task 8. we experiment with techniques such as undersampling, oversampling, and data augmentation to address the datasets imbalance for these tasks. the rest of the paper is organized as follows. section 2 covers the underlying datasets for the three shared tasks, their characteristics, preprocessing details, and sampling techniques to address the inherent imbalance in the dataset. section 3 presents the classification models for the shared tasks. re- ## dataset: sampling techniques and preprocessing the datasets for the shared tasks were collected in the form of english tweets. the datasets were well annotated for each of the shared tasks. we majorly employ three dataset balancing techniques viz. undersampling, oversampling, and augmentation. ## system description and model we employ transformer based models and their architectural variants for all the shared tasks, along with dataset balancing techniques described in the previous section. for all the tasks, the experiments have been performed using the scikit-learn, tensorflow foot_3 , pytorch foot_4 and flair @xcite frameworks. ## results & discussions all the experiments were performed on an intel core i5 cpu @2.50ghz, 8gb ram machine having 4 logical cores. the task wise results can be presented as follows: ## conclusions we proposed a text classification pipeline while also making an attempt to handle dataset imbalance corresponding to three different shared tasks in smm4h'21 @xcite . we conclude that data augmentation gives best performance on highly imbalanced datasets. moreover, augmentation provides better results in case of comparatively balanced datasets. as part of future work, additional experiments are planned to further analyze strategies to improve the performance of the model on the dataset.