ARGUABLY @ SMM 4 H ’22: Classification of Health Related Tweets using Ensemble, Zero-Shot and Fine-Tuned Language Model (2022)

Abstract: With the increase in the use of social media, people have become more outspoken and are using platforms like Reddit, Facebook, and Twitter to express their views and share the medical challenges they are facing. This data is a valuable source of medical insight and is often used for healthcare research. This paper describes our participation in Task 1a, 2a, 2b, 3, 5, 6, 7, and 9 organized by SMM4H 2022. We have proposed two transformer-based approaches to handle the classification tasks. The first approach is fine-tuning single language models. The second approach is ensembling the results of BERT, RoBERTa, and ERNIE 2.0.

## introduction a rapid increase in the use of social media has been seen in the past decade. social media platforms such as twitter, reddit, and facebook have become a place for people to articulate their views and emotions. twitter especially has become a medium for people to share their medical lifestyle and the health-related problems that they are facing. thus making twitter an essential resource for extracting meaningful data that can help better understand and improve health services. the advancement of natural language processing (nlp) in deep neural models and its ability to effectively process and understand data has attracted the attention of the healthcare research community. these healthcare researchers have developed a keen interest in processing this available data efficiently using deep learning. the social media mining for health applications (smm4h) @xcite aims to bring researchers worldwide for the mining, representation, and analysis of data related to health, such as updates regarding covid-19 and its vaccination status, drugs, and medical treatments that can help gain medical insights. this year smm4h has proposed ten tasks that involve data classifi-cation, extraction, and named entity recognition. our team has participated in various classification tasks, namely, task 1a, 2a, 2b, 3, 5, 6, 7, and 9. task 1 focuses on better understanding adverse drug reactions (adrs). the aim of task 1a was to distinguish tweets mentioning adverse drug effects (ade) from other tweets (noade). task 2a focused on determining an author's stance toward various issues related to covid-19. the training data were annotated for perspective according to three categories: favor (positive stance), against (negative stance), and neither (neutral stance). task 2b focused on identifying whether a tweet contains a premise (a statement that can be used as an argument in a discussion), where "1" indicates that the tweet has a premise (argument), and "0" means that the tweet doesn't contain a premise. task 3 focused on designing a binary classifier to detect twitter users who self-declare that they are changing their treatment medications despite being advised by a health care professional to follow the prescription. task 5 deals with identifying personal mentions of covid-19 symptoms that have been tweeted in spanish. the dataset needed to be classified into non-personal_reports for non-personal reports, lit-new/s_mentions for news and literature mentions, and self_reports for self-reports. task 6 focuses on distinguishing the self-reported covid-19 vaccination status, which is labeled as "self_reports," and users discussing vaccination status in general, labeled as "vaccine_chatter". task 7 @xcite deals with identifying victims of intimate partner violence (ipv) who seek help on social media like twitter. the label "1" indicates a self-reported ipv, and "0" shows an average tweet about domestic violence. task 9 @xcite focuses on the detection of demographic information on social media and distinguishing the reddit posts that self-report the exact age of the social media user at the time of posting (annotated as "1") from those that do not (annotated as "0"). we propose two transformer-based @xcite approaches for the classification of all the aforementioned tasks. the first approach is fine-tuning existing transformer models. the second approach uses a voting-ensemble model that comprises fine-tuned bert @xcite , roberta @xcite , and ernie 2.0 @xcite . the xnli @xcite model (zero-shot) was used to address the data imbalance in task 1a. the multilingualism of the spanish dataset in task 5 was handled using xlm-roberta model @xcite . ## methodology this section describes a detailed explanation of the approaches we have used for handling all the classification tasks. this paper proposes two architectures, all of which follow a common data preprocessing (section 2.1). ## error analysis this section describes the qualitative analysis of the labels predicted by the proposed architectures, as seen in task text original label predicted label task 1 this vimpat shit is working but the side effects are hell ade noade i started shaking so i had to eat something :( but now i just had some tylenol pm so hopefully i'll go straight to sleep noade ade task 2a (stay at home) support our p.m. and promise to follow lockdown. i request govt to kindly postponed bank emj till lockdown. favor none (stay at home) stay at home, relax back on your couch and find your dream home. click here to explore your options. none against task 2b another great perk of wearing a mask is that you can curse at people under your breath in public and they can't read your lips!! 0 1 my daughter is immune compromised due to a rare genetic disorder, trump can sacrifice his and the republicans children, but not my baby girl! 1 0 task 3 just get cortisone shot neck cool 1 0 lisainlouky doc gave ok take one dose excederin migraine today i took it slept woke headache 0 1 task 5 vengo desde hace días con un dolor que puedo de la espalda alcanzó el coronavirus pero si la escoliosis non-personal_reports self_reports el coronavirus empieza con diarrea quien sepa pueda decir algo tengo un familiar con diarrea perdida del olfato gusto fiebre malestar en los huesos non-personal_reports lit-news_mentions task 6 the vaccineinduced reduction anxiety starting creep out good thing i good reason feel anxious cortisol levels spike self_reports vaccine_chatter just got injection mean injected vaccinecovidvaccine vaccine_chatter self_reports task 7 teamgivingcom my husband helped friend currently domestically violentabusive relationship we helping get situation 0 1 johnpavlovitz deciding testify ex trial domestic violence child abuse he got 35 years crimes best decision ever 1 0 task 9 dmek can give you 20/20 but not every time. 0 1 i wonder it i had it 15yrs ago just dryeye and now im in 40s so it went full blown cuza lak ot estrogen! ahhh ughh. 1 0 a similar trend can be seen in task 3, 6, and 7 where the model has focused more on individual words like "dose","i","injected","violence", and "help" rather than understanding the true semantics of the sentence. the model performed relatively well in task 9. however it failed to understand complex semantic like "now im in 40s" that indirectly indicates the age of the user. the model also puts special focus on numbers since those are commonly used for the indicating age and thus in the fifteenth instance it has made an incorrect prediction. our models did not perform well in tasks that comprised of medical discussions because they were not fully able to understand the underlying medical context. in case of task 3, where bio bert and bio_med models were used the predictions were yet not very accurate perhaps due to the fact that these models were trained before the covid-19 period and were thus not able to fully understand the terms that were used. ## conclusion for the smm4h tasks we propose an ensemble model that leverages on pretrained representations from bert, roberta, ernie 2.0, and a single fine tuned language model as our submission systems. our system was able to report 91.8% f1 score in task 9 and 55.7% in task 3. for other tasks our results were comparable to the arithmetic median released for all the tasks with f1 reaching 83% in task 6 and 68% in task 5. for task 2a, 2b the scores achieved were 50.1% and 62.13% respectively.