UCCNLP @ SMM 4 H ’22:Label distribution aware long-tailed learning with post-hoc posterior calibration applied to text classification (2022)

Abstract: The paper describes our submissions for the Social Media Mining for Health (SMM4H) workshop 2022 shared tasks. We participated in 2 tasks: (1) classification of adverse drug events (ADE) mentions in english tweets (Task-1a) and (2) classification of self-reported intimate partner violence (IPV) on twitter (Task 7). We proposed an approach that uses RoBERTa (A Robustly Optimized BERT Pretraining Approach) fine-tuned with a label distribution-aware margin loss function and post-hoc posterior calibration for robust inference against class imbalance. We achieved a 4% and 1 % increase in performance on IPV and ADE respectively when compared with the traditional fine-tuning strategy with unweighted cross-entropy loss.

## introduction social media platforms are becoming part of our every day life with an estimation of about 4.2 billion people using some sort of social media @xcite . the 7 th social media mining for health applications workshop (smm4h) 2022 organized tasks involving automatic methods for collection, extraction, representation, analysis and validation of social media data for health informatics. the tasks we have participated in utilized data from twitter, which is one of the largest social media platforms with approximately 192 million daily active users @xcite . our team uccnlp participated in 2 tasks: (1) classification of adverse drugs events (ade) mentions in english tweets (task-1a) and (2) classification of self-reported intimate partner violence (ipv) on twitter (task 7). adverse drug events (ades) are negative effects related to the use of drugs. ades mentions on social media are a useful indicator of the efficacy of medications and also can potentially reveal previously unknown side effects of drugs @xcite . intimate partner violence (ipv) on the other hand refers to the abuse or aggression that occurs in a romantic relationship. ipv is a serious health problem and can have lifelong impact on health and well-being of individuals. victims of ipv sometimes share their stories on twitter making it an important tool in early identification for timely intervention and support @xcite . exploration of the provided training datasets of both tasks revealed that the datasets were heavily imbalanced with long tailed distributions for example the ipv data contains only 11% of the provided tweets identified as self-reported ipv. the skewed nature of these datasets complicates efficient training even for state of the art models like bert (bidirectional embeddings from transformers) @xcite and roberta (a robustly optimized bert pretraining approach) @xcite . the difficulty in model training arises from the fact that naive empirical risk minimization for imbalanced datasets tends to be biased towards learning the distribution on these head classes which deteriorates its performance on tail classes. in this work, we proposed to solve the class imbalance in classification on both tasks using roberta with label distribution-aware loss function (ldam) combined and posterior calibration to alleviate the problem of class imbalance during training. ## experiments and results the datasets for both tasks: (ipv and ade) were released by smm4h 2022 shared tasks. ipv self report classification (task 7) consisted of 4523 training tweets, 534 validation tweets and 1291 test tweets @xcite . ade detection task contained 17173 train tweets , 908 validation tweets and 10968 test tweets. we conducted experiments with pre-trained transformer language models; roberta @xcite and @xcite with different loss functions and also with post-hoc posterior calibration (pc). experiments were done for 10 epochs, max length of 128, batch size of 10 and the learning rate was set at 0.0005. the final submission were evaluated using a median f 1-score over 5 runs. the system was written in pytorch using hugging-face transformer library @xcite . ## conclusion in this work, we developed two systems based on pre-trained transformer based language models fine-tuned with label distribution aware loss functions and posterior calibration on the test set. we experimented with various loss functions as well as different transformer models. the results on the validation set revealed that incorporating class prior probabilities at both training and testing helps to boost performance under class imbalance. further more our system achieved 62.25% f 1-score on ipv (task 7) and 8% for ade task (task 1a) on our codalab submission.