Yet@ SMM 4 H ’22: Improved BERT -based classification models with Rdrop and P oly L oss (2022)

Abstract: This paper describes our approach for 11 classification tasks (Task1a, Task2a, Task2b, Task3a, Task3b, Task4, Task5, Task6, Task7, Task8 and Task9) from Social Media Mining for Health (SMM4H) 2022 Shared Tasks. We developed a classification model that incorporated Rdrop to augment data and avoid overfitting, Poly Loss and Focal Loss to alleviate sample imbalance, and pseudo labels to improve model performance. The results of our submissions are over or equal to the median scores in almost all tasks. In addition, our model achieved the highest score in Task4, with a higher 7.8% and 5.3% F1-score than the median scores in Task2b and Task3a respectively.

## introduction nowadays, people are more and more willing to share their life status on social networks, and analyze and discuss their illness and medication. these texts can be useful for analyzing an individual's health, but they are rare among the vast number of social tweets and particularly difficult to collect. the social media mining for health applications (smm4h) 2022 workshop aims to bring together researchers interested in automatic methods for the collection, extraction, representation, analysis, and validation of social media data (e.g., twitter, facebook) for health informatics @xcite . there are total 14 tasks in smm4h 2022 shared tasks, and we participated in all classification tasks, including task1a: classification of adverse drug events (ades) in english tweets @xcite ; task2a & task2b: classification of stance and premise in tweets about health mandates (covid-19) @xcite ; task3a & task3b: classification of changes in medication treatments in tweets and webmd reviews; task4: classification of tweets self-reporting exact age (in english) @xcite ; task5: classification of tweets of selfreported covid-19 symptoms in spanish; task6: classification of tweets indicating self-reported covid-19 vaccination status; task7: classification of self-reported intimate partner violence on twitter @xcite ; task8: classification of self-reported chronic stress on twitter @xcite ; task9: classification of social media forum posts self-reporting exact age. to solve the above classification challenges, we developed an improved bert-based classification model. it incorporates rdrop for data augmentation, poly loss and focal loss for balancing label distributions, and pseudo label for boosting the model performance. our model scored above the median score in all tasks, and finally, our model achieved the highest score in task4, with a higher 7.8% and 5.3% f1-score than the median scores in task2b and task3a respectively. ## data analysis of all the 11 tasks we participated, only task2a and task5 are three-way classification tasks, the others are all binary-classification tasks, and all tasks use f1-score of certain categories as the evaluation metric. each of the eleven classification tasks has a different classification objective and the data takes different forms. for example, task2a is a sentence pair matching task, while task3b contains several meta-information items such as 'effectiveness', 'satisfaction' and so on. however, all of these tasks can be transformed into either a single text classification or a sentence pair matching task, both of which can be trained and tested with the same model structure. therefore, we focus more on designing generic models to solve as many problems as possible. after analysing the training and test data, we found that the distribution of labels for each task was consistent between the training and valid sets, but the ratio between the categories within the task ## method the whole procedure of our system model follows the bert-style training and inference frame. the difference between the different models lies in the process of data pre-processing, encoders and tricks. ## experiments and analysis all the model we used shared a fixed training config. they were all trained for 5 epochs with learning rate 4e -5. besides, the max length of the input was 128, the weight decay rate was 0.01 and the adam parameter was 1e-8. results show that the introduction of the tricks help the model get the highest performance in task2a, task8 and task9. while in task2b, biobert achieved the best performance, and the tricks reduced the f 1 by 0.17. the reason for this decline is likely to be the way we processed the data. the purpose of task2b is to predict whether at least one premise/argument is mentioned in the text, but we transformed this into a text pair matching problem where the claim part may have been introduced as noise, and the tricks increased the noise. besides, without tricks, deberta performed the best in task3b, but with tricks, the scores dropped by 1.09 to 87.58. it may imply that there is a problem with the way we changed meta-information into text information and concatenate it. except for task4, the label distribution of other tasks in however, in task4, these tricks did not seem to improve the performance of the model, but decreased the score a lot, which may be caused by the relatively balanced label distribution of the task. so we took the pseudo label trick to augment the training data. specifically, we first used the three bestperforming models to predict the samples in the validation set (here is bertweet, deberta and biobert, and we use the test set in the testing phase), and then processed the obtained three log-its in the following way: if the two logits are greater than 0.8, it means that the sample is likely to belong to a certain category, and we would add the sample and the predicted label to the training set as a new training sample. based on the augmented training dataset, we retrained bertweet encoder with rdrop and poly loss, and got the highest score, which helped us win the first place in this task. we submitted the highest scoring models incorporating tricks at each task, and the final scores are shown in the ## conclusion in this work, we developed a classification model based on the bert. it incorporated with rdrop to do the data augmentation, with poly loss to mitigate the label imbalance, and with pseudo label to boost the model performance. we participated 11 classification tasks in smm4h 2022 shared tasks, and our model scored above the median score in all tasks. finally, our model achieved the highest score in task4, with a higher 7.8% and 5.3% f1score than the median scores in task2b and task3a respectively.