AIR - JPMC @ SMM 4 H ’22: Identifying Self-Reported S panish COVID -19 Symptom Tweets Through Multiple-Model Ensembling (2022)

Abstract: We present our response to Task 5 of the Social Media Mining for Health Applications (SMM4H) 2022 competition. We share our approach into classifying whether a tweet in Spanish about COVID-19 symptoms pertain to themselves, others, or not at all. Using a combination of BERT based models, we were able to achieve results that were higher than the median result of the competition.

## introduction the social media mining for health applications (smm4h) 2022 shared tasks competition @xcite aim to encourage the use of natural language processing for health research. this paper will describe our group's response to task 5, which deals with the classification of spanish-language tweets containing self-reported covid-19 symptoms. the motivation for this task is the need to further develop the volume of natural language processing research on languages other than english. as @xcite point out, only a small number of the world's 7000 languages are represented in natural language processing technologies and related conferences. task 5 identifies a need to increase the amount of nlp and social media research with regards to covid-19 in all languages other than english. task 5 is of a three-way classification problem where a spanish-language tweet has to be identified as one of three possible classes: self_reports, non_personal_reports, and lit-news_mentions. what follows is a description the data-sets provided, various experiments performed, and our final submission results. ## dataset information we were presented with three data-sets consisting of labeled tweets to be used for training, an unlabeled validation set to check the final performance dataset selfreports nonpersonalreports lit-news-mentions total training 1654 2413 5984 10051 validation 572 859 2146 3577 test ---6850 the motivation for the 80%-20% split was to feed as much data as possible when training while allowing us to measure the performance of our models. however, since the distribution of the training set skews towards more lit-news_mentions tweets, the 50%-50% split approach is used to provide a better performance metric. ## results the results of our ensemble predictions for the test data compared with the baseline and median results are shown in task 4. the two submissions have a different f1-scores when experimenting compared to the codalab submission because when experimenting, the f1-score with respect to the self_report label was used while the microaveraged f1 score was used in the codalab submission, which is akin to accuracy with a multiclass classification. the 50%-50% split ensemble performed better than the 80%-20% split ensemble with regards to the micro-average f1-score and vice-versa when considering the f1-score for the positive label only. we hypothesize this is because the 80%-20% split ensemble was optimized to predict tweets belonging to the positive class. this led to an overall lowering of its accuracy, which is shown in its micro-averaged metrics. however, both the 80%-20% split and the 50%-50% performed similar to the median score. ## conclusion as demonstrated in our results section, the best performance was achieved through a majority-vote ensemble of all models. possible opportunities for improvement would be exploring the usage of data augmentation techniques such as back-translation to address the class imbalance present in the training data-set. moreover, we could explore different pre-processing techniques to remove possible noise from our data such as user mentions, hyperlinks or hashtags.