Fraunhofer FKIE @ SMM 4 H 2022: System Description for Shared Tasks 2, 4 and 9 (2022)

Abstract: We present our results for the shared tasks 2, 4 and 9 at the SMM4H Workshop at COLING 2022 achieved by succesfully fine-tuning pre-trained language models to the downstream tasks. We identify the occurence of code-switching in the test data for task 2 as a possible source of considerable performance degradation on the test set scores. We successfully exploit structural linguistic similarities in the datasets of tasks 4 and 9 for training on joined datasets, scoring first in task 9 and on par with SOTA in task 4.

## introduction this contribution describes the system submissions for thee shared tasks at the social media mining for health (#smm4h) workshop at coling 2022 @xcite . we participated in tasks 2, 4 and 9. all models were developed using the flair framework @xcite and we used the pre-trained models based on pytorch @xcite provided by huggingface @xcite . based on previous experience and the baseline results provided for the stance detection covid-19 tweets in @xcite , we focused mainly on the models bert @xcite , roberta @xcite , and bertweet @xcite . ## task 2 we participated in both subtasks of shared task 2. in task 2a, stance detection, participants were asked to determine an author's stance in relation to three different mandates related to the covid-19 pandemic @xcite . the following tweets are examples relating to the mandate school closures: in task 2b, premise classification, the aim is to determine whether or not a tweet contains an argument that could be used to convince an opponent about one of the given covid-19 mandates. it is a binary classification task in which the data is annotated as positive (contains a premise) or negative (does not contain a premise). the following two tweets are examples from the mask wearing mandate: the training data consists of 3,556 tweets, the validation data of 600 tweets, and the test data of 10,000 tweets. ## task 4 task 4 is a binary classification task in which participants were asked to determine whether a tweet contains a self-report of an exact age or not. for example, in tweet 1 below, the person posting the tweet reports that they are currently 29. in tweet 0, an age is reported, but it is annotated as negative because the age that is reported is that of the user's dog, and not their own age. the data consists of 8,800 tweets as training data, 2,200 tweets as validation data, and 10,000 tweets as test data. ## task 9 task 9 is very similar to task 4, in that it also involves a binary classification task on self-reported ages. the main difference is that the data stems from reddit, not twitter, and that the data is disease-specific and was collected using specific keywords related to dry eye disease. for example, in (1) a specific age is provided, and the second post in annotated as (0) because the user only provides an age range. the training data consists of 9,000 posts, the validation data of 1,000 posts, and the test data of 2,000 posts. ## conclusion we applied state-of-the art transformer language models to three different shared tasks, succesfully employing dataset fusion to broaden the training base for two closely related tasks. we observed on par results between models in the vicinity of human performance on the gold annotation. the resulting model for task 4 achieved f1 scores of 0.912 (p 0.924, r 0.901) and 0.917 (p 0.891, r 0.904) on the blind test set, outperforming both mean and median f1 scores (0.847, 0.869) of all submissions in the task by a significant margin. these results, despite generalization loss, are also on par with the results of the datataset authors' benchmark f1 on the validation set of 0.914 @xcite . our best model for task 9 based on the same strategy scored 0.956 f1 on the blind test set (@xmath0.948, @xmath1.963), making it the best performing contribution in the competition.