SMM 4 H 2022 Task 2: Dataset for stance and premise detection in tweets about health mandates related to COVID -19 (2022)

Abstract: This paper is an organizers’ report of the competition on argument mining systems dealing with English tweets about COVID-19 health mandates. This competition was held within the framework of the SMM4H 2022 shared tasks. During the competition, the participants were offered two subtasks: stance detection and premise classification. We present a manually annotated corpus containing 6,156 short posts from Twitter on three topics related to the COVID-19 pandemic: school closures, stay-at-home orders, and wearing masks. We hope the prepared dataset will support further research on argument mining in the health field.

## introduction nowadays, people are actively sharing their views on various issues related to the covid-19 pandemic on social media. for example, users express their attitude towards a quarantine and wearing masks in public places. some statements are reasoned by arguments, and other statements are just emotional claims. automated approaches for detecting people's stances and their premises towards health orders related to covid-19 can help to estimate the level of cooperation within the health mandates announced by the government. thereby, since the beginning of the pandemic, new datasets for argument mining in the health field have been created. the first and the largest dataset of twitter users' stances in the context of the covid-19 pandemic is covid-cq @xcite . it contains controversial tweets about the efficacy of hydroxychloroquine as a treatment. similarly, @xcite presented a dataset for biomedical claim detection in twitter posts. @xcite created the lockdown-tweets -mostly unlabelled tweet dataset, which is related to the lockdown policy in new york state during the pandemic. people's opinions towards health mandates in germany are discovered in @xcite : first, relevant tweets were identified and then the expressed stances were detected. while most researchers concentrate on the stance detection task, there are far fewer datasets for premise classification @xcite . in this work, we aim to fill in this gap and focus not only on stance detection, but also on premise classification. therefore, we organised a competition on detecting both of these argument structures from english tweets related to covid-19 health mandates. this competition was carried out as one of the shared tasks during the social media mining for health applications (#smm4h) 2022 workshop. the smm4h shared tasks aim to take a community-driven approach to address nlp challenges of utilising social media data for health informatics, including informal, colloquial expressions of clinical concepts, noise, data sparsity, ambiguity, and multilingual posts @xcite , @xcite . in 2022, the seventh iteration of the smm4h shared tasks, including task 2 on automatic classification of stance and premise in tweets about health mandates related to covid-19 (in english), was held @xcite . the dataset for this task contains tweets that express views towards three claims/topics: (i) keeping schools closed, (ii) stay-at-home orders, and (iii) wearing a face mask. the main task consists of two sub-tasks: (i) task 2a on stance detection, and (ii) task 2b on premise classification. task 2a: stance detection the first task aims to determine the point of view (stance) of the text's author concerning the given claim (e.g., wearing a face mask). the tweets are manually annotated for stance according to three categories: in-favor, against, and neither. ## datasets we provided the participants of the smm4h 2022 competition the training and validation sets. during the evaluation phase, all participants had five days to submit their predictions for test sets on codalab for evaluation. the training set included 3,556 tweets, a good mix of three topics: 37%, 33% and 30% of tweets are about face masks, school closures and stay-at-home orders, respectively. the validation and test sets contained 600 and 2,000 tweets, respectively. as a source of tweets for training and validation sets, we leveraged a covid-19 stance detection dataset @xcite along with stance annotation guidelines and stance labels. tweets for test sets were collected using 33 keywords such as #openschools, #lockdownnow, #nomasks. after this, we removed the hashtags to exclude annotation bias toward specific classes. the test set for task 2a and all three sets for task 2b were manually annotated. each tweet was labelled by three yandex.toloka foot_2 annotators. we followed annotation guidelines of an argument mining shared task ruarg-2022 @xcite • a statement is evaluated as an argument if it contains a statement that can be used in a dispute to persuade an opponent. for example, masks help prevent the spread of the disease. to measure annotation quality on this platform, we mixed raw tweets with control tasks (tweets ## experiments we used f 1 as the main evaluation metric in each of the two subtasks, which is calculated according to the following formula: f 1 = 1 n p c2c f 1 relc , where @xmath0, "stay at home orders", "school closures"}, n is the size of c, f 1 relc is macro f 1 -score averaged over two classes for each task (in-favor & against classes for stance; 0 & 1 classes for premise). we used two models as baselines: random and roberta. random baseline is rather simplistic: we randomly assigned labels for each tweet in both tasks according to the label distribution. the second baseline leverages roberta architecture @xcite for multiclass (task 2a) and binary classification (task 2b). we fine-tuned pretrained roberta-base model from huggingface foot_3 on our data. the validation set was utilised to select appropriate hyperparameters for the models. for each model, adamw optimizer @xcite was used with a learning rate of 4e-5, and gradient clipping with a max norm of 1.0. each model was trained for 5 epochs, with a mini-batch size of 8 in each iteration and a maximum sequence length of 128. the results of both baseline models in terms of f 1 scores are described in ## conclusion in this paper, we have presented the dataset for stance and premise detection in tweets written in english about health mandates related to covid-19. we hosted a shared task on smm4h 2022 workshop and released this dataset to the research community. the 15 teams that took part in the task proposed a variety of classification architectures, ranging from just fine-tuning bert models to multi-task learning on both subtasks and combining tweets with claims. we plan to extend our dataset for future work with a broader set of healthrelated claims.