Enolp musk@ SMM 4 H ’22 : Leveraging Pre-trained Language Models for Stance And Premise Classification (2022)

Abstract: This paper covers our approaches for the Social Media Mining for Health (SMM4H) Shared Tasks 2a and 2b. Apart from the baseline architectures, we experiment with Parts of Speech (PoS), dependency parsing, and Tf-Idf features. Additionally, we perform contrastive pretraining on our best models using a supervised contrastive loss function. In both the tasks, we outperformed the mean and median scores and ranked first on the validation set. For stance classification, we achieved an F1-score of 0.636 using the CovidTwitterBERT model, while for premise classification, we achieved an F1-score of 0.664 using BART-base model on test dataset.

## introduction with the growth of social media platforms such as twitter and facebook, people can express their views on any topic. this lets them reach a vast audience, as these platforms have millions of users, and helps them get recognition for their issues online. in the case of trending events on social media, many people post their opinions around the same time, thus rapidly shaping the public opinion on those events. misinformation may be spread unknowingly or knowingly by malicious parties, creating the wrong public opinion among social media users. generally, when governments introduce new schemes or policies, they need to know the people's reviews, suggestions, and complaints. social media sites like twitter are essential in this aspect as they let users post microblog tweets of up to 280 characters and allow them to use hashtags to link the tweet to other tweets of the same topic that use this hashtag. during the covid crisis, many public mandates were imposed regarding the closure of public places, lockdown in many places, and social distancing norms. this unusual change made many people post their views on these topics. to derive the public opinion on these topics, we must know the text's claim i.e. towards what entity is this argument targeted, the user's stance associated with this claim i.e. whether the argument is in favor or against the claim, and the premise associated with this claim i.e whether the text has a convincing argument or not. similar to the work carried out by @xcite , we perform stance classification for health mandate-related tweets by employing pre-trained transformer models and then trying to improve classification results by using additional features as described in the section 3.2. ## task & dataset description task 2 of the shared tasks for social media mining for health @xcite has the motive for stance and premise classification of tweets related to the health mandates -"stay at home orders", "face masks" and "school closures". the two subtasks are: the dataset (davydova and tutubalina, 2022) consists of views of the general public on various issues and government mandates for covid-19 in the form of twitter comments. it is manually labeled for claims, stance, and premise. the claims are of three categories "stay at home orders", "face masks" and "school closures". each of the annotated entries in the dataset consists of stance which is of one of the three classes-"favor", "against" or "neither", and the premise which contains the binary annotations 0 (if the tweet does not have any premise) or 1 (if the tweet contains a premise). the training, validation, and test sets are of size 3556, 600, and 2000 tweets, respectively. ## implementation details section 3.1 describes the baseline architectures used for our experiments, section 3.2 and section 3.3 describe the techniques tried to improve the models using additional features and contrastive learning respectively. we discuss the details of our experimental setting in section 3.4. ## summary of results we note the crucial observations for stance & premise prediction on the development dataset in we observe that, in general the transformer models pre-trained on in-domain datasets perform better than the base models. furthermore, we also notice a jump in performance gained with covidtwit-terbert by pretraining the baseline bert large model with in-domain data. we performed contrastive training only for the best model for each subtask. however, the performance didn't improve with this strategy. passing more contextual information in the form of additional features gives a considerable boost to the performance of baseline models for stance prediction but fails for premise prediction.