mattica@ SMM 4 H ’22: Leveraging sentiment for stance & premise joint learning (2022)

Abstract: This paper describes our submissions to the Social Media Mining for Health Applications (SMM4H) shared task 2022. Our team (mattica) participated in detecting stances and premises in tweets about health mandates related to COVID-19 (Task 2). Our approach was based on using an in-domain Pretrained Language Model, which we fine-tuned by combining different strategies such as leveraging an additional stance detection dataset through two-stage fine-tuning, joint-learning Stance and Premise detection objectives; and ensembling the sentiment-polarity given by an off-the-shelf fine-tuned model.

## introduction the social media mining for health applications (smm4h) shared task 2022 @xcite is aimed to apply natural language processing to address different challenges of using social media for health research. specifically, we participated in task-2, which consisted of detecting the stance of a tweet towards a given topic (subtask 2a), and detecting if a tweet contained or not a premise (subtask 2b). the organizers provided a manually labeled dataset @xcite split into training (3,669 tweets) and validation (600 tweets). each record in the dataset was labeled on both axis; the stance classification included the labels favor, against, and none; and the premise axis with values 1 and 0 indicating the presence or absence of a premise respectively. for the evaluation phase, an unlabeled test set of 2,000 tweets was provided. our approach consisted of leveraging an indomain pretrained language model (plm) and adapting it to the tasks through two consecutive fine-tuning steps. in the second, we used the predictions of an off-the-shelf model for sentiment polarity as additional features and applied joint learning of the stance and premise detection. related work @xcite @xcite use sentiment analysis as an auxiliary task in an mtl setting. whereas @xcite , like us, use the sentiment as an additional input feature for the stance classification. to our knowledge, no other approaches jointly-learned stance and premise detection leveraging sentiment polarity. foot_0 2 extra data and pre-processing we used covidlies @xcite ) as an extra training dataset. the dataset consists of 6,761 foot_1 covid-19-related tweets, each paired with a misconception and annotated with the concerning tweet's stance. in covidlies, the misconceptions are misinformation statements that were manually rephrased as short and positive expressions (e.g., "coronavirus is caused by 5g"). thus, different from a topic stance, in this dataset, the annotated standpoint is with respect to a specific statement. to combine both datasets, we migrated the provided data from topic-stance to the covidlies statement-stance format by manually formulating a positive statement from each related topic (see ## system description the base of our approach is a transformer-based language model pretrained on a corpus of twitter messages on the topic of covid-19 (ct-bert v2) @xcite . we leveraged the covidlies dataset by an initial fine-tuning (ft) for stance-detection. next, we performed a second fine-tuning on the data provided for this task. the second fine-tuning included a multitask-learning (mtl) setup where the objectives were stance and premise classification. both losses were computed by cross entropy and combined using homoscedastic uncertainty to weight each task loss @xcite . also, in the second fine-tuning, the resulting logits of sentiment classification were appended directly to the pooled output of the encoder just before passing it to the classification heads foot_3 . the sentiment classification was obtained with an already finetuned roberta-base model for sentiment-analysis in tweets @xcite . we trained the models with adamw (loshchilov and hutter, 2019) optimizer without a warm-up period. a weighted random sampling with replacement was used in both fine-tuning steps (except in the multi-task setups), with learning rates of α = 10 -4 and α = 10 - foot_4 respectively. for single task configurations, we used a training batch size of @xmath0; in both cases, the maximum sequence length was set to 160. an early stop with patience of 3 using losses scores of the held-out validation split (@xmath1. 5 ## results and discussion we used the covidlies dataset for the first finetuning. for the second ft we used the provided data as follows: we created a validation split with 5% of the training and 12% of the validation data, and used the rest of the training data as a training split and the rest of the validation data as a test split. all our submissions were based on the two-stage fine-tuning (2st-ft) and what differed was the strategies combination in the 2nd ft. for subtask-2a, we submitted a run with a base 2st-ft, another including the logits from sentiment classification (sent), another with the 2st-ft and mtl, and finally, one that combined all the strategies (2st-ft + sent + mtl). for subtask-2b, we used the two setups that included mtl. this is, our two submissions corresponded to the premise inference from the (2st-ft + mtl) and the (2st-ft + sent + mtl) runs. evaluating our test-set, we observed that the cumulative combination of the different strategies resulted in small but consistent gains for the stance detection task performance (see ## conclusions our approach involved leveraging a related dataset by a preliminary fine-tuning, and combining sentiment analysis along with multi-task learning of premise and stance. in the official test set, our best result for stance detection (subtask 2a) was 0.633, which is 14 percentage points (p.p.) above the mean and 8 p.p. above the median of all participants' submissions. for the premise detection (subtask 2b), our best score was 0.647, which is precisely the median but 7 p.p. above the mean of all submissions. the results show that jointly learning to detect premise and stance is beneficial for both tasks. combined with the tweet's sentiment polarity, the two-stage fine-tuned model gave the best results.