Multi-modal Stance Detection: New Datasets and Model (2024)

Abstract: Stance detection is a challenging task that aims to identify public opinion from social media platforms with respect to specific targets. Previous work on stance detection largely focused on pure texts. In this paper, we study multi-modal stance detection for tweets consisting of texts and images, which are prevalent in today’s fast-growing social media platforms where people often post multi-modal messages. To this end, we create five new multi-modal stance detection datasets of different domains based on Twitter, in which each example consists of a text and an image. In addition, we propose a simple yet effective Targeted Multi-modal Prompt Tuning framework (TMPT), where target information is leveraged to learn multi-modal stance features from textual and visual modalities. Experimental results on our five benchmark datasets show that the proposed TMPT achieves state-of-the-art performance in multi-modal stance detection.

## introduction stance detection is an important task for learning public opinion from social media platforms, which aims to determine people's opinionated standpoint or attitude (e.g., favor, against, or neutral, etc.) expressed in the content towards a specific target, topic, or proposition @xcite . existing conventional machine learning-based methods @xcite @xcite and deep learning-based methods @xcite @xcite @xcite have made promising progress in different types of stance detection tasks for pure texts. figure 1 : an example of a user expressing an "against" stance towards "donald trump" and a "favor" stance towards "joe biden" using multi-modal information. however, more and more present-day social media platforms like twitter allow people to post multi-modal messages, which encourages people to express their stances and opinions through multimodal content, posting texts with images for example. that is, detecting stance from the pure text modality may not accurately identify the user's real view of a target. for example, figure 1 shows a post composed of a text and an image. the stance expression towards "donald trump" and "joe biden" in this example can not be accurately identified based on text information unless combined with the information of the visual modality. therefore, how to detect users' stances on a topic from multi-modal posts might help better identify public opinion in social media. for multi-modal stance identification, weinzierl and harabagiu (2023) discussed the multi-modal stance towards frames of communication. different from conventional stance detection tasks that concentrate on the stance towards several predefined targets, their focus was centered on the frames of communication within multi-modal posts. therefore, aiming to push forward the research of multi-modal stance detection, we create five new datasets, in which each example consists of a target, a text, and an image. these datasets contain a total of 17,544 examples across 5 domains and 12 targets, including hot topics, politicians, and debates. to deal with multi-modal stance detection, we propose a simple yet effective targeted multimodal prompt tuning framework (tmpt), where the targeted prompt tuning is employed to adapt pretrained models for learning stance features from different modalities. specifically, to leverage the target information in stance detection, we first devise targeted prompts for both textual and visual modalities. then, the targeted prompts are fed to the pre-trained language model and pre-trained visual model to learn stance features for the target from different modalities. further, a simple vector concatenation is used to fuse the features from different modalities for multi-modal stance detection. the main contributions of our work are summarized as follows: ## related work textual stance detection various methods based on conventional machine learning @xcite and deep learning @xcite @xcite have been proposed to deal with the stance detection regarding a specific target. for performing stance detection in real-world scenarios, many existing methods focus on the task of zero-shot stance detection foot_1 @xcite @xcite . prompt tuning prompting @xcite was originally aimed to design language instructions for pre-trained language models (plms) to transfer learning in downstream tasks @xcite . recent works begin to treat prompts as continuous vectors and optimize them during fine-tuning, called prompt tuning @xcite @xcite . besides, @xcite ; @xcite ; @xcite ; @xcite introduce prompt into vision-language models to leverage the ability of prompt tuning in multi-modal tasks. ## multi-modal stance detection datasets based on three open-source textual stance detection datasets: twitter stance election 2020 (kawintiranon and singh, 2021), covid-cq (mutlu et al., 2020), and will-they-won't-they @xcite , and two hot topics in recent years: russo-ukrainian conflict foot_2 and taiwan question foot_3 . we create five multi-modal stance detection datasets of different domains to provide available data for this task: multi-modal twitter stance election 2020 (mtse), multi-modal covid-cq (mccq), multi-modal will-they-won't-they (mwtwt), multi-modal russo-ukrainian conflict (mruc) and multi-modal taiwan question (mtwq). ## conclusion in this paper, we present mtse, mccq, mwtwt, mruc and mtwq, five new datasets for multimodal stance detection. based on the created datasets, we present in-target multi-modal stance detection and zero-shot multi-modal stance detection, aiming to advance and facilitate research in the field of multi-modal stance detection. in addition, we propose a simple yet effective targeted multi-modal prompting framework (tmpt) to deal with multi-modal stance detection, where the target information is explored to prompt the pre-trained models in learning multi-modal stance features. extensive experiments on the new datasets show that our proposed tmpt achieves overall better performance than state-of-the-art baseline methods.