GC -Hunter at I mage A rg Shared Task: Multi-Modal Stance and Persuasiveness Learning (2023)

Abstract: With the rising prominence of social media, users frequently supplement their written content with images. This trend has brought about new challenges in automatic processing of social media messages. In order to fully understand the meaning of a post, it is necessary to capture the relationship between the image and the text. In this work we address the two main objectives of the ImageArg shared task. Firstly, we aim to determine the stance of a multi-modal tweet toward a particular issue. We propose a strong baseline, fine-tuning transformer based models on concatenation of tweet text and image text. The second goal is to predict the impact of an image on the persuasiveness of the text in a multi-modal tweet. To capture the persuasiveness of an image, we train vision and language models on the data and explore other sets of features merged with the model, to enhance prediction power. Ultimately, both of these goals contribute toward the broader aim of understanding multi-modal messages on social media and how images and texts relate to each other.

## introduction argumentative stance detection is an important problem within the field of natural language processing (nlp). its primary objective is to discern the underlying position of a text in relation to a specific topic. accurate identification of a text's stance enhances the performance of several other nlp applications, including text summarizing, information retrieval, fact-checking, and broadly contributes to enhanced understanding of the text. in recent years, the landscape of information dissemination has evolved beyond text, and a growing number of online users express themselves on social media using multi-modal messages. this shift underscores the need for more sophisticated approaches in argumentative stance detection . the emergence of pre-training models based on transformer architecture @xcite has introduced new horizons for analyzing and understanding text. all areas of natural language processing have been impacted by transformers and the subsequent models derived from them. although remarkable strides have been made in most uni-modal applications of language processing, researchers are now shifting to multi-modal problems such as vision-language learning. similar to the uni-modal challenges, large high quality labeled datasets are needed to pre-train the multi-modal models and fine-tune them for the downstream task. in this work, we use lightweight vision and language learning models to learn joint representations of image-text pairs to capture patterns that help us predict how an image contributes to persuasiveness of a tweet comprised of an image and text. in addition to multi-modal models, we argue that to capture the stance of a tweet toward a given topic, only processing the text modality acts as a strong baseline for any multi-modal learning models. this is because detecting the stance of the text will provide valuable insights into the overall stance of the tweet itself. the remainder of the paper is organized as follows. we first review previous studies on argument mining and stance detection, as well as vision and language learning in section 2. then we introduce the dataset used in this work in section 3. in section 4, we detail our experiments and results obtained. finally, we conclude in section 5 and summarize the main findings of this work. ## related work numerous works have studied the problem of classifying argumentative stance, focusing on developing robust and accurate models for identifying the stance expressed in text. existing studies have explored a different approaches, such as feature based classification, structure based classification, neural networks and attention based models, and domain specific knowledge and lexicons @xcite @xcite . earlier studies of argument mining mostly focused on learning the argumentative structure of a text document or classifying different argumentation strategies. recently, researchers have begun to study persuasiveness-related tasks related to argument mining. @xcite proposed several features to capture persuasiveness in online forums. they argue that online persuasive texts contain an argument strategy that is not common in other genres. in a similar study, researchers created an annotated dataset comprised of argumentative text pairs on the same topics and performed a thorough analysis of how to quantify each argument's persuasiveness @xcite . despite these efforts to develop methods to identify persuasiveness of arguments in text, studying image persuasiveness is a largely unexplored problem. multi-modal learning involves joint processing of information from two or more modalities. in recent years, multi-modal learning has gained substantial attention in the machine learning community. researchers have explored various architectures to effectively fuse information from different modalities. some successful models use separate embeddings for image and text modalities and then capture the similarities using dot products or attention models @xcite . other models use deeper networks to model the image-text representations @xcite . in this work we build on prior studies to explore models and approaches for multi-modal stance detection. we use lightweight neural models for learning joint embeddings of image-text pairs in the data. we also capture similarity scores with more computationally expensive transformer embedders to gain more information about how both modalities interact with the given topic. ## dataset we use the data provided for the imagearg shared task 2023 @xcite . the data consists of a multi-modal corpus (imagearg) of tweets on two social topics, gun control and abortion. the corpus was collected with the aim of studying the persuasiveness of a post that contains both text and an image, and also the argumentative stance of multi-modal tweets towards the topic. they develop schemes to annotate images based on their stance and persuasiveness. while stance detection is an established discipline with many resources, persuasiveness in a multimodal context is an underexplored problem without existing labeled corpora. to annotate the stance of the tweets, tweets are assumed to hold a consistent stance towards the topic in both modalities. the pipeline to annotate persuasiveness is designed in a way that only the tweets that have a clear stance towards the topic are annotated for how persuasive they are. the corpus is divided into train, validation, and test sets. the dataset details are provided in the train datasets are slightly imbalanced with regard to persuasiveness labels. both datasets have more instances where the image is not making the tweet more persuasive. in terms of supporting or opposing the stance however, gun control dataset is quite balanced but in abortion dataset, "oppose" is the dominant class. ## conclusions in this paper, we presented a strong model for multi-modal stance detection towards a given topic.