Argumentative Stance Prediction: An Exploratory Study on Multimodality and Few-Shot Learning (2023)

Abstract: To advance argumentative stance prediction as a multimodal problem, the First Shared Task in Multimodal Argument Mining hosted stance prediction in crucial social topics of gun control and abortion. Our exploratory study attempts to evaluate the necessity of images for stance prediction in tweets and compare out-of-the-box text-based large-language models (LLM) in few-shot settings against fine-tuned unimodal and multimodal models. Our work suggests an ensemble of fine-tuned text-based language models (0.817 F1-score) outperforms both the multimodal (0.677 F1-score) and text-based few-shot prediction using a recent state-of-the-art LLM (0.550 F1-score). In addition to the differences in performance, our findings suggest that the multimodal models tend to perform better when image content is summarized as natural language over their native pixel structure and, using in-context examples improves few-shot learning of LLMs performance.

## introduction argumentative stance studies related to ideological topics offer valuable insights into complex dynamics of opinion, belief and discourse in various domains. these insights have far-reaching implications, extending their influence over areas including public opinion, social dynamics, and policy efficacy. by predicting the stance in real-time, policymakers and stakeholders can get immediate feedback on public reaction to new proposals or laws, allowing them to make timely and informed decisions. argumentative stance prediction is becoming a major endeavor in multiple research fields as the reliance on sentiment detection may be sub-optimal @xcite . while the stance prediction task appears similar to sentiment analysis, it has many theoretical differences. sentimental * equal contribution analysis primarily focuses on emotions, whereas the stance prediction need not necessarily coincide with the sentiment directed towards the target. stance prediction for sensitive and polarizing topics can be more challenging, particularly within the brief context of informal social media text @xcite . previous studies have primarily concentrated on examining stance prediction in textual modalities @xcite . however, an increasing number of recent works are widening the focus to include other modalities, such as images. since multimodality helps us understand language from the modalities of text, vision and acoustic, @xcite , the application of multimodal inputs in argumentative stance prediction seems promising. towards the perpetuation of multimodality in argumentative stance prediction as a part of the im-garg 2023 @xcite ) challenge, we explore the following questions using a dataset of tweets on gun control and abortion topics: our work shows that an ensemble of fine-tuned language models performs the best for argumentative stance prediction from tweets. incorporating image information into text using state-of-the-art multimodal models does not outperform the ensemble model. llms (particularly, llama-2) in few-shot setting exhibit high recall but suffer from low precision. though using in-context examples in few-shot setting improves performance, they underperform the ensemble model. ## related work existing work has explored the interplay between stance and sentiment to enhance stance detection. @xcite investigated the relationship between stance and sentiment, utilizing svm with ngram, word embedding, and sentiment lexicon features. they concluded that while sentiment features offer utility, they are insufficient on their own for effective stance detection. meanwhile, @xcite showcased the prowess of bidirectional transformers in achieving competitive performance without fine-tuning, harnessing sentiment and emotion lexicons. their findings show the efficacy of sentiment information, as opposed to emotion, in discerning the stance. @xcite conducted an extensive analysis of 96 primary studies spanning eight machine learning techniques for stance detection and its applications. the analysis suggests that deep learning models with self-attention mechanisms were found to be frequently outperforming the traditional machine learning models such as svm, and emerging techniques like few-shot learning and multitask learning were increasingly applied for stance detection. multimodal stance detection is being increasingly used for social applications such as rumor verification @xcite and identifying public attitudes towards climate change on twitter @xcite . despite recent advancements in multimodal language models @xcite , the use of image modality for stance detection remains an underexplored area. our work conducts an exploratory study to investigate the necessity of multimodal models for stance detection and compares different ways to incorporate image information into text modality. ## dataset and task the imgarg dataset @xcite is a part of the multimodal argument mining @xcite competition. curated with the goal of expanding argumentation mining into multimodal realm, the dataset consists of twitter texts along with their images from two topics-gun control and abortion. each text-image pair corresponding to a tweet are annotated with a stance (support or oppose) along with its persuasiveness (no persuasiveness to ex-tremely persuasive). in this paper, we focus on the stance prediction task. briefly, the task can be described as given an image-text pair corresponding to a tweet, predict if it supports or opposes the topic. it is important to note that while the gun control dataset is balanced, the abortion dataset is imbalanced by a 1:3 support:oppose stance ratio. the gun control and abortion training sets are 920 foot_0 and 891 tweets respectively. both datasets have an equal number of tweets in the validation (100 tweets) and test (150 tweets) sets. ## approach to predict argumentative stance over multimodal tweets from gun control and abortion topics, we leverage three different ideas. we explore an ensemble of llms against its constituent models, incorporate image information through multimodal models as well as evaluate out-of-the-box llms in few-shot setting. this section describes the experimental approaches used in the process. further details can be found in the appendix. ## discussion popular pre-trained language models such as xl-net, bloom, transformer-xl, deberta-v2 and xlm-roberta were fine-tuned for stance prediction on tweets about gun control and abortion. results demonstrate that the ensemble of these models performs better than any of the constituent models. however, the disparity is limited. xlnet achieves better recall than the ensemble model and similarly, the bloom-560m underperforms the ensemble by 0.026 (though precision of the ensemble is higher by 0.036). this raises the trade-off question between ensemble performance vs. the large computational requirement justified for marginal improvement in the performance. the best performing multimodal model used the image content summarized as text, unlike its counterpart models that operate in pixel space. we believe the diversity of the images contributes to this difference. in addition to typical images containing people and objects such as guns, trucks and so on, the training set also contained propagandarelated material such as posters with statements. while vision-language models are increasingly getting better at object-centric tasks, understanding such material is closely related to problems such as optical character recognition, which are not often explored in pretraining vision-language models. our instruction-based image summarization suggests that when explicitly prompted, visionlanguage models excel not just at object-centric descriptions of images but also at recognizing text from images. attempts were made to incorporate demographic factors such as number of people in the image, their skin color and gender. however, manual inspection revealed that the resultant in-structblip predictions were not reliable. despite augmenting language modality with images in different ways, text-based models outperformed the multimodal models. out-of-the-box llama-2 underperforms the baseline support only prediction model. however, prompting through four-shot examples greatly improves the performance. this is further enhanced by using in-context examples. this demonstrates that in-context examples that potentially share similar theme (not necessarily the stance) tend to capture the stance better than arbitrary examples from the dataset. the themes were found to include discussions along mental health, effects on children, racism, illegal acquisition, etc. for the gun con-trol dataset; supreme court, birth control, religion, reproductive rights, etc. for the abortion dataset. ## conclusions and future work our investigation questions the necessity of images to predict stance in multimodal tweets through different ways of using image-based information in conjunction with text-based language models and investigating the inherent capabilities of llms for stance prediction. results suggest that the best performance can be achieved using an ensemble of language models. our experiments with multimodal models do not completely refute the utility of images for stance prediction, rather they merely evaluate the current state-of-the-art multimodal models. incorporating domain knowledge @xcite , and alternative prompting methods like question decomposition @xcite and tree-of-thought @xcite which provide the rationale for the prediction in addition to the stance provide a future direction to address the limited performance with llama-2. classification head for binary sequence classification tasks. the summary of the size of the classification head for each model is provided in