SPLIT : Stance and Persuasion Prediction with Multi-modal on Image and Textual Information (2023)

Abstract: Persuasiveness is a prominent personality trait that measures the extent to which a speaker can impact the beliefs, attitudes, intentions, motivations, and actions of their audience. The ImageArg task is a featured challenge at the 10th ArgMining Workshop during EMNLP 2023, focusing on harnessing the potential of the ImageArg dataset to advance techniques in multimodal persuasion. In this study, we investigate the utilization of dual-modality datasets and evaluate three distinct multi-modality models. By enhancing multi-modality datasets, we demonstrate both the advantages and constraints of cutting-edge models.

## introduction persuasion encompasses the art of one party endeavoring to influence another's thoughts, beliefs, or actions, and it stands as a fundamental and versatile human capability. its significance goes far beyond the realms of business and politics, permeating numerous facets of our everyday existence. in the fast-changing realm of natural language processing (nlp) and artificial intelligence (ai), there has been a notable increase in enthusiasm for creating techniques and datasets to enhance and assess persuasiveness in natural language applications @xcite @xcite . the capacity to convince, sway, and captivate using language has long been a fundamental element of human communication, and with the emergence of advanced language technologies, the pursuit of leveraging persuasive capabilities in digital interactions has gained remarkable momentum. in today's digital age, the proliferation of social media platforms has ushered in a new frontier for the practice of persuasion. these platforms serve as fertile ground, affording both organizations and individuals the opportunity to engage in activities that extend beyond mere persuasion and can include disinformation campaigns. the pervasive reach and influence of social media amplify the potential impact of persuasive efforts, making it imperative for individuals and society as a whole to exercise discernment and critical thinking in navigating this dynamic landscape. most of current works in argumentation mining solely focus on textual format, such as the argumentation dialogues @xcite , contextual advertising @xcite , and other works @xcite . in their work, @xcite introduced a comprehensive deep multimodal fusion approach to predict persuasiveness, incorporating three modalities: visual, acoustic, and text. nevertheless, in light of the current trend observed on twitter, as depicted in figure 1 , it becomes evident that numerous images accompanied by text are surfacing. mere application of computer vision (cv) techniques for object recognition proves inadequate for addressing this challenge. @xcite designed two tasks based on the tweets, stance detection and persuasion prediction. stance detection (sd) involves the automated task of ascertaining, based on textual content, whether the author expresses a supportive, opposing, or neutral position regarding a particular proposition or subject. this subject can encompass individuals, organizations, government policies, movements, products, and more. as an illustration, considering the tweet and accompanying image in figure 1 , it is evident that the stance conveyed is one of support. persuasion prediction (pp) determines the degree of persuasiveness or the potential impact that a given tweet may have on its readers or the broader audience. given the unique characteristics of the existing twitter data, this paper will design additional feature extraction methods, such as using optical character recognition (ocr) to extract text from images, in order to enrich the feature space. this will enable a more comprehensive analysis of the stance and persuasion in the current tweets. our code is publicly available in github ( https: //github.com/jzcs2018/act-cs ). in summary, our contributions are as follows: cv and nlp models as split, to utilize the image and textual information for the sd and pp tasks. multi-modal learning thanks to the progress in language models and alignment techniques, multimodal learning with text and image have recently received significant attention in the cv and nlp communities. as the majority of sota works are built upon transformersand and its variants, different alignment strategies have been proposed and applied to fuse representations from each modality. on the one hand, many works ( radford ## approach let d be a tweet dataset, where each tweet d i is represented as a tuple (i i , t i ). i i represents the image associated with the tweet and t i represents the textual content of the tweet. a model f that maps the input tuples (i i , t i ) to a predicted stance score or persuasiveness score ŷi , where ŷi ∈ [0, 1]. in addition, we will also extract the text in the image it i , and generate its caption c i as an optional feature. then the representation of the tweet can be shown as the new tuple (i i , t i , it i /c i ). the framework is illustrated in figure 2 . ## experiments we designed the experiments to answer two key questions: (1) how accurate is split in automating the entity matching? (2) how important are the different components of split? ## conclusion in light of the recent advancements in persuasiveness and stance prediction research, this study combines state-of-the-art computer vision (cv) and natural language processing (nlp) models under the name split, and explores various fusion approaches. the findings indicate that the crossattention mechanism outperforms other methods. in the future, we will focus on how to visualize and interpret the predictions from the model, which could provide more comprehensive analysis to the researchers.