I mage A rg: A Multi-modal Tweet Dataset for Image Persuasiveness Mining (2022)

Abstract: The growing interest in developing corpora of persuasive texts has promoted applications in automated systems, e.g., debating and essay scoring systems; however, there is little prior work mining image persuasiveness from an argumentative perspective. To expand persuasiveness mining into a multi-modal realm, we present a multi-modal dataset, ImageArg, consisting of annotations of image persuasiveness in tweets. The annotations are based on a persuasion taxonomy we developed to explore image functionalities and the means of persuasion. We benchmark image persuasiveness tasks on ImageArg using widely-used multi-modal learning methods. The experimental results show that our dataset offers a useful resource for this rich and challenging topic, and there is ample room for modeling improvement.

## introduction argumentation mining (am) aims to analyze authors' argumentative stance by automatically identifying argumentative structures and their relationships @xcite . as a fundamental component in am, computational persuasiveness analysis has gained considerable momentum due to growing resources and downstream applications @xcite @xcite @xcite . aiming at automatically evaluating how well one party can change another party's opinions or behaviors, computational persuasiveness tasks are critical yet challenging. recent work in am has brought attention to mining persuasiveness in essays. @xcite and @xcite developed the argument annotated essays corpus (aaec) where stance, argument components, and * these authors contributed equally to this work. argumentative relations were annotated. @xcite extended aaec annotations with persuasiveness scores, as well as with argumentative attributes that potentially impact persuasiveness (eloquence, specificity, relevance, and evidence) and the means of persuasion (ethos, pathos, or logos). these are all text-based annotations, however, missing the opportunity to leverage other modalities (e.g., images) that potentially enhance the persuasiveness of the argument. for example, the image showing statistic charts in fig. 1 makes the tweet text more convincing. to address the gap that image persuasivness has rarely been explored in the am community, we create a new multi-modal dataset, imagearg, that annotates image persuasiveness in tweets and extends persuasiveness mining to a multi-modal realm. regarding imagearg construction, we first extend annotation schemes that are previously developed to capture the persuasive strength of text arguments in aaec @xcite @xcite to a new modality of image. specifically, we develop a novel strategy (sec. 3.2) to annotate multi-modal persuasiveness gains that measure if the persuasivness of a tweet's text increases after adding a visual image. second, we devise a taxonomy to annotate image content (sec. 3.3) that explicitly identifies image functionalities from a persuasive perspective. furthermore, we adapt existing text attributes used in @xcite to annotate image persuasion modes (sec. 3.4) by exploring different annotation strategies (sec. 4.2). we evaluate the inter-rater agreement on our proposed annotation schemes as well as the quality of the annotated samples. with imagearg, we first report the basic statistics of the dataset and conduct a thorough analysis between different annotation dimensions (sec. 4.3). we observe a strong correlation between human political ideology (i.e. stance towards a social topic) and the argumentative features in their posted tweets, as well as mutual influences between image content and persuasion mode. in addition, we benchmark model performance on multiple argumentative classification tasks annotated in im-agearg (sec. 5.2). specifically, we employ multimodal learning methods to classify stance, image persuasiveness, image content, and image persuasion mode. our benchmark results highlight the challenge of these tasks and indicate there is ample room for model improvement. we demonstrate the limitation of these general multi-modal methods and discuss possible future work. we further conduct a qualitative study on a real-world application, retrieving the most persuasive images given a tweet text, by using our trained classifiers (sec. 5.3), which offers a starting point for developing an intelligent tool that recommends persuasive images to users based on their textual inputs. our code and data is publicly available at: https://github. com/meiqiguo/argmining2022-imagearg . ## annotation scheme we propose an annotation scheme to capture an image's impact on the persuasiveness of multi-modal 6 ). tweets. we build a corpus of twitter posts on a social topic (e.g., gun control), then annotate the image within each post along four dimensions. the annotation pipeline is shown in fig. 2 . first, we determine (1) the stance of the entire tweet (sec. 3.1). specifically, we assume one tweet holds a consistent stance in its text and image since the author would intend to deliver a consistent argument. for those tweets annotated with a positive or negative stance, we also annotate (2) the persuasiveness scores of the tweet image (sec. 3.2) and (3) the image content type. the content types identify image roles from an argumentative perspective (sec. 3.3). finally, we (4) identify the persuasion mode of an image that is annotated as persuasive. the persuasion mode indicates how the images persuade audiences (sec. 3.4). note that with this annotation pipeline, all tweets will first be annotated for stance. then, only tweets with a clear stance will be annotated for content type and persuasiveness scores. finally, only tweets where the images are persuasive will be annotated for persuasion mode. ## conclusion and future work we create a brand-new multi-modal persuasiveness dataset imagearg that focuses on image functionality and persuasion mode for persuasive arguments. we extend the argumentative annotation scheme from text to vision, and demonstrate its feasibility. we then establish a benchmark on our defined tasks using computational models, with multiple input modalities. our experimental results reveal that image persuasiveness mining is challenging and that there is ample room for model improvement. we identify the image encoder as a key modeling bottleneck through a series of qualitative and quantitative analysis, which offers a good starting point for further exploration on this rich and challenging topic. the first version of imagearg has 1003 annotations on the gun control topic. in the future work, we will work on constructing datasets on the topics of immigration and abortion, and scaling up the annotations. hao tan and mohit bansal. 2019. lxmert: learning cross-modality encoder representations from transformers. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 5103-5114. yao-hung hubert tsai, paul pu liang, amir zadeh, louis-philippe morency, and ruslan salakhutdinov. 2018. learning factorized multimodal representations. in international conference on learning representations. henning wachsmuth, manfred stede, roxanne el baff, khalid al khatib, maria skeppstedt, and benno stein. 2018. argumentation synthesis following rhetorical strategies. in proceedings of the 27th international conference on computational linguistics, pages 3753-3765. zhongyu wei, yang liu, and yi li. 2016. is this post persuasive? ranking argumentative comments in online forum. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 195-200. mingda zhang, rebecca hwa, and adriana kovashka. 2018. equal but not the same: understanding the implicit relationship between persuasive images and text. in proceedings of the british machine vision conference (bmvc).