A General Framework for Multimodal Argument Persuasiveness Classification of Tweets (2023)

Abstract: An important property of argumentation concerns the degree of its persuasiveness, which can be influenced by various modalities. On social media platforms, individuals usually have the option of supporting their textual statements with images. The goals of the ImageArg shared task, held with ArgMining 2023, were therefore (A) to classify tweet stances considering both modalities and (B) to predict the influence of an image on the persuasiveness of a tweet text. In this paper, we present our proposed methodology that shows strong performance on both tasks, placing 3rd team on the leaderboard in each case with F1 scores of 0.8273 (A) and 0.5281 (B). The framework relies on pre-trained models to extract text and image features, which are then fed into a task-specific classification model. Our experiments highlighted that the multimodal vision and language model CLIP holds a specific importance in the extraction of features, in particular for task (A).

## introduction how convincing are the arguments put forward in a discussion? are these arguments effective in persuading a dissenting voice to change its opinion or behavior? automatically answering such questions of argument persuasiveness holds significant importance within the field of argument mining. there has been a growing body of research on tasks pertaining to persuasiveness @xcite @xcite . works like @xcite and @xcite have brought persuasive essays into focus. to capture the persuasiveness of arguments based on @xcite 's idea of logos, ethos and pathos, different annotation schemes have been developed @xcite @xcite . moreover, phenomena of argument persuasion were examined using a variety of data sources, including online debates @xcite @xcite and news editorials @xcite . what these works have in common is their emphasis on argumentation in textual form. however, the options for persuading the counterpart of one's own view are by no means limited to written speech @xcite . there are further means that can be employed, usually as supplements, like images or videos @xcite @xcite . in this paper, we present our solution approach to the imagearg shared task @xcite . we propose using a general framework to solve tasks related to argument persuasiveness in multimodal settings. the framework comprises two feature extraction modules designed for processing text and image modalities, which are subsequently inputted into a classifier. in our experiments, clipextracted features @xcite excelled for subtask (a), and supplementing them with additional features (convnext @xcite , reformer @xcite , electra @xcite , layoutlm @xcite , camem-bert @xcite , swin v2 @xcite ) proved most beneficial for subtask @xcite . we begin with a brief description of task and dataset ( §2), followed by a detailed description of our methodology ( §3). we then present the experimental results ( §4) and analyze the errors that occur ( §5). in addition, we report progress on our approach in the post-evaluation phase, which has enabled us to further improve classification performance ( §6). finally, we draw a conclusion and make recommendations for future work ( §7). ## task description the shared task relies on imagearg @xcite , a multimodal dataset for argument persuasiveness. it consists of english-language argumentative tweets supported by images as provided by users. the version of the dataset used for the shared the n tweet feature vectors then jointly form a matrix @xmath0. along with the n task-specific labels, they serve as input for model 3. ## methodology motivated by @xcite , we developed a versatile framework (illustrated in figure 1 ) that takes tweet texts and images as input, extracts features for both modalities, and feeds the combined features into a classification model. this framework is designed to work readily for both tasks and 1 our statistics differ slightly from the organizer's data due to inconsistencies in the downloading process. ## error analysis & discussion in the following, we analyze the outputs of our best model for as and ip in terms of misclassifications: ## additional experiments as can be seen from the results presented so far, predicting ip in particular presents a challenge. for this reason, we present additional experiments that we conducted as a follow-up to the shared task. in our experimental efforts, we obtained notably positive results when using camembert as text model, particularly for abortion in combination with convnext or swin transformers v2 as image model. given the significant disparity between our dev and test scores for ip in the shared task submissions, we proceeded to conduct additional experiments with various adaptations of this model in order to find more robust models. it turned out, that employing camembert-base foot_5 to extract text features and swin_v2_s foot_6 to extract image features for the abortion topic, while retaining the proven combination of rel and convnext for _v2_s the gun control topic, resulted in promising results. the classifier was logistic regression. with this setup, we managed to attain an f 1 score of 0.5941 for the test set, while the f 1 score for the dev set was 0.5950. as can be seen, the approach significantly increases previous test scores (cf. we performed further experiments, eventually achieving test scores above 0.66. at the same time, however, the dev performance deviated strongly downwards in these cases. despite the very encouraging results, additional investigations are needed in order to ensure reliable performance. ## conclusion & future work on social media, users have the freedom to use informal, formal, or mixed styles of language, and to incorporate elements such as hashtags, mentions, links to websites, and emojis. in addition, images can be used to substantiate textual statements. this variety presents a challenge when trying to classify argument stances and their persuasiveness from sources such as x (formerly known as twitter). as the analysis of our approach was able to reveal, the prevalence of sarcasm and the limited information content in tweets substantially complicates the classification. this observation underscores the need for further improvement of models tailored to the specific characteristics of social media data. in the context of classifying the additional persuasive power of images over text, it is crucial to use models that not only extract image features or detect objects in images, but can also extract the attitude and persuasion expressed through the images themselves. this necessitates the design of visual argument extraction models. the particular difficulty of evaluating the argumentative persuasiveness of images, as well as the inherently subjective nature of the task, require special attention. what is more, due to the training dataset's limited size, it becomes challenging to differentiate learned image features at a granular level from those in other images. a larger dataset may assist us in improving classification results, particularly to overcome the challenges outlined in section 5. possible research directions also include delving into the applicability of camembert to english texts and exploring the reasons why this model surpasses english models in the task at hand. https://t.co/3tocapro1b via @twitchyteam 9 . in the test set, this tweet has been labeled as opposing gun control. however, there is no clear evidence that the tweet explicitly expresses a contrary view, as the presence of sarcasm might be a factor to consider in this case given the particular use of quotation marks.