Overview of I mage A rg-2023: The First Shared Task in Multimodal Argument Mining (2023)

Abstract: This paper presents an overview of the ImageArg shared task, the first multimodal Argument Mining shared task co-located with the 10th Workshop on Argument Mining at EMNLP 2023. The shared task comprises two classification subtasks - (1) Subtask-A: Argument Stance Classification; (2) Subtask-B: Image Persuasiveness Classification. The former determines the stance of a tweet containing an image and a piece of text toward a controversial topic (e.g., gun control and abortion). The latter determines whether the image makes the tweet text more persuasive. The shared task received 31 submissions for Subtask-A and 21 submissions for Subtask-B from 9 different teams across 6 countries. The top submission in Subtask-A achieved an F1-score of 0.8647 while the best submission in Subtask-B achieved an F1-score of 0.5561.

## introduction research in argument mining (am) typically centers around the examination of an author's argumentative position, achieved through the automated identification of argument structures. this research has predominantly concentrated on domains presented in textual formats, encompassing endeavors such as mining persuasiveness in essays @xcite and user-generated web discourse @xcite . recently, there has been a growing recognition of the need for multimodality in am research. a noteworthy development in this regard is the retrieval for argument shared task @xcite . this task is designed to retrieve images related to a controversial topic that aligns with the textual stance, whether it supports or contradicts the topic. in a related context, @xcite introduced the imagearg corpus, which is designed to investigate multimodal persuasiveness within tweets. this corpus represented an advancement in the field of automated persuasive text identification @xcite by introducing a new modality through the inclusion of images. this paper introduces the imagearg shared task foot_0 , building upon the groundwork laid by @xcite and conducted as a part of the 10 th workshop on argument mining foot_1 . the shared task comprises two subtasks that center around two highly controversial topics (gun control and abortion): figure 1 shows examples of the two subtasks. the upper left tweet expresses a strong stance towards supporting gun control by indicating a house bill about the requirement of background checks for all gun sales. the upper right tweet opposes gun control because it is inclined to self-defense. the lower left tweet has an image irrelevant to the gun control topic. it does not improve the persuasiveness of the tweet text (and its stance) that argues to focus on mental health instead of gun restriction. the lower right tweet makes the tweet text (and its stance) more persuasive because it provides strong evidence to show the statistics of the murder rate in major u.s. cities due to restrictive gun control laws, so citizens cannot easily arm themselves. the shared task received 31 submissions for subtask-a and 21 submissions for subtask-b from 9 diverse teams, comprising both academic experts from various universities and industry researchers, across 6 different countries. in general, the submissions that utilized text information from tweet images and performed data augmentation yielded favorable results for subtask-a. the submissions that utilized unified multimodal models also achieved good performance in subtask-b. the highest subtask-a f1-score of 0.8647 was attained by team knowcomp @xcite , while the leading subtask-b f1-score of 0.5561 was attained by team feeds @xcite . details about task submissions are described in section 4. ## related work multimodal learning recently, there has been increasing attention to assessing the ability of artificial intelligence models to process and understand multimodal input signals that occur in real-world applications @xcite . in the vision-language domain, tasks are primarily designed to evaluate the capacity of models to comprehend visual data and articulate reasoning in language @xcite . in addition, @xcite are interested in the discourse relations between text and its associated images in recipes, while @xcite explores the multimodal document intent of instagram posts. more recently, @xcite introduce imagearg, the first multimodal learning corpus for argument mining. however, the size of the imagearg corpus is small, which motivates our construction of an extension of the original corpus. regarding multimodal modeling, researchers have developed methods to derive strong representations for each modality and implement fusion techniques @xcite @xcite . although several shared tasks in machine translation @xcite and argument retrieval @xcite have revealed the effectiveness of multimodal learning, none of them focused on argument persuasiveness. therefore, this shared task provides opportunities to benchmark the new multimodal argument persuasiveness corpus by utilizing various image and text encoders along with effective fusion strategies. in a multimodal fashion @xcite , it is important to note that their focus is on sentiment analysis, intent reasoning, and persuasive strategies tailored specifically for advertisements. in contrast, our shared task is interested in argument mining, marking an aligned goal to the imagearg work @xcite , offering substantial value to multimodal computational social science. ## corpus we extended the imagearg corpus @xcite by following its annotation protocol to annotate new data on abortion and gun control topics. specifically, we annotated 1141 new abortion tweets and 301 new gun control tweets. parts of the new gun control tweets were used to replace 131 out of the original 1003 gun control tweets in the imagearg corpus which were no longer available due to deletions or account suspensions. the other extras were annotated to ensure gun control and abortion tweets have close data distributions. therefore, we obtained 1173 gun control tweets in total. in addition to using the original annotation protocol @xcite , we required annotators to score confidence levels, which was designed to improve the inter-annotation agreement. confidence was divided into 5 levels: l5-extremely confident (understood and answered all annotations carefully), l4-quite confident (tried to understand and answered most annotations carefully), l3-somewhat confident (confused about some annotations), l2-not very confident (did not understand some annotations), and l1-not confident (mostly educated guesses). in the annotation process, each tweet was annotated by three annotators on amazon mechanical turk (amt) 3 who had done more than 5,000 approved annotations with at least 95% approved rates in their historical hits. annotators were required to pass a qualification exam that annotated 3 https://www.mturk.com/ pilot examples with at least 0.7 accuracy. the remaining new as annotations for abortion and gun control have alpha scores of 0.78 and 0.74, respectively. the new ip annotations were also inherited from the imagearg protocol. first, annotators annotated two persuasiveness scores: one for tweet text (s t ), another for tweet text and image (s it ). then we computed a score difference ∆s @xmath0, 0) as a persuasiveness gain from adding a tweet image. the final image persuasiveness score for each tweet was the average of persuasiveness gains from three annotators. to interpret image persuasiveness, we used the same threshold (0.5) in imagearg to split them into binary labels, indicating whether the image made the tweet text more persuasive or not. we split the corpus into train, development, and test sets in the shared task, which obtained 1814 train, 200 development, and 300 test samples for both subtasks foot_2 . the data statistics are shown in id system score modality model notes 1* knowcomp-4 0.8647 i+t resnet50 + deberta augment text with back translation + wordnet 2 knowcomp-5 0.8571 i+t resnet50 + deberta augment text with translation + wordnet + semantic similarityattention 3 knowcomp-1 0.8528 i+t resnet101 + deberta augment text with translation + wordnet 4* semantists-4 0.8506 t+e ensemble of all models 5 semantists-3 0.8462 t+e bertweet ocr on image 6 semantists-5 0.8417 t+e bert dual contrastive loss + ocr on image 7 semantists-1 0.8365 t+e bert contrastive loss + ocr on image 8 semantists-2 0.8365 t+e t5 ocr on image 9 knowcomp-2 0.8365 i+t resnet50 + deberta augment text with translation + wordnet + semantic similarityattention 10 knowcomp-3 0.8346 i+t layoutlmv3 + deberta augment text with translation + wordnet 11* mohammad soltani-2 0.8273 i+t clip32 adaboost for abortion + xgboost for gun control 12* pitt pixel persuaders-2 0.8168 t emsemble all the model 13 mohammad soltani-1 0.8142 i+t clip32 adaboost for abortion and gun control 14 mohammad soltani-4 0.8093 i+t clip32 xgboost for abortion and gun control 15* gc-hunter-2 0.8049 t xlmroberta 16 mohammad soltani-3 0.8000 i+t clip32 adaboost for abortion + rusboost for gun control 17 pitt pixel persuaders-1 0.7910 t bloom-560m 18 mohammad soltani-5 0.7782 i+t clip32 svm-poly for abortion and gun control 19 gc-hunter-1 0.7766 t bert 20* iust-1 0.7754 t+e bertweet augment text with chatgpt paraphraser + ocr on image 21 iust-2 0.7752 t+e roberta augment text with chatgpt paraphraser + ocr on image 22 pitt pixel persuaders-4 0.7710 t bloom-1b 23 pitt pixel persuaders-5 0.7415 t xlnet 24* kpas-1 0.7097 i+t clip 25* act-cs-4 0.6325 i+t+e+c vit+bert cross-attention 26 act-cs-3 0.6178 i+t+e vit+bert cross-attention 27 act-cs-2 0.6116 i+t vit+bert cross-attention 28 act-cs-1 0.5863 i+t vit+bert simple concatenation of features 29 iust-3 0.5680 i+t+e clip+bert augment text with chatgpt paraphraser + ocr on image 30 pitt pixel persuaders-3 0.5285 i+t vilt 31* feeds-1 * * 0.4418 t bert ## submission results we provide summaries about subtask-a (sec. 4.1) and subtask-b (sec. 4.2) submissions for all the teams. in cases where a team did not submit a description paper, we include their results and provide a brief description based on the survey completed by the team at the time of submission. ## conclusion in this paper, we introduced the imagearg shared task, marking a significant milestone as the inaugural shared task in multimodal argument mining, co-located with the 10