IUST at I mage A rg: The First Shared Task in Multimodal Argument Mining (2023)

Abstract: ImageArg is a shared task at the 10th ArgMining Workshop at EMNLP 2023. It leverages the ImageArg dataset to advance multimodal persuasiveness techniques. This challenge comprises two distinct subtasks: 1) Argumentative Stance (AS) Classification: Assessing whether a given tweet adopts an argumentative stance. 2) Image Persuasiveness (IP) Classification: Determining if the tweet image enhances the persuasive quality of the tweet. We conducted various experiments on both subtasks and ranked sixth out of the nine participating teams.

## introduction argumentation mining, a task in natural language processing (nlp), aims to automatically detect argumentative structures in a document @xcite . this process unveils not only people's viewpoints but also the reasons behind their beliefs @xcite . it offers valuable insights across a wide spectrum of fields, ranging from predicting financial market trends to public relations. however, prior research in this field mainly concentrates on text and does not exploit multimodal data. imagearg, a multimodal dataset introduced by @xcite , is designed to bridge this gap. it includes persuasive tweets accompanied by images and its goal is to identify the image's stance towards the tweet and assess its persuasiveness score on specific topics. imagearg constitutes a collaborative challenge @xcite tailored to advance multimodal persuasive techniques, using the imagearg dataset. it is made of two subtasks: argumentative stance (as) classification and image persuasiveness (ip) classification which will be further discussed in subsection 3.1 and subsection 3.2 respectively. the whole system architecture is shown in fig. 1 . we make three experiments on the as subtask. while in two of them, we only used the text as input data, in the third, we adopted a multimodal approach, considering both image and text inputs. in the former, we utilize bertweet @xcite in one experiment and roberta @xcite in the other. for our final experiment, we employed the multimodal bitransformer (mmbt) architecture @xcite , harnessing tweets' text, text within images, and the images themselves. our first approach, which leveraged bertweet, achieved the highest f1-score compared to the two other methods. for the ip task, we conducted a single experiment employing the mmbt model. we employed tweets' text, text extracted from images, and the images themselves as inputs. multimodal learning: the recent surge in attention towards ai models lies in their capability to handle and comprehend inputs from multiple sources, thanks to the complementary nature of these multimodal signals in real-world applications @xcite ; @xcite ; alwassel et al. (2020)). within the field of vision and language, tasks primarily revolve around assessing the models' proficiency in both grasping visual data and articulating reasoning through language @xcite ; @xcite ; @xcite ). although some research diverges from this mainstream which explores the connection between images and text: @xcite delve into annotating discourse relations between textual and accompanying visual elements in recipe instructions, while @xcite delve into understanding multimodal document intent in instagram posts. ## task and data imagearg shared task includes two subtasks: argumentative stance (as) classification and image persuasiveness (ip) classification. the dataset provided for this task encompasses two distinct topics of societal significance, namely abortion and gun control. within the training subset of the dataset foot_0 , a total of 912 examples are allocated to the domain of gun control, while 887 examples pertain to the topic of abortion. in the development subset, there are 100 data entries related to abortion and 97 data records related to gun control. in the testing partition, both the abortion and gun control categories are represented equally, each comprising 150 examples. in the following parts, we will provide more details about subtasks and statistics related to the data specified for each subtask. ## methods we first present preprocessing techniques used for both subtasks, as well as some ideas to make the performance of both tasks better before training models. next, we introduce models developed for the argumentative stance followed by image persuasiveness models. ## experiments and results first, we discuss our results of the first subtask, which is summarised in tab.2. our first and best submission for argumentative stance classification was bertweet which is a variant of bert specifically trained for tweets. we achieved 0.7754 f1score on test data and we stand out as the 6-th team among others. bert-based models are known for their strong performance in various nlp tasks, and this experiment confirms their utility for argumentative stance classification in tweets. roberta was the second submission and its result was highly close to bertweet, with a score of 0.7752 based on f1. it suggests that incorporating text from images did not notably enhance the model's performance, which is an interesting finding. it is possible that the text within images may not have provided much additional useful information for this specific task. both bertweet and roberta were trained for 10 epochs with batchsize of 8, using adamw as optimizer @xcite . mmbt was the last approach and it did not perform as well as the two first approaches. it yielded a noticeably lower f1-score of 0.5680 compared to the text-only models. although we employed separate models for each topic, the image encoder was the same and we utilized clip-rn50x4 for this purpose. in addition, weighted binary cross entropy (bce) was used as a loss function and we specified a weight according to class distribution for each topic for a better performance. the drop in performance could indicate that the addition of image information did not help and may have even introduced noise or complexity into the model. it's important to note that multimodal models can be challenging to train and may require a substantial amount of data and careful tuning to outperform text-only models in specific tasks. figure 5: examples of image persuassieveness that were classified correctly in the second subtask, the only approach we followed was mmbt. the specified model for the gun control topic employed clip-rn101 as its image encoder whereas the abortion model used clip-rn50x16. these models were trained for 10 epochs with batch-size of 32. its result is shown in tab.3. while the model's performance may not be exceptionally high, it demonstrates some capability in assessing the persuasiveness of tweets with both text and image content. some instances of the dataset with the model's predictions are shown in fig. 4 and fig. 5 . ## conclusion in this paper, we presented our approach in the im-agearg shared task which was the first shared task in multimodal argument mining. we proposed three methods for the first subtask. these models have different varieties from models solely dependent on text to multimodal pre-trained models. we also had only one submission for the second subtask and we achieved 6-th place in both tasks among other groups that participated.