Semantists at I mage A rg-2023: Exploring Cross-modal Contrastive and Ensemble Models for Multimodal Stance and Persuasiveness Classification (2023)

Abstract: In this paper, we describe our system for ImageArg-2023 Shared Task that aims to identify an image’s stance towards a tweet and determine its persuasiveness score concerning a specific topic. In particular, the Shared Task proposes two subtasks viz. subtask (A) Multimodal Argument Stance (AS) Classification, and subtask (B) Multimodal Image Persuasiveness (IP) Classification, using a dataset composed of tweets (images and text) from controversial topics, namely gun control and abortion. For subtask A, we employ multiple transformer models using a text based approach to classify the argumentative stance of the tweet. For sub task B we adopted text based as well as multimodal learning methods to classify image persuasiveness of the tweet. Surprisingly, the text-based approach of the tweet overall performed better than the multimodal approaches considered. In summary, our best system achieved a F1 score of 0.85 for sub task (A) and 0.50 for subtask (B), and ranked 2nd in subtask (A) and 4th in subtask (B), among all teams submissions.

## introduction persuasiveness mining is an important task within argument mining @xcite , that is aimed at detecting and analyzing the ability to influence one's beliefs, attitude, intentions, motivation, and behavior @xcite . it has gained increased attention recently @xcite though most of the research focused on texts. persuasion, however, may depend not only on natural language but on other modalities (eg. visual means) as well. imagearg is an initiative that attempts to capture this opportunity and expand persuasiveness mining into a multi-modal realm @xcite . it presents a multi-modal dataset consisting of annotations on tweets along with associated images, that supports benchmark-ing of state-of-the-art models on multiple argumentative classification tasks. imagearg shared task 2023 proposes two subtasks viz. subtask (a) multimodal argument stance (as) classification: given a tweet composed of a text and image, predict whether the given tweet supports or opposes the topic, and subtask (b) multimodal image persuasiveness (ip) classification: given a tweet composed of text and image, predict whether the image makes the tweet more persuasive or not. in this paper, we report our systems for addressing both the subtasks. transformer based multimodal text-embedded classification has been a promising approach recently @xcite @xcite @xcite @xcite . taking inspiration from this, we explore multiple transformer models using text as well as multimodal learning methods, for both subtasks (a) and (b). surprisingly, the text-based approach of the tweet performed better than the multimodal approaches considered. in particular, our best text based model achieved a f1 score of 0.85 for sub task (a) and 0.50 for subtask (b), and ranked 2nd in subtask (a) and 4th in subtask (b), among all teams submissions. also, our benchmark results highlight the challenge of these tasks and indicate there is ample of room for model improvement. we demonstrate the limitation of these general multi-modal methods and discuss possible future work. ## task and dataset description imagearg dataset is composed of tweets (images and text) from controversial topics, namely gun control and abortion. imagearg shared task is divided into two subtasks. subtask a: argumentative stance (as) classification given a tweet composed of a text and image, predict whether the given tweet supports or opposes the given topic, which is a binary classifi-cation task. subtask b: image persuasiveness (ip) classification given a tweet composed of text and image, predict whether the image makes the tweet text more persuasive or not, which is also a binary classification task. for convenience, below we refer to the subtasks (a) and (b) simply as tasks a and b. for task a, as the training data is not large, we ventured to explore a predominantly text-based approach, with tweet text and tweet image contents extracted from ocr fed as the inputs to the system. our idea was to build a model capable of learning their corresponding unified representations which could be sufficiently discriminative in the stance detection classifier space. we considered multiple candidate models that satisfy this criteria and evaluated them on the imagearg dataset. for all our approaches, we randomly split the instances into 80/20 percent and performed 5-fold cross-validation on the validation(dev) set to select the best model. approach 1: (t5 nli) we used pretrained t5(text-to-text transformer) to fine tune the model for the given dataset and also adjusted the hyper-parameters based on the best performance. during t5 training, we set the number of beams as 50 and the number of returned sequences as 5. approach 2: (bertweet-based model) sentiment based classifier using bertweet(nguyen et al., 2020), a large-scale language model pretrained for english tweets using roberta model and cross-entropy loss with custom linear layers. the positive and negative labels of the classifier corresponds to support and oppose labels of stance classification task. we have used the pretrained bertweet model and fine-tuned the model for its best performance. approach 3: (contrastive bert model) : we adopt a multi-task contrastive learning framework with a two step representation learning paradigm, similar to @xcite . firstly, stance label prefixed textual sequences were fed as inputs to a transformer encoder as the target input anchor. second, the corresponding positive and negative reference input samples were fed as inputs to a shared bert encoder in the parameter space. then, the final hidden state classifier token [cls] is used as the standalone output label representation of the input sequence and the remaining hidden layer outputs served as sequence representation with labels encoded. among the different variants of contrastive learning available, we used dual stance aware supervised contrastive learning technique with linear classifier. we also evaluate several candidate groups to serve as the anchor, positive, negative reference triplets in the proposed dual stance aware supervised contrastive learning space and found the more straightforward tweet text to act as efficient anchors in this space. approach 4: (ensemble model) : we also considered a final model that uses an ensemble approach. in this model, we classify new data points by first applying the above 3 models, and then taking a majority vote of the predictions. in other words, the final prediction is determined by the class predicted by at least two models. we have experimented a few other approaches, but as we observed on validation set, contrastive bert performed the best, followed by t5 and bertweet based model. the ensemble model was marginally better in comparison. thus we considered only these four models. ## experiments & results: for both task a and task b, we used tweet text, tweet image-text (ocr to text, using easyocr tool foot_0 ) and custom pre-processing techniques to refine and clean up the textual sequences. for the latter, we used the bertweet preprocessing scripts foot_1 . all the images are resized to (224*224) dimension and minor data augmentation(i.e., horizontalflipped, rotation) was performed during training.subsequently, we trained and performed experiments as outlined in section 4. to measure performance, we employ precision, recall and f1-score as metrics. in addition, we also consider class-weighted f1-score to account for class imbalance. the experiments were executed on nvidia-geforce tesla v100 series sxm2-32gb with 5 cores of gpu machines. models were trained for 10 epochs, and the pretrained weights for the transformers prior to fine-tuning were downloaded from the huggingface library. ## conclusion this paper described our system for imagearg-2023 shared task consisting of two subtasks viz. subtask (a) multimodal argument stance (as) classification, and subtask (b) multimodal image persuasiveness (ip) classifi-cation. the tasks used a dataset composed of tweets (images and text) from controversial topics, namely gun control and abortion. for subtask (a), we employ multiple transformer models using a text based approach to classify the argumentative stance of the tweet. for sub task (b) we adopted text based as well as multimodal learning methods to classify image persuasiveness of the tweet. surprisingly, the text-based approach of the tweet overall performed better than the multimodal approaches considered. in summary, our best system achieved a f1 score of 0.85 for sub task (a) and 0.50 for subtask (b), and ranked 2nd in subtask (a) and 4th in subtask (b), among all teams submissions. the results imply that image persuasiveness classification is a far more challenging problem and there is a significant room for improvement. however, it might require larger datasets to deal with the multimodal classification challenges.