BCA mirs at S em E val-2024 Task 4: Beyond Words: A Multimodal and Multilingual Exploration of Persuasion in Memes (2024)

Abstract: Memes, combining text and images, frequently use metaphors to convey persuasive messages, shaping public opinion. Motivated by this, our team engaged in SemEval-2024 Task 4, a hierarchical multi-label classification task designed to identify rhetorical and psychological persuasion techniques embedded within memes. To tackle this problem, we introduced a caption generation step to assess the modality gap and the impact of additional semantic information from images, which improved our result. Our best model utilizes GPT-4 generated captions alongside meme text to fine-tune RoBERTa as the text encoder and CLIP as the image encoder. It outperforms the baseline by a large margin in all 12 subtasks. In particular, it ranked in top-3 across all languages in Subtask 2a, and top-4 in Subtask 2b, demonstrating quantitatively strong performance. The improvement achieved by the introduced intermediate step is likely attributable to the metaphorical essence of images that challenges visual encoders. This highlights the potential for improving abstract visual semantics encoding.

## introduction in this digital age, the influence of persuasive techniques, particularly in memes, is a key focus. propaganda, using various psychological techniques, shapes information for specific agendas. research on political memes, such as @xcite 's work, emphasizes their role in communication and satire. another study on covid-19 memes @xcite underscores the importance of expert-sourced, objective memes in influencing public opinion and aiding public health campaigns. as a result, understanding persuasive techniques in memes within disinformation campaigns is crucial for grasping their impact on public perception and discourse. these campaigns usually succeed in influencing users by employing various rhetorical and psychological strategies in memes, including but not limited to causal oversimplification, thoughtterminating cliché, and smear techniques. to address this concern, we participated in the semeval-2024 shared task 4, as outlined by @xcite . the primary objective of this shared task is to develop models specifically designed to detect rhetorical and psychological techniques within memes. in summary, this task involves three subtasks. in subtask 1 the input is the textual content of a meme only. this could include any written information present within the meme, and the goal is to identify one of the 20 persuasion techniques present in the meme's textual content. the identification is based on a hierarchical structure, and the techniques are organized in a tree-like fashion. subtask 2a involves both textual and visual content analysis of memes, and information present in both the written content and the visual elements of the meme are considered. the task is to identify the presence of 22 persuasion techniques, utilizing a hierarchical structure similar to subtask 1. subtask 2b is a binary classification version of subtask 2a. the training set released for all subtasks contains only english memes. however, alongside the english language, the test datasets contain memes in three low-resource languages (arabic, bulgarian, and north macedonian) that aim to evaluate the zero-shot capability of the proposed models. although we participated in all subtasks, we specifically focused on subtask 2 which uses both the textual and visual modality of memes to do a multi-label classification. to achieve better results, we introduced an intermediate step, the meme captioning step. subsequently, we employed these generated captions to compare the performance of different models like llava-1.5 @xcite , vicuna-1.5 @xcite , bert (devlin et al., 2018), and roberta @xcite . this comparative analysis aimed to elucidate the role of the memes' text, the generated captions, and the memes' images in understanding persuasion techniques used in memes. the results of our best model (at the time of submitting evaluation results) that uses roberta and our ranking relative to other teams in different subtasks are summarized in persuasion technique classification. in exploring persuasion techniques in texts and images, @xcite present a comprehensive framework for meme analysis. the study defines 22 techniques and provides an annotated dataset for conducting nuanced examinations of textual and multimodal memes. the incorporation of historical and mythological references adds depth to understanding the challenges in this domain. moreover, in the work by @xcite , the authors introduce transformer-based @xcite models, vtte and dvtt, for processing textual and visual content in memes. these models effec-tively identify persuasion techniques, with dvtt showing superior performance, particularly in finetuning feature extractors. given the prevalence of large language models (llms) employing similar architectures as dvtt, our experiments include utilizing llms and mllms, which we explain in section 3, to further investigate and advance the detection of persuasion techniques in memes. ## methodology building upon prior research on mllms, the prevailing method involves tokenizing image concepts and conveying these tokens alongside textual tokens to a language model. while these models possess the ability to impart more semantic information from the image, their focus typically centers on identifying objects and their relationships within the image @xcite . consequently, this study explores the impact of initially prompting the model to generate descriptive information aimed at conveying semantic context. we utilize this information for data classification, comparing it to the conventional approach of fine-tuning an end-to-end model. in this section, we outline our approach for generating meme captions and subsequently discuss our classifiers. ## experiments in this section, we outline our conducted experiments and provide a concise overview of the results obtained. readers are referred to appendix a for the details of our experimental settings. for evaluation of the performance of the models for hierarchical classifications, we use hierarchicalprecision, -recall, and -f1 introduced by @xcite . for more information about hierarchical evaluation metrics, see appendix e. in the initial set of experiments, we perform hierarchical multilabel classification using the textual content of memes to fine-tune unimodal models (vicuna, bert, and roberta) directly for identifying specific persuasion techniques. we compare this approach to multimodal models (llava, visu-albert, and concatroberta) where both textual and visual contents of the meme are provided. additionally, we conduct a similar experiment with llava model, feeding only the image of the meme without textual data. this comparison aims to assess the information derived from each modality. the motivation behind our decision to compare encoder-only lrms such as roberta or visu-albert to larger generative models like vicuna or llava, is their promising results in classification tasks compared to generative models @xcite . during the second stage of our experiments, we proposed to ## conclusion and future work this paper explores the persuasive communication within memes, emphasizing their role in shaping public perception. through participation in the semeval-2024 shared task 4, our study delves into the detection of rhetorical and psychological techniques within memes. by employing multimodal models and introducing an intermediate step of meme captioning using llava and gpt-4, we aimed to bridge the gap between textual and visual modalities, thereby enhancing the classification of persuasion techniques. our experiments demonstrate the effectiveness of this approach, with our best model, concatroberta, achieving notable performance improvements. however, we observed that the performance gains varied based on the dataset's nature and the models' sophistication. nevertheless, our findings contribute to advancing understanding in this domain and pave the way for future research endeavors aimed at combating online disinformation campaigns. regarding future work, a deeper analysis into why the model struggles to utilize its image analysis capabilities for classification, despite its proficiency in generating captions (even in zero-shot settings with gpt-4), could be explored through the implementation of chain-of-thought approaches. additionally, exploring how well the proposed method withstands adversarial attacks is another interesting direction. adversarial examples, as shown by different studies @xcite @xcite , have uncovered vulnerabilities in neural models across various tasks. studying how adding the caption generation step affects the adversarial robustness of our approach compared to end-to-end methods for this task holds promise for future research.