Snarci at S em E val-2024 Task 4: Themis Model for Binary Classification of Memes (2024)

Abstract: This paper introduces an approach developed for multimodal meme analysis, specifically targeting the identification of persuasion techniques embedded within memes. Our methodology integrates Large Language Models (LLMs) and contrastive learning image encoders to discern the presence of persuasive elements in memes across diverse platforms. By capitalizing on the contextual understanding facilitated by LLMs and the discriminative power of contrastive learning for image encoding, our framework provides a robust solution for detecting and classifying memes with persuasion techniques. The system was used in Task 4 of Semeval 2024, precisely for Substask 2b (binary classification of presence of persuasion techniques). It showed promising results overall, achieving a Macro-F1=0.7986 on the English test data (i.e., the language the system was trained on) and Macro-F1=0.66777/0.47917/0.5554, respectively, on the other three “surprise” languages proposed by the task organizers, i.e., Bulgarian, North Macedonian and Arabic. The paper provides an overview of the system, along with a discussion of the results obtained and its main limitations.

## introduction in recent years, the natural language processing (nlp) community has witnessed an evergrowing number of contributions aimed at identifying and analyzing various forms of harmful language found on the web, including offensive language @xcite , hate speech @xcite -also comprising misogyny and transphobia @xcite , and propaganda techniques @xcite . these linguistic phenomena not only harm civil debate but can also fuel the polarization and radicalization of users' opinions. in an increasingly multimodal context, particular attention has also been paid to memes @xcite , which, due to their virality and communicative immediacy, can easily become key tools in online disinformation campaigns. therefore, the development of techniques to effectively classify possible nuances of information manipulation within these forms of content sharing assumes a central role in online disinformation research. this motivated our participation in the semeval 2024 task 4 foot_0 @xcite , which focuses on "multilingual detection of persuasion techniques in memes". the task aims to develop models capable of identifying rhetorical and psychological techniques employed in memes to influence users' opinions. our team participated in substask 2b, consisting of a binary classification problem to determine whether a meme contains at least one persuasion technique among the predefined set of 22 techniques. the dataset released to participants is made up of memes with textual content in english. however, to assess the robustness of the systems during the evaluation phase, the organizers made test sets available in three other languages besides english, i.e., arabic, bulgarian, and north macedonian. for the purpose of this task, we developed a system that combines large language models (llms) and contrastive learning image encoders to discern the presence of persuasive elements in memes across diverse platforms. the following sections will thus describe the system architecture and its deployment in the task. a discussion of the results obtained and of some most recurring errors will also be proposed, aiming to highlight possible research paths for the further improvement of the approach. ## system overview the proposed method, named "themis", is a modular neural network architecture designed to analyze multimodal data, specifically targeting memes that often contain both textual and visual elements. ## experiment setup the experiments were executed on a workstation featuring an intel core i7-12700 @ 2.1ghz cpu, 32 gb ram, and an nvidia rtx3060 gpu with 12gb of memory. among the different experiments, one main issue is denoted by the limited availability of vram; this issue not only limited our approach to smaller llm and image encoders but also limited batch size. our experiments aim to enable efficient prediction even in such lowend system requirements. as a result, we opted for a pre-trained contrastive language-image pre-training (clip) @xcite encoder as our main image encoder. specifically, we used both clip base foot_1 and clip large foot_2 in our experiments. for the textual part, instead, we used tinyllama foot_3 @xcite , phi-1.5 foot_4 @xcite and phi-2 foot_5 . for each combination of image and text models, the system was trained for 20 epochs, using a batch of 2 and a learning rate of 1e -4, adamw as the ## results the task was organized into two main evaluation phases: a development phase, during which only training and unlabeled development data were accessible, and a test phase, wherein the gold labels for the development set were disclosed alongside the unlabeled test sets in four languages: english, arabic, bulgarian, and north macedonian. in this section, we outline the results obtained by our experiments in both phases. ## discussion and error analysis despite achieving promising results, in terms of macro-f 1 scores, our model still occasionally misclassifies instances, particularly in cases involving the propagandistic nature of memes. although the labeled test set was not made available by the organizers, we were still able to inspect more in detail the results obtained on the development set. figures 3 and 4 illustrate examples of false positive and false negative predictions, respectively. in both instances, we can formulate hypotheses regarding why and when our model generates errors. the false positive example may be misconstrued as employing a "slogan" based persuasion technique, possibly due to text present in the sign. the false negative could stem from the model's inability to recognize inherent sarcasm due to limited sarcastic examples, further train with a larger dataset could mitigate this issue. the confusion matrix depicted in figure 2 reveals an uneven distribution of errors across both classes, indicating a bias towards the propagandistic class. this bias could be attributed to the imbalance in the training set. ## conclusions in this study, we introduced themis, a novel model for analyzing multimodal memes by integrating llms and contrastive learning image encoders. through comprehensive experiments, themis demonstrated remarkable efficacy in detecting persuasion techniques within memes, achieving a notable f1 score of up to 83.4%. our findings underscore the critical role of meticulous model architecture design and hyperparameter optimization in meme analysis tasks. notably, themis presents a robust solution to combat societal challenges posed by biased content online, offering a promising avenue for mitigating the spread of misinformation and promoting digital discourse integrity.