UMUT eam at S em E val-2024 Task 4: Multimodal Identification of Persuasive Techniques in Memes through Large Language Models (2024)

Abstract: In this manuscript we describe the UMUTeam’s participation in SemEval-2024 Task 4, a shared task to identify different persuasion techniques in memes. The task is divided into three subtasks. One is a multimodal subtask of identifying whether a meme contains persuasion or not. The others are hierarchical multi-label classifications that consider textual content alone or a multimodal setting of text and visual content. This is a multilingual task, and we participated in all three subtasks but we focus only on the English dataset. Our approach is based on a fine-tuning approach with the pre-trained RoBERTa-large model. In addition, for multimodal cases with both textual and visual content, we used the LMM called LlaVa to extract image descriptions and combine them with the meme text. Our system performed well in three subtasks, achieving the tenth best result with an Hierarchical F1 of 64.774%, the fourth best in Subtask 2a with an Hierarchical F1 of 69.003%, and the eighth best in Subtask 2b with a Macro F1 of 78.660%.

## introduction the rise of social media has facilitated the rapid spread of information. however, its unconstrained nature has also led to the spread of information whose accuracy is difficult to verify. as a result, misinformation and disinformation have become serious problems in everyday life. for example, during the covid-19 pandemic, social media enabled healthcare professionals to quickly communicate professional information to the public; however, studies also revealed the spread of inaccurate health-related information @xcite . a special case of spreading misinformation is the use of memes. memes consist of images overlaid with text created by internet users and have become one of the primary forms of content in online disinformation campaigns. designed specifically to actively spread inaccurate information, "disinformation memes" are particularly effective on social media platforms, where they can quickly reach large audiences @xcite . using various rhetorical and psychological techniques such as causal oversimplification, name-calling, and smear tactics, memes play a pivotal role in influencing users' perceptions and beliefs. to address this phenomenon, the multilingual detection of persuasion techniques in memes shared task has been organized at semeval-2024 @xcite . the goal of this task is to develop models for detecting persuasion techniques in the textual content of a meme, as well as in a multimodal setting where both textual and visual content are analyzed together. the task is divided into three main subtasks: to solve the english challenge, we propose an approach based on fine-tuning transformer models for binary and hierarchical multi-label classification problems of persuasion techniques using textual and visual content. during training for subtasks 2a and 2b, we used a large multimodal model (lmm) called llava @xcite to extract textual and visual features from the memes. we then refined the monolingual model, as roberta-large @xcite , to identify persuasion techniques and their type. in multimodal classification problems, our experiments showed that including the textual description of the meme obtained by an lmm improves the overall performance. in our experiments, this strategy achieved better results and required fewer resources than merging the image and text embeddings into the same vector space. ## 655 the rest of this paper is organized as follows. section 2 provides a summary of important details about the task setup. section 3 provides an overview of our system for two subtasks. section 4 presents the specific details of our systems. section 5 discusses the results of the experiments, and finally the conclusions are presented in section 7. ## background recently, there has been a significant increase in the use of memes on social media as a means of spreading misinformation. memes consist of a combination of text and images that together have a meaning that is very difficult to automatically verify. in addition, the image and text of the meme in isolation may convey a benign meaning, but their combination may be derogatory, or vice versa. fake news and hate speech purveyors use memes as a tool to spread misinformation and hateful content. they may spread hate to create unrest among the people, and such hateful content may target communities or individuals based on religion, ethnicity, race, national origin, affiliation, sexual orientation, gender, sex, disability, and disease @xcite . many studies have focused on identifying memes that contain negative content or misinformation. for example, the authors of @xcite published a dataset of religiously hateful memes and evaluated it fine-tuning visualbert, which was pre-trained on the conceptual caption (cc) dataset for the top-down classification task. visual features were extracted using resnext-152 aggregated residual transformations based masked regions with convolutional neural networks (r-cnn) and bert without textual encoding for the early fusion model. regarding multimodal approaches, there have been tasks previously organized in the same area of interest. mami (multimedia automatic misogyny identification) at semeval-2022 @xcite , which explored the detection of misogynistic memes on the web using available text and images; and dra-vidianlangtech at eacl-2021 @xcite , which explored the detection of offensive language and classification of troll memes. the novelty of this shared task is the focus on disinformation propaganda through memes. propaganda uses psychological and rhetorical techniques to achieve its goal. these techniques include the use of logical fallacies and appeals to the audience's emotions. logical fallacies are often difficult to detect because the argument seems correct and objective at first glance. however, careful analysis reveals that the conclusion cannot be deduced from the premise without the misuse of logical rules. therefore, memes are a perfect medium for spreading disinformation because they consist of an image superimposed on text, and the image can be deceptive, reinforcing or complementing one or more persuasive techniques in the text or image. thus, the goal of this task is to identify the existence and type of persuasion techniques through memes with different subtasks. the persuasion techniques can be viewed on the official task page. foot_0 . it is worth noting that a similar propaganda technique was used in dipromats 2023 (iberlef) @xcite . the dataset used for this task is the one provided by the organizers. it consists of a set of texts and images labeled with their corresponding persuasion techniques and a binary annotation for subtask 2b. the data set provided by the organizers is divided into train, dev, and validation. note that we do not actually need two datasets for validation (dev and validation), since the dev set was used for the development phase. therefore, we have combined the train and dev sets into a single training set. the training dataset contains 8000 examples for subtasks 1 and 2a and 1499 examples for subtask 2b. figure 1 shows the distribution of the training set for subtasks 1a and 2a and figure 2 for subtask 2b. ## system overview figure 3 shows the architecture of our system for the three subtasks. we can see that for subtask 1, only the text of the memes is used, which is a multi-label classification problem of different persuasion techniques. to address subtask 1, we have fine-tuned roberta-large @xcite . for subtasks 2a and 2b, we have used a similar approach as in subtask 1, but including textual and visual features. we rely on llava @xcite to extract the image description and then concatenate this information with the textual content of the memes, as shown in figure 4 . llava is an endto-end multimodal large language model (llm) that incorporates a vision encoder for general pur- . the techniques are: (01) black-and-white fallacy / dictatorship; (02) loaded language; (03) glittering generalities (virtue); (04) thought-terminating cliché; (05) whataboutism, (06) slogans, (07) causal oversimplification; (08) smears; (09) name calling/labeling; (10) appeal to authority; (11) exaggeration/minimisation; (12) repetition; (13) flag-waving; (14) appeal to fear/prejudice; (15) reductio ad hitlerum; (16) doubt; (17) misrepresentation of someone's position (straw man); (18) obfuscation, intentional vagueness, confusion; (19) bandwagon; (2) presenting irrelevant data (red herring); (21) transfer; (22) appeal to (strong) emotions. ## experimental setup in this work, we used only the dataset provided by the organizers and we did not rely on external data except for the use of llm and lmm models that were pre-trained with general purpose data. before fine-tuning, we performed a preprocessing step to remove line breaks, hashtags, symbols, references, and hyperlinks. next, for all the subtasks, we performed the fine-tuning process using an epoch-based evaluation strategy with the huggingface trainer library foot_2 . this involves training the pre-trained model for a certain number of epochs and performing an evaluation with the evaluation set after each epoch. once all epochs have been completed, the model with the best macro f1 score in the evaluation set is selected. in this way, overfitting or underfitting resulting in low variance and high bias can be avoided. we used the same hyperparameters for finetuning in all the subtasks: (1) a batch size of 8 for both training and validation, (2) 10 epochs, (3) a learning rate of 2e-5, (4) and a weight decay of 0.01. during training, we used macro-f1 as a reference. for the evaluation of subtasks 1 and 2a, the organizers used hierarchical-f1 as the primary evaluation metric, and for subtask 2b, macro-f1. it should be noted that in order to ensure the reproducibility of the experiment, we modified the llava generation configuration by setting the value of do_sample to false. hierarchical precision, recall and f1 (h-p, h-r, and h-f1) are metrics used in hierarchical classification problems where classes are organized in a hierarchical structure @xcite . h-f1 considers both precision and recall of the prediction for each class in the hierarchy, taking into account the relationship between parent and child classes in the hierarchy. the binary task (subtask 2b) is evaluated using the macro f1 score, which is an evaluation metric used in classification problems to measure the precision and recall of a model in predicting multiple classes. it assigns equal weight to each class, meaning that all classes have the same impact on the final metric, regardless of their size or distribution in the data. figure 4: example of multimodal input for subtasks 2a and 2b. ## results for subtask 2b, which is a binary classification problem to identify the presence of persuasion techniques in memes, we obtained a macro-f1 score of 78.660%, which puts us in eighth place according to the official ranking table (see based on the results obtained, it's clear that combining image descriptions with textual content im-4 658 ## error analysis as far as we know, the organizers did not provide the gold labels of the test set to the participants. therefore, we did a bug analysis based on the results of the development set. in subtask 1 our approach is based on a fine-tuned model of roberta-large. we obtained an h-f1 of 62.201 and an m-f1 of 35.514. from the confusion matrices (see figure 5 ), we can see that the model didn't correctly predict any instances of the classes misrepresentation of someone's position (straw man), obfuscation, intentional vagueness, confusion, presenting irrelevant data (red herring), and reductio ad hitlerum, indicating a possible class imbalance or lack of representative features for these classes. in addition, the f1 score of the whataboutism and causal oversimplification class is relatively low compared to other classes, suggesting that the model has difficulty correctly identifying instances of this class, possibly due to ambiguous or overlapping features with other classes. subtask 2a is a hierarchical multi-label classification problem, but unlike subtask 1, it uses a multimodal dataset, i.e. it uses both textual and visual multimodal features to identify 22 persuasion techniques. in this case, our model achieved an h-f1 of 67.902 and an m-f1 of 36.841, which is an improvement over the unimodal approach (subtask 1). however, similar to the model in subtask 1 (see figure 6 ), it failed to predict any instances of the classes misrepresentation of someone's position (straw man), obfuscation, intentional vagueness, confusion, and presenting irrelevant data (red herring), and it obtained a lower f1 score in casual oversimplification and appeal to (strong) emotions, except for the class reductio ad hitlerum, for which it correctly predicted 2 instances. this could be due to insufficient training data or ineffective feature representation for these classes. regarding subtask 2b, a multimodal binary classification problem, our approach achieved an m-f1 of 76.836, and in figure 7 we can see that the model misclassified 40% of the examples as nonpropagandistic and 9% as propagandistic. ## conclusion in this paper, we describe our participation in a se-meval task focused on identifying persuasive techniques in memes using a multimodal approach. for all three subtasks, we used the fine-tuning approach with the roberta-large model for text features and llava to extract image descriptions and combine them with the meme text. our system achieved the tenth best result with an h-f1 of 64.774%, the fourth best in subtask 2a with an h-f1 of 69.003%, and the eighth best in subtask 2b with a macro-f1 of 78.660%. as further work, we will evaluate the relationship between the persuasion techniques used in the different domains evaluated by our team. in this sense, we propose to re-annotate the spanish sati-corpus 2021 @xcite and the politices 2022 dataset @xcite , which are focus on figurative language and politics respectively, with the 22 persuasion techniques and evaluate the reliability of using binary and hierarchical multi-label classification approaches. another area where persuasion techniques may be present is in the identification of misogyny @xcite . 5 659