S ute A lbastre at S em E val-2024 Task 4: Predicting Propaganda Techniques in Multilingual Memes using Joint Text and Vision Transformers (2024)

Abstract: The main goal of this year’s SemEval Task 4 isdetecting the presence of persuasion techniquesin various meme formats. While Subtask 1targets text-only posts, Subtask 2, subsectionsa and b tackle posts containing both imagesand captions. The first 2 subtasks consist ofmulti-class and multi-label classifications, inthe context of a hierarchical taxonomy of 22different persuasion techniques.This paper proposes a solution for persuasiondetection in both these scenarios and for vari-ous languages of the caption text. Our team’smain approach consists of a Multimodal Learn-ing Neural Network architecture, having Tex-tual and Vision Transformers as its backbone.The models that we have experimented with in-clude EfficientNet and ViT as visual encodersand BERT and GPT2 as textual encoders.

## introduction in nowadays society, the role of social media in opinion formation is more important than ever. a fundamentally form of social media leisure, the meme has become a powerful resource which can easily be abused by various entities with political interests. the most well known platforms have a strict policy regarding misleading information, especially of political nature. however, posts containing such information are hard to automatically detect, and the administrators mostly rely on user reports. this paper proposes an automatic detection solution for arabic, bulgarian, english and north macedonian, suitable for both text-only and textimage memes. @xcite only english training data was provided for all the subtasks, all the other languages' tasks requiring a one-shot learning approach. the proposed model excels on subtask 2a, scoring 2 nd place for all languages, besides english and also on subtask 2b where it also ranked second, achieving an f 1-score of over 0.8 in the binary setup. it struggled, however, on subtask 1, especially in the case of the subtask variants for languages without training samples, achieving an f 1-score of about 0.2 on average. the full results table can be found in the results subsection in python code for all the used models and algorithms is available in our github repository. ## experimental setup for the training of models we have used the data splits in the following manner. before the dev gold labels were available, we have used the validation dataset in order to find the best hyperparameters and after that added the validation dataset to the training data. after the dev gold labels were published, in order to submit on the test dataset, we have also trained on the dev dataset. this maximized the number of training samples available. the preprocessing techniques used for the images are: the preprocessing techniques used for the texts are: the experiments conducted can be seen in tables 2, 3 and 4. in the first task we approached was subtask 2b. we first used varieties of the efficientnet model for the visual encoder, later switching to vit encoding thanks to its higher performance. the metrics used for subtask 1 and subtask 2a are hierarchical precission (hp ), hierarchical recall (hr) and hierarchical f1 score (hf 1 ). in the multi-label settings, we define ĉi as the set of ground truth classes and all their ancestors and ĉ′ i as the set of predicted classes and all their ancestors @xcite . thus the metrics are represented by : for subtask 2b, the metrics used are the classic f 1 macro and f 1 micro. ## conclusion our model accurately creates dense embeddings for both the memes and their captions, managing to make use of their most prominent features in computing the result. the performance peaks on subtask 2a. the transformer components used in the proposed architectures are capable of learning from vast datasets with low risk of overfitting. thus, one way of improving the solution would be the use of additional datasets related to the subject, in various languages. last but not least, our future work will revolve around decision making on the hierarchical dag, further making use of relationship between different labels.