nowhash at S em E val-2024 Task 4: Exploiting Fusion of Transformers for Detecting Persuasion Techniques in Multilingual Memes (2024)

Abstract: Nowadays, memes are considered one of the most prominent forms of medium to disseminate information on social media. Memes are typically constructed in multilingual settings using visuals with texts. Sometimes people use memes to influence mass audiences through rhetorical and psychological techniques, such as causal oversimplification, name-calling, and smear. It is a challenging task to identify those techniques considering memes’ multimodal characteristics. To address these challenges, SemEval-2024 Task 4 introduced a shared task focusing on detecting persuasion techniques in multilingual memes. This paper presents our participation in subtasks 1 and 2(b). We use a finetuned language-agnostic BERT sentence embedding (LaBSE) model to extract effective contextual features from meme text to address the challenge of identifying persuasion techniques in subtask 1. For subtask 2(b), We finetune the vision transformer and XLM-RoBERTa to extract effective contextual information from meme image and text data. Finally, we unify those features and employ a single feed-forward linear layer on top to obtain the prediction label. Experimental results on the SemEval 2024 Task 4 benchmark dataset manifested the potency of our proposed methods for subtasks 1 and 2(b).

## introduction modern social media represents a prominent environment to disseminate information to a vast community in real time. hence, persuasion techniques are often embedded in social media content to subliminally influence people and their unconscious opinions. such techniques are now incorporated in memes due to the increasing popularity among social media users. the visual aspect of memes adds to the effectiveness of grabbing people's attention than purely word-based messages. manip-ulators and propagandists now treat it as an effective tool to promote and achieve their nefarious agendas. sometimes different organizations use it to spread fake news or propaganda which causes social chaos and incitement of hate, which could result in harm or even human casualties. hence, the detection of persuasion techniques embedded in memes appears as a formidable task to shield individuals from deceit. moreover, detecting these techniques from memes is a challenging task since it requires a nuanced understanding of images, and texts, and a proper appreciation of the satirical characteristics of memes. to address these challenges, semeval-2024 introduced a shared task focusing on detecting persuasion techniques from multilingual memes @xcite . this task comprises three subtasks. whereas the first task is based on identifying 20 persuasion techniques from meme texts. this is a hierarchical multilabel text classification task. tasks 2(a) and 2(b) are based on multi-modal contents. task 2(a) is a hierarchical multimodal multilabel classification task where the proposed system needs to identify 22 persuasion techniques from multimodal memes. task 2(b) is a multimodal binary classification task where the participants need to apply the multimodal information expressed by memes to classify them into whether they contain a persuasion technique or not. a data sample of each task along with corresponding labels was articulated in however, some prior works have been done on identifying persuasion techniques from texts and visuals. semeval 2023 shared task 3 introduced a subtask based on identifying persuasion techniques used in news articles @xcite . most of the participants used different multilingual transformer models to tackle the challenge of this task. apatt @xcite utilized an ensemble of different pre-trained transformer models e.g., xlnet, roberta, bert, albert, and deto detect persuasion techniques in texts and images, semeval 2021 task 6 introduced three subtasks including multilabel text classification, span identification, and multi-modal multilabel classification task @xcite . the top-performing team on the multilabel text classification task @xcite leveraged five fine-tuned transformer models: bert, roberta, xlnet, deberta, and albert. they made use of external ptc corpus @xcite along with given training data to train these transformer models. team nlpiitr @xcite made use of a fine-tuned roberta model to address the challenge of this task. team volta @xcite explored the potency of fine-tuned bert and roberta models for both multi-label text classification and span identification tasks and used roberta large for the final model. their proposed architecture ranked top on span identification tasks. for the multi-label multimodal classification task, they tested the performance of the ensemble of multimodal transformers e.g., uniter, visualbert, and lxmert alongside unimodal transformers e.g., bert, and roberta. the winning team on subtask 3 @xcite experimented with the ensemble of fine-tuned deberta and resnet, deberta and butd, and ernie-vil models to address the challenge of leveraging features from different data modalities. in this paper, we demonstrate our proposed architecture to address the challenges of subtask 1 (multi-label hierarchical text classification) and subtask 2(b) (multi-modal binary classification) of semeval 2024 task 4. for subtask 1, we utilize a fine-tuned language-agnostic bert sentence embedding (labse) model to extract effective contextual features of meme texts. next, we utilize an ensemble of vision transformer and xlm-roberta models to address the challenge of multilingual and multi-modal characteristics of subtask 2(b). the remaining part of the manuscript is outlined as follows: the pictorial description of our proposed methods for both tasks is articulated in section 2. section 3 presents the experimental setup, result, and evaluation. we conclude this manuscript with some future research directions in section 4. ## conclusion and future works in this manuscript, we presented our proposed frameworks to address the challenge of semeval 2024 task 4 subtasks 1 and 2(b). we employed the labse model to address the multilingual characteristics of subtask 1 whereas vision transformer and xlm-roberta models were employed to address the multi-modal and multilingual characteristics of subtask 2(b). both of our methods showed competitive performance over other participant's systems. in the future, we aspire to explore the effectiveness of different multimodal transformer models' performance on this task. we also have a plan to exploit the external knowledge for a better understanding of memes for this task.