A lex UNLP - MZ at A r AIE val Shared Task: Contrastive Learning, LLM Features Extraction and Multi-Objective Optimization for A rabic Multi-Modal Meme Propaganda Detection (2024)

Abstract: The rise of memes as a tool for spreading propaganda presents a significant challenge in the current digital environment. In this paper, we outline our work for the ArAIEval Shared Task2 in ArabicNLP 2024. This study introduces a method for identifying propaganda in Arabic memes using a multimodal system that combines textual and visual indicators to enhance the result. Our approach achieves the first place in text classification with Macro-F1 of 78.69%, the third place in image classification with Macro-F1 of 65.92%, and the first place in multimodal classification with Macro-F1 of 80.51%

## introduction in today's digital era, memes have emerged as a potent instrument for disseminating propaganda. it is a method employed to influence individuals to adopt specific thoughts or behaviors. while this practice has existed for an extended period, social media has become a prevalent platform for its dissemination, especially considering its role as a primary news source. those who engage in social media propaganda may have intentions to trick or control people. memes combining humor or visuals with specific messages, can bypass traditional filters and influence opinions surprisingly well (nieubuurt, 2020) @xcite . their effectiveness stems from their ability to be understood across languages and cultures. this makes them ideal for political groups, extremists, and even advertisers, who can leverage their virality to spread ideas or promote agendas. memes can be persuasive, using humor, sarcasm, or fear to either reinforce existing beliefs or stir up conflict. spotting propaganda in memes is tough. machine learning models need tons of knowledge across many areas (politics, history, etc.) to sort real content from propaganda. ## related work traditional methods for detecting propaganda have focused on analyzing textual content. researchers have identified recurring linguistic features in propaganda @xcite ) @xcite . although detecting propaganda in text is common, truly understanding bias in memes requires analyzing both text and image elements. a recent study explored ways to label memes to identify political bias, showing how important it is to consider both the image, text @xcite . furthermore, the survey in @xcite ) and arabic shared tasks specifically for propaganda detection in @xcite , @xcite and @xcite ) highlight the growing importance of addressing this issue in a multilmodal context. ## data the data set for meme propaganda detection collected from different social media (e.g., facebook, twitter, instagram and pinterest). it comprises a collection of images, captions (mesme text content) and the corresponding labels as shown in figure 1 stored in json file. the data is split into training, validation, and testing sets. the statistics of the training and validation sets are given in additionally, we processed our data by eliminating english words, stripping diacritics, and removing any html elements if present. furthermore, we incorporated the extracted features from the language model models (llms), as illustrated in figure 3 , into both our training and validation datasets in binary form (0 and 1). ## system our system relies on two encoders components for the two modalities and a fusion module along with three objective functions incorporating contrastive loss, as depicted in figure 2 . we incorporate a range of techniques, with a strong focus on the multimodal approach, which yields the best performance. ## results in our submission to the official test set, we used bloomz-1b1 for text classification which achieves the first place with a macro-f1 of 78.69%, resnext101_32x8d, for image classification which achieves the third place with a macro-f1 of 65.92%, and a combination of these models with attention fusion for the multimodal task and we achieved the first place with a macro-f1 of 80.51%. to assess the impact of our system design on the bloomz-1b1 model with a resnext-101_32x backbones, we compared the performance of three training approaches on the validation set using extracted features, contrastive loss, and employing both text and image losses. our findings demonstrate that the best results are achieved by combining all three methods as shown in ## discussion our system that analyzes both text and images (multi-modal) significantly beats analyzing just text or images alone (uni-modal). this shows the value of combining different data types for better understanding. features extracted from large language models (llms) are key factors, as they provide a strong foundation for the model understanding of the data. in future work, we aim to leverage ensemble methods by combining our most effective models. we will also explore incorporating specialized propaganda detection models and leverage the power of large vision language models for multimodal analysis. by extracting richer features from the data we anticipate significant improvements in propaganda detection accuracy. figure 4 reveals some weaknesses in the model for predicting the correct class. it demonstrates the limitations of the current model ability to accurately identify public figures within images, also it indicates that mentioning countries or names may mislead the model. ## conclusion this research investigated the role of memes as propaganda tools within the ever-expanding landscape of social media. we developed a system capable of identifying propaganda in memes by analyzing their multimodal elements text, image, and the combination of both. our findings highlight the critical role of textual content in propaganda detection within memes. analyzing the language proved to be a key factor, as words often carry more explicit messaging compared to visuals alone. furthermore, the system demonstrated the effectiveness of utilizing multiple objective optimization techniques. this, combined with the inclusion of strategically extracted features, enhanced the models overall accuracy in identifying propagandistic memes.