ASOS at A r AIE val Shared Task: Integrating Text and Image Embeddings for Multimodal Propaganda Detection in A rabic Memes (2024)

Abstract: This paper describes our participation in the ArAIEval Shared Task 2024, focusing on Task 2C, which challenges participants to detect propagandistic elements in multimodal Arabic memes. The challenge involves analyzing both the textual and visual components of memes to identify underlying propagandistic messages. Our approach integrates the capabilities of MARBERT and ResNet50, top-performing pre-trained models for text and image processing, respectively. Our system architecture combines these models through a fusion layer that integrates and processes the extracted features, creating a comprehensive representation that is more effective in detecting nuanced propaganda. Our proposed system achieved significant success, placing second with an F1 score of 0.7987.

## introduction with the rise of social media platforms and the growing number of users worldwide, people have found a more accessible way to express themselves. they have turned to humour and wit, creating jokes and memes to succinctly convey thoughts, emotions, and behaviours. memes are compositions of images and text, merging them to collectively communicate ideas. they have become a prevalent form of communication on various social media platforms such as twitter and facebook, where content often combines text with images or videos. with the widespread use of memes on social media platforms across all age groups, it is important to categorize and understand the sentiment and purpose behind them. this involves determining whether memes convey positive or negative implications and whether they are used to spread hatred, ridicule, and misinformation (dupuis and williams, 2019). it's also crucial to identify whether they contain hateful or nonhateful content @xcite . from another perspective, some focus on processing text only that extracted from memes @xcite , while others view it as a multi-modal task @xcite . despite the diversity of tasks, the ultimate goal remains the same: understanding the impact of memes on society. our team participated in the araieval shared task, specifically focusing on task 2c, which involves analyzing multimodal content @xcite . this content includes textual data extracted from memes and their corresponding images to identify propagandistic elements. we developed a methodology that combines the best-performing text and image pre-trained models, using marbert for textual analysis and resnet50 for visual analysis. ## related work recent research has focused on classifying memes and extracting features using advanced computational methods. for example, one study @xcite addresses the classification of multimodal offensive memes by employing natural language inference (nli) techniques and fine-tuning roberta models to refine the task into a unimodal offensive text classification challenge. another significant study @xcite proposes a multi-label multimodal approach for detecting propaganda techniques in memes. this method involves annotating a corpus of 950 memes with 22 different propaganda techniques found in text, images, or both, utilizing a multilabel setup for the annotations. a different work @xcite proposes a feature extraction method for multimodal meme classification using deep learning approaches. they aim to classify the sentiment in memes using multi-embedding, residual network, and bimodal autoencoder techniques, highlighting the importance of sentiment analysis for monitoring and managing negative content online. addition-ally, a study @xcite introduces a meme-integrated deep learning (midl) framework for meme classification and analysis, employing bert and resnet architectures. they discuss the limitations of traditional methods and the potential of the proposed framework to advance research in online culture analysis. furthermore, surveys address challenges in meme classification and understanding harmful memes, discussing a typology of harmful memes and the lack of suitable datasets for studying certain types, such as those featuring selfharm and extremism. they emphasize the need for cutting-edge solutions and fine-grained analysis in this area. works @xcite , and ( afridi @xcite collectively aim to enhance meme classification, sentiment analysis, and understanding of harmful content on social media platforms, contributing to research in online culture analysis and combating misinformation. ## methodology this section outlines our methodology for detecting propagandistic content in arabic memes. it details the dataset, pre-trained models for text and image processing, system architecture, and training procedures, collectively enabling our system to analyze multimodal data effectively. ## conclusion our participation in the araieval shared task 2024 led to the development of a robust system that integrates advanced text and image processing techniques to detect propagandistic content in arabic memes. we used marbert for textual analysis and maxvit for visual analysis. our approach placed second with an f1-socre of 0.798, showing its effectiveness in handling complex multimodal data. these results validate our methodology and highlight the importance of combining different modalities to improve content classification systems. in the future, we plan to refine these techniques, explore additional model integrations, and expand our dataset to enhance the system's performance and adaptability across various multimedia platforms.