M eme M ind at A r AIE val Shared Task: Generative Augmentation and Feature Fusion for Multimodal Propaganda Detection in A rabic Memes through Advanced Language and Vision Models (2024)

Abstract: Detecting propaganda in multimodal content, such as memes, is crucial for combating disinformation on social media. This paper presents a novel approach for the ArAIEval 2024 shared Task 2 on Multimodal Propagandistic Memes Classification, involving text, image, and multimodal classification of Arabic memes. For text classification (Task 2A), we fine-tune state-of-the-art Arabic language models and use ChatGPT4-generated synthetic text for data augmentation. For image classification (Task 2B), we fine-tune ResNet18, EfficientFormerV2, and ConvNeXt-tiny architectures with DALL-E-2-generated synthetic images. For multimodal classification (Task 2C), we combine ConvNeXt-tiny and BERT architectures in a fusion layer to enhance binary classification. Our results show significant performance improvements with data augmentation for text and image classification models and with the fusion layer for multimodal classification. We highlight challenges and opportunities for future research in multimodal propaganda detection in Arabic content, emphasizing the need for robust and adaptable models to combat disinformation.

## introduction the rapid proliferation of social media has led to an unprecedented spread of information, including propagandistic content designed to manipulate public opinion @xcite . propaganda employs various techniques to influence the emotions, attitudes, and actions of its target audience @xcite . the rise of multimodal content, such as memes, has added a new dimension to the challenge of detecting propaganda online @xcite @xcite . memes, which combine text and images, have become a popular medium for spreading propaganda on social media platforms @xcite . detecting propaganda in memes is a complex task that requires understanding both textual and visual content. previous work on multimodal propaganda detection has primarily focused on englishlanguage content @xcite . however, research on detecting propaganda in arabic memes is limited despite the increasing prevalence of such content on arabic-language social media @xcite . in this paper, we present a novel framework to tackle the araieval 2024 shared task, focusing specifically on task 2: multimodal propagandistic memes classification. for task 2a, we finetune various pre-trained arabic language models, including arabertv1, arabertv2, marbert, camelbert, aragpt-2, qarib-bert, electra-gpt, and arabic-bert @xcite @xcite @xcite . we augment these models with a classification layer and employ class weighting to handle the imbalanced dataset. we further enhance the models by fine-tuning on a combination of original and synthetic text data generated using chatgpt4. for task 2b, we fine-tuned pre-trained resnet18 @xcite , convnext-tiny @xcite , and efficientnetv2 @xcite ) models for meme image classification. these models were augmented with additional synthetic images generated by dall-e-2 to improve binary classification performance. for task 2c, we combined the arabert model trained in task 2a with the convnext-tiny model trained in task 2b to jointly extract features from both text and images. these features were concatenated and fed into a fusion layer, followed by a binary classification layer. the main contributions of our work are the following: ## related work the araieval shared task, organized by @xcite , focuses on detecting propagandistic techniques in unimodal and multimodal arabic content. this shared task builds upon the previous edition, araieval 2023 @xcite , which targeted persuasion techniques and disinformation detection in arabic text. the organizers have also explored the use of large language models for propaganda span annotation @xcite and investigated the ability of gpt-4 to identify propaganda spans in news articles @xcite . the araieval shared task is a continuation of the efforts made in the wanlp 2022 shared task on propaganda detection in arabic @xcite . the spread of propaganda on social media platforms is not limited to english-language content. arabiclanguage social media has also seen a rise in the dissemination of propagandistic content, including memes @xcite . @xcite presented a corpus of arabic propaganda posts on social media, highlighting the need for research on detecting propaganda in arabic content. @xcite conducted a survey on author profiling, deception, and irony detection for the arabic language, discussing various aspects of arabic natural language processing that could be relevant to the task of identifying propaganda in arabic memes. @xcite focused on detecting deceptive tweets in arabic for cyber-security purposes, demonstrating the importance of identifying manipulative content on arabic social media platforms. @xcite presented a large multi-dialect twitter corpus for arabic, which could be useful for understanding the lin-guistic characteristics of arabic social media content and developing models for detecting propaganda in memes. while not directly related to propaganda detection, other studies have demonstrated the application of text analysis techniques to arabic content in various social contexts. for example, @xcite presented a large-scale arabic social media corpus for detecting youth depression, highlighting the importance of analyzing arabic social media content for mental health monitoring purposes. in this work, we alleviated the class imbalance through the automatic generation of synthetic texts by chat-gpt, and we improved multimodal classification by using a fusion layer concatenating image and text features to feed a dense neural layer for classification. ## data the araieval 2024 shared task consists of two main tasks focusing on the detection of propagandistic content in arabic media. task 1, unimodal (text) propagandistic technique detection, is a sequence tagging task that involves identifying propaganda techniques and their corresponding spans within a given text snippet. the input can be either a news paragraph or a tweet. the goal is to detect the specific propaganda techniques used in the text and the exact spans where each technique appears. task 2, multimodal propagandistic memes classification, focuses on the classification of propagandistic content in multimodal memes. this task is further divided into three tasks. task 2a, textbased propaganda classification, aims to classify whether the text extracted from a meme is propagandistic or not. task 2b, image-based propaganda classification, involves determining whether the content of a meme (an image with overlaid text) is propagandistic. task 2c, multimodal propaganda classification, requires detecting whether the content of a meme is propagandistic based on both the extracted text and the meme image itself. our focus is solely on task 2, specifically the multimodal propagandistic memes classification task. the dataset @xcite consists of 2143, 312, and 607 pairs of text and images for the training, validation, and test splits, respectively. the training set is imbalanced, with 603 pairs in the propaganda class and 1540 pairs in the non-propaganda class. ## methodology in this section, we present the various techniques used for developing the final model for the task. the objective is to classify memes as either propagandistic or non-propagandistic. the task involves multimodal classification of propagandistic memes, subdivided into three distinct tasks: task 2a, task 2b, and task 2c (description in sec 3). each of these tasks entails a binary classification objective. task 2a: we fine-tuned various pre-trained models trained on arabic content, including arabertv1, arabertv2, marbert, camel-bert, aragpt-2, qarib-bert, electra-gpt, and arabic-bert @xcite @xcite @xcite . to accomplish this, we augmented these models by attaching a classification layer with random weights. given the imbalanced nature of the dataset, we employed a class weighting scheme during the fine-tuning process. initially, we fine-tuned the pre-trained model using the original dataset. subsequently, we optimized the model by adjusting the dropout rate. further enhancement was achieved by fine-tuning the last layer of the model using both synthetic and original text, resulting in improved performance over the original dataset (see ## results and discussion the official metric used for each task is the macro f1-score. furthermore, in task 2c, our team's ensemble approach garnered significant recognition, achieving a macro f1-score of 0.7972 and securing a prominent position on the leaderboard. these results underscore the effectiveness of our approach in addressing the challenges posed by propagandistic content detection across diverse modalities. ## conclusion we presented our contribution to the araieval shared task, utilizing advanced language and vision models. our approach includes augmentation through generated texts and layers to fuse multimodal features. this initiative and our proposed solutions highlight the increasing significance of developing effective methods for analyzing arabic social media content, especially in detecting propaganda and manipulative content. by leveraging state-of-the-art language models and multimodal architectures, our goal is to advance propaganda detection in arabic memes and the broader field of arabic natural language processing.