MODOS at A r AIE val Shared Task: Multimodal Propagandistic Memes Classification Using Weighted SAM , CLIP and A rabian GPT (2024)

Abstract: Arabic social media platforms are increasingly using propaganda to deceive or influence people. This propaganda is often spread through multimodal content, such as memes. While substantial research has addressed the automatic detection of propaganda in English content, this paper presents the MODOS team’s participation in the Arabic Multimodal Propagandistic Memes Classification shared task. Our system deploys the Segment Anything Model (SAM) and CLIP for image representation and ARABIAN-GPT embeddings for text. Then, we employ LSTM encoders followed by a weighted fusion strategy to perform binary classification. Our system achieved competitive performance in distinguishing between propagandistic and non-propagandistic memes, scored 0.7290 macro F1, and ranked 6th among the participants.

## introduction the term "meme" was introduced by richard dawkins in his renowned publication, the selfish gene @xcite . dawkins derived the term from the greek word "mimeme," modifying it to denote a concept signifying "something which is imitated." in the era of social media, memes have frequently become a common method of communication, often used to spread messages with varied degrees of propaganda @xcite . propaganda is defined as deliberate expressions to influence opinions or actions, often using rhetorical and psychological devices.3 understanding the dynamics of online conversation and preventing the spread of false information depends on the ability to identify propagandistic content. consequently, researchers in natural language processing (nlp) are increasingly interested in automatically detecting the application of propaganda techniques in text, images, and multimodal information @xcite 1 https://araieval.gitlab.io/ @xcite . prior work in arabic has focused primarily on detecting propaganda in a social media text, particularly tweets @xcite @xcite . several shared tasks have been launched to address the detection of propaganda in arabic tweets and the identification of persuasion techniques and disinformation in arabic text. foot_0 , foot_1 these tasks have attracted many researchers @xcite @xcite @xcite @xcite , leading to significant advances in recognizing and extracting linguistic cues that indicate the use of propaganda techniques. in these tasks, researchers have achieved good results in detecting propaganda in arabic text by identifying the propaganda techniques used, which often include playing on the audience's emotions (e.g., appealing to fear, using loaded language, etc.) @xcite . despite significant progress conducted in multimodal learning for arabic, which focused on tasks such as sentiment analysis, emotion recognition, summarization, and more, rather than propaganda detection. for instance, haouhat et al. @xcite developed an arabic multimodal dataset for sentiment analysis that comprises transcripts, voice recordings and videos, enhancing the ability to capture nuanced content. these efforts highlight the potential of multimodal approaches in improving understanding across different domains. however, the specific challenge of detecting propaganda in multimodal formats remains less explored, indicating a critical gap in this area. to address the gap in arabic multimodal propaganda detection research, hasanain et al. @xcite introduced a subtask to the araieval shared task at arabicnlp 2024. 4 previous research predominantly focused on unimodal textual detection, leaving a notable gap in understanding the propagandistic content presented in multimodal formats. this subtask specifically focused on the classification of memes, representing a significant advancement in the field. our team actively participated in this initiative, specifically targeting the second subtask, subtask2c, where we aimed to contribute to the detection of propagandistic content in multimodal memes. despite the complexity of the task, our system demonstrated promising results, securing the sixth position in the competition. our contribution involved the development of a comprehensive approach that integrates various techniques, including image segmentation, feature extraction, multimodal weighted fusion, and classification. firoj et al. @xcite focus on the critical problem of identifying and labeling propagandistic spans within the textual content. the authors explore the potential of large language models (llms) like gpt-4 to function as both general and expert annotators for this task. the study looks at how well gpt-4 annotates spans and if it can take the place of consolidators during the annotation process. notably, the study makes significant contributions by releasing annotations from multiple human annotators and gpt-4, aiming to benefit the research community and facilitate further advancements in propaganda detection and analysis. in @xcite , the authors meticulously construct arpro, the largest propaganda dataset in arabic, comprising 8k paragraphs from newspaper articles labeled at the text span level across 23 propagandistic techniques. additionally, the study delves into the potential of large language models (llms) for fine-grained propaganda detection from text. however, results reveal that while gpt-4 performs acceptably in classifying paragraphs as propagandistic or not. ## task definition the araieval shared task is part of arabicnlp 2024 conference, comprises two main challenges focused on the automated identification of propagandistic content within mainstream and social media. the first one deal with unimodal data by detecting propagandistic textual spans with persuasion techniques and second challenge focus on distinguishing between propagandistic and nonpropagandistic multimodal memes. participants are tasked with analyzing tweets, news articles, and memes to identify misleading information, aiding fact-checkers, journalists, and the general public in combating disinformation. ## data in this work, we utilize the dataset released for the araieval shared task @xcite . the dataset consisted of a diverse collection of memes labeled as propaganda or non-propaganda. table1 ## system our methodology consists of several key steps: image segmentation, feature extraction, multimodal fusion, and classification. a github repository for our project is provided. 5 figure 1 provides an overview of our approach. ## results and discussion we evaluated our model against the baseline models provided by shared task organizers: ngram, random, imgbert, and majority. the metrics used are precision, recall, f1 score, and accuracy. the performance of the models is presented in our model demonstrates superior performance across all metrics, achieving an accuracy of 0.8040 and an f1 score of 0.7348 on the test set, ranking sixth in the leaderboard, the best performance with an f1 score of 0.8051 achieved by the alexunlp-mz team. our finding indicate the effectiveness of our approach in correctly classifying the memes, with a good balance between precision and recall, efficiently handling both false positives and false negatives. however, there are observed limitations in model generalization due to not employing several propaganda techniques (e.g., name-calling, virtue words, deification, testimonial, fear, etc.) that could enhance the detection. furthermore, the unbalanced class distribution in the dataset poses a challenge, potentially biasing the model towards the majority class and affecting its generalization capability. ## conclusion in this study, we presented our approach to the multimodal propagandistic memes classification subtask at the araieval shared task. by leveraging a combination of image segmentation, feature extraction, multimodal fusion, and fusing weighted strategy, we captured rich information from both modalities. we developed a model capable of effectively identifying propagandistic content in memes. our experimental results demonstrate that the proposed model outperforms several baseline methods, achieving higher accuracy and f1 scores. overall, our findings indicate good participation. however, there is room for improvement, particularly in addressing overfitting and enhancing generalization capabilities. future work will explore advanced multimodal fusion techniques, incorporate additional contextual information, and utilize larger and more diverse datasets.