BERT astic at S em E val-2024 Task 4: State-of-the-Art Multilingual Propaganda Detection in Memes via Zero-Shot Learning with Vision-Language Models (2024)

Abstract: Analyzing propagandistic memes in a multilingual, multimodal dataset is a challenging problem due to the inherent complexity of memes’ multimodal content, which combines images, text, and often, nuanced context. In this paper, we use a VLM in a zero-shot approach to detect propagandistic memes and achieve a state-of-the-art average macro F1 of 66.7% over all languages. Notably, we outperform other systems on North Macedonian memes, and obtain competitive results on Bulgarian and Arabic memes. We also present our early fusion approach for identifying persuasion techniques in memes in a hierarchical multilabel classification setting. This approach outperforms all other approaches in average hierarchical precision with an average score of 77.66%. The systems presented contribute to the evolving field of research on the detection of persuasion techniques in multimodal datasets by offering insights that could be of use in the development of more effective tools for combating online propaganda.

## introduction propaganda is an ancient technique that has existed for thousands of years 1 . the way propaganda is understood today was formalized between 1937 and 1942 by the institute of propaganda analysis through a series of publications @xcite , @xcite , @xcite ). britannica defines propaganda as the "dissemination of information-facts, arguments, rumours, half-truths, or lies-to influence public opinion." 1 . propaganda can be beneficial when it unites people behind a noble or beneficial cause. it can also be harmful if it leads to tensions, destabilization, and the death of millions. in our digitally mediated world, transmitting (dis)information 1 https://www.britannica.com/topic/ propaganda to millions of people occurs in seconds. hence, the adverse effects of propaganda are accelerated and amplified. propaganda has been used to influence public opinion on brexit @xcite , us elections @xcite , and the ukraine crisis @xcite . thus, it is easy to see the damaging effects propaganda has already caused and continues to inflict. propaganda takes many forms. it could be broadcast on television @xcite , spread through coordinated communities on social media @xcite , transmitted across national borders through loudspeakers @xcite , disseminated via news articles @xcite , published on blogs @xcite , or could even exist on postage stamps @xcite . more recently, memes have become powerful tools for the dissemination of political messages. the visual and textual simplicity of memes, combined with their viral nature, allows them to be rapidly consumed and shared across social media platforms, reaching vast audiences with minimal effort. this level of accessibility makes memes an attractive medium for propagandists seeking to subtly influence public opinion, disseminate misinformation, and polarize communities. consequently, there has been an increased need for and interest in propaganda identification in the research community. the most difficult challenge is that propaganda is often based on kernels of truth and is presented in a misleading way, making it seem genuine. hence, training a model to detect propaganda is challenging, given the subtlety in how propaganda masquerades as an ordinary text or an innocent, funny meme. moreover, interpreting any such model's results could also be problematic. with memes, the challenge, is further exacerbated due to the inherent complexity of memes' multimodal content, which combines images, text, and often, nuanced context, making the detection of propaganda all the more challenging. furthermore, the language used in memes is characteristically concise, often consisting of mere sentence fragments or a few keywords. consequently, developing systems that consider only the textual content in isolation from the accompanying image presents a significant challenge. the model explainability challenge has been tackled by @xcite for texts and by @xcite for memes via the introduction of fine-grained propaganda detection tasks and datasets. the tasks required identifying the propaganda technique(s) out of over eighteen techniques in a multi-label classification formulation. this fine level of granularity increases interpretability of propaganda detection models. however, the subtlety challenge is still prevalent. in addition to the subtlety challenge, propaganda detection research is scarce on multimodal datasets in general, and on memes in particular, correlating with a scarcity of datasets. moreover, the majority of research and datasets are monolingual and consider mainly the english language. therefore, the difficulty of tackling propaganda in memes also extends to include the scarcity of multilingual meme datasets. there are recent efforts to address this challenge, which are currently spearheaded by the shared task on multilingual detection of persuasion techniques in memes @xcite described in detail in 2.1. in this paper, we capitalize on recent advances in vision language models (vlms) @xcite and pretrained language models (plms) @xcite , and highlight the following contributions: ## results we show the results on binary and multilabel classification tasks in tables 2 and 3, respectively. for binary classification, our system ranks first on north macedonian propaganda detection task, and achieves competitive results ranking third on arabic and bulgarian. collectively, we achieve the top rank in average f1 across all four languages. as for multilabel classification, our system suffers from low recall but compensates for that by a very high precision performance. our system achieves the highest precision on bulgarian, the second highest on arabic, english, and north macedonian, and the top precision score on average across all four languages. this means our system is very conservative in that it only makes predictions it is highly confident of. we also experimented with other models for both the binary and multilabel tasks as shown in tables 4 and 5, respectively. in the binary setting, we train several models on embeddings obtained using a clip-vit-b-32 model, but the zero-shot vlm approach performs better in comparison. for the multilabel task, we try several approaches. we attach a classification head and fine-tune an mpnet model on the meme descriptions that we obtained as described earlier in figure 1 . however, this did not yield a high performance. we also fine-tune deberta-v3-large @xcite and xlm-r-large on different subsets of available text (i.e., meme text and the vlm descriptions). out of all combinations, we observe that deberta performed best when it was fine-tuned on vlm descriptions only. this is likely due to the fact the meme texts are often short, incoherent and do not form complete sentences; hence, we deduce they may contaminate the much more coherent descriptions generated by the vlm. moreover, we also observe that we cannot achieve competitive results using only the text modality. in all experiments involving transformers, we fine-tune with early stopping with a patience of three epochs, use a batch size of eight, a learning rate of 5e -5, and we accumulate the gradients for eight steps making our effective batch size 64. we also use a binary-cross entropy loss and set a classification threshold of 0.5. ## conclusion and future work in this study, we introduced a state-of-the-art multilingual propaganda detection in memes using zeroshot learning with vlms. our approach uniquely addressed the complexities of multimodal content in memes, merging visual and textual cues in a manner that comprehensively understands and identifies propagandistic content across languages. achieving an average macro f1 score of 66.7% across all assessed languages, our system demonstrated high performance over existing methods, particularly excelling in north macedonian memes and showing competitive performance in bulgarian and arabic in the binary setting. in addition, our early fusion technique for identifying persuasion techniques in memes within a hierarchical multilabel classification setting outperformed all other approaches with an average hierarchical precision score of 77.66%. looking forward, our research opens several directions for further exploration and improvement. first, the exploration of advanced fusion techniques that could more intricately combine the strengths of textual and visual analyses may yield even higher accuracies in propaganda detection. additionally, the adaptability and performance of our model in detecting subtler forms of propaganda and across a broader spectrum of languages present an exciting challenge, especially considering the highly nuanced contextual nature of meme content and its cultural intricacies.