S heffield V era AI at S em E val-2024 Task 4: Prompting and fine-tuning a Large Vision-Language Model for Binary Classification of Persuasion Techniques in Memes (2024)

Abstract: This paper describes our approach for SemEval-2024 Task 4: Multilingual Detection of Persuasion Techniques in Memes. Specifically, we concentrate on Subtask 2b, a binary classification challenge that entails categorizing memes as either “propagandistic” or “non-propagandistic”. To address this task, we utilized the large multimodal pretrained model, LLaVa. We explored various prompting strategies and fine-tuning methods, and observed that the model, when not fine-tuned but provided with a few-shot learning examples, achieved the best performance. Additionally, we enhanced the model’s multilingual capabilities by integrating a machine translation model. Our system secured the 2nd place in the Arabic language category.

## introduction research of online misinformation is growing @xcite as fake news and propagandistic content spreads further and replaces more real news on social media, detrimentally impacting society, including loss of lives, loss of health and economic loss @xcite . a common online propaganda format is a meme, where text and image(s) are combined to share a message, often political @xcite . this paper describes sheffieldver-aai's approach for semeval 2024 task 4 subtask 2b, involving detecting the presence of persuasion technique(s) within memes, a binary visual/textual classification task @xcite . the previous research on this task including @xcite @xcite used nonautoregressive encoder representation techniques, using models such as bert @xcite or roberta @xcite , fused with a vision representation model such as resnet @xcite . unlike the previous research, we experimented with a different approach to use and train an autoregressive vision-language model that receives image and text as input, outputting text only. prompting the model with a meme and expecting the model to generate a classification output. specifically, we use the llava-1.5 model @xcite , which directly projects an image encoding into tokens computed as text tokens by an llm. this technique allows us to utilise the llms' "knowledge" of persuasion techniques they have learnt through massive pre-training and explained outputs through prompting, improving the model's interoperability and error analysis. ## background previous semeval tasks have looked at this problem of online misinformation/persuasion/propaganda: this theme of persuasion technique detection is prominent in recent years of semeval, with the most similar task being semeval 2021 task 6 subtask 3, which involved visual and textual persuasion techniques in memes. ## system overview our system follows these steps: ## conclusion we have presented our system for semeval 2024 task 4 subtask 2b. we prompted and fine-tuned an auto-regressive large visual language model and showed that llava can be used for non-english persuasion technique detection in memes through improved prompting and machine translation, with our best ranking being 2nd on the arabic leaderboard. we experimented with different techniques for prompting to discover which produced the bestperforming output. we also analysed the type of errors llava can produce when classifying memes, showing how this model can easily be prompted for explainability. further work is required to improve training lvlms for this task, as we could not see improvements through fine-tuning.