whatdoyoumeme at S em E val-2024 Task 4: Hierarchical-Label-Aware Persuasion Detection using Translated Texts (2024)

Abstract: In this paper, we detail the methodology of team whatdoyoumeme for the SemEval 2024 Task on Multilingual Persuasion Detection in Memes. We integrate hierarchical label information to refine detection capabilities, and employ a cross-lingual approach, utilizing translation to adapt the model to Macedonian, Arabic, and Bulgarian. Our methodology encompasses both the analysis of meme content and extending labels to include hierarchical structure. The effectiveness of the approach is demonstrated through improved model performance in multilingual contexts, highlighting the utility of translation-based methods and hierarchy-aware learning, over traditional baselines.

## introduction persuasion techniques in politics have a significant impact on democratic processes, which was particularly evident in contexts such as the 2020 us elections, where cognitive dissonance and media messages played a crucial role in influencing voter behaviour and attitudes @xcite . these techniques, which utilise psychological insights, align people's attitudes with their actions and thus influence political affiliations and opinions. the interplay of crises -pandemic, economic downturn, protests against racial justice, and debates over electoral legitimacy -has further highlighted the impact of persuasive narratives on public perception and democratic resilience @xcite . this complicated relationship underscores the crucial role of persuasion in political discourse and its potential to shape democratic outcomes at crucial historical moments in society. manually recognizing persuasion in textual content is increasingly challenging due to the vast amount of information generated daily and the nuanced nature of persuasion techniques. efforts in this area have expanded to include the development of collaborative tasks @xcite aimed at recognizing persuasion across languages and levels of hierarchy, reflecting the global and complex nature of persuasive communication in digital spaces. in the past, researchers have used statistical text analysis methods that focused on lexical and syntactic features to identify patterns and markers of persuasive language @xcite . while these approaches provided basic insights, they were often not deep enough to fully capture the subtleties of human language and persuasion. the detection of persuasion in texts has shifted from statistical text analysis to the use of large language models (llms), such as bert @xcite and gpt @xcite . these models use deep learning to understand the context, semantics and complex interplay of language elements, providing more effective means of recognizing persuasive tactics in text. the collective effort in data collection and the joint tasks have contributed significantly to belief detection, with initiatives such as the semeval joint tasks fostering community-wide collaboration. the nlp4if-2019 shared task @xcite was another example of the collective effort to refine detection methods through standardised tasks. the task was divided into two parts: the identification of propagandistic text fragments and their specific techniques at fragment level and a binary classification at sentence level to recognise sentences containing propaganda. the joint task attracted a large participation and showed that most of the systems were able to significantly outperform the established baselines. @xcite found that for some propaganda techniques, it is not enough to look at just one sentence to make an accurate prediction (e.g. repetition) and therefore the whole article needs to be included as context @xcite presented a novel method for detecting propaganda at the level of fragments in news articles that goes be-yond traditional document-level detection. their method addressed the need for more nuanced and explainable analysis by manually annotating news articles with specific propaganda techniques and developing a multi-granularity neural network model that outperformed bert-based baselines. @xcite showed that cross-lingual and multitask training combined with an external balanced dataset can improve genre recognition and framing, on a recently proposed task by @xcite . recent studies have shown that translating texts from low-resource languages to a high-resource language, such as english, improves performance of end-to-end approaches on tasks such as classification @xcite and document similarity @xcite . @xcite show that cross-lingual validation can lead to improvement on classification performance for some tasks. some earlier works also confirm this to be true for non transformer architectures @xcite . the rest of this article is organised as follows: section 2 describes the task, while section 3 presents the proposed method and the results. finally, the conclusion and proposed future work in section 4. ## task description semeval-2023 task 4 @xcite focuses on multilingual detection and classification of persuation techniques in memes. it is composed of three subtasks. subtask 1 was a multi-label text classification task. the text was extracted from the image data that contained the original meme. although the text contains less information than original image, the annotation procedure accounted for this and allows for differences between labels in the text-only data and image data that provides additional context. subtask 2a was a multimodal extension of the subtask 1 by providing both a text and the image data. it is also a multi-label classification task, with the labels annotated based on both text and image data. subtask 2b was also a multimodal task with the same inputs as subtask 2a, but the task was a simpler binary classification task to detect if any persuasion technique is used. although only english dataset was provided for training, the evaluation was additionally done on three surprise test datasets in bulgarian, macedonian, and arabic. we further focus only on the subtask 1 and its data as this was the only task we participated in. ## methodology and results pre-trained language models, such as bert @xcite and its variants, have shown remarkable performance across many nlp tasks. we evaluate several bert-like models and their performance on the task. we are particularly interested in the impact of the hierarchy on model performances. we describe three approaches to understand the role of hierarchy information in the task. first, we establish a baseline using bert and mbart @xcite models without using any hierarchy information and grid search to tune the hyperparameters. we explore different tokenization strategies and evaluate the model with micro-f 1 . second, we evaluate approaches based on modifying the set of ground truth labels by extending the labels with ancestors to include hierarchy information and its influence on model performance. finally, we translate english train and validation data to macedonian, arabic, and bulgarian with the nllb-200 model @xcite and fine-tune our models for the multi-label classification task. we also compare the performance of the performance on translated data to zero-shot cross-lingual approaches using multilingual models and to translation of test sets. ## conclusion and future work in this paper, we describe the methods and models used by the whatdoyoumeme team in semeval 2024 subtask 1 to detect multilingual persuasion in memes. we combined two different approaches to solve this task: 1) machine translation, where we used the nllb model @xcite to translate articles from english into the target languages and vice versa, and 2) including hierarchy information, where we extend a set of provided labels with labels corresponding to the ancestors nodes from the hierarchy dag. we find that with the two proposed strategies, we can outperform both traditional encoder and decoder models, which emphasizes the importance of translation for downstream cross-lingual tasks. in the future, we would like to extend our work in a few different directions. we would like to explore ensemble modelling techniques by building separate models for each belief category and using their joint predictions to improve overall performance. in addition, we would like to investigate the effects of translation quality and model size on the performance of this task.