EURECOM at S em E val-2024 Task 4: Hierarchical Loss and Model Ensembling in Detecting Persuasion Techniques (2024)

Abstract: This paper describes the submission of team EURECOM at SemEval-2024 Task 4: Multilingual Detection of Persuasion Techniques in Memes. We only tackled the first sub-task, consisting of detecting 20 named persuasion techniques in the textual content of memes. We trained multiple BERT-based models (BERT, RoBERTa, BERT pre-trained on harmful detection) using different losses (Cross Entropy, Binary Cross Entropy, Focal Loss and a custom-made hierarchical loss). The best results were obtained by leveraging the hierarchical nature of the data, by outputting ancestor classes and with a hierarchical loss. Our final submission consist of an ensembling of our top-3 best models for each persuasion techniques. We obtain hierarchical F1 scores of 0.655 (English), 0.345 (Bulgarian), 0.442 (North Macedonian) and 0.178 (Arabic) on the test set.

## introduction online misinformation is a complex research topic. it can appear in many shape and forms @xcite , within different contexts (political debates, news articles, social media posts, etc.). as memes generate high engagement on social media, they are also used for disinformation campaign by exploiting rhetoric persuasion. this year's "semeval-task4: multilingual detection of persuasion techniques in memes" task aims at detecting named persuasion techniques in memes. the full overview of the task and its sub-tasks are detailed in @xcite . as a brief summary, semeval-2024 task 4 consist of detecting persuasive techniques in memes. the task breaks down into 3 sub-tasks; sub-task 1 use the textual content of the meme to detect the persuasion techniques, sub-task 2a use the whole image and text to detect the persuasive techniques, while sub-task 2b only consist of binary detection. in sub-task 1, a total of 20 persuasion techniques are used. in this work, we only describe our solution to tackle this sub-task 1. our approach consists of an ensembling model of our top-3 best models for each persuasion techniques. in our experiments, we reached the best results leveraging the hierarchical nature of the data, with hierarchical loss, and outputting ancestor classes. our method can be reproduced using the code at https://github.com/d2klab/ semeval-2024-task-4 . ## system description in this section, we describe the system used in our submission. we also present approaches that were considered but not kept in our final submission. ## results we share our results on the dev set provided by the organisers in ## discussion model-wise, our best results were obtained using bert, roberta and bert-harme. we ultimately did not use any of albert, deberta and dis-tilbert models in our final submission as those were not in any top-3 best performing models of any persuasion techniques. the bert-harme models were the best-performing on the detection of 'slogans', 'appeal to authority', 'flag-waving', 'appeal to fear/prejudice', 'black-and-white fallacy/dictatorship', 'thought-terminating cliché', 'presenting irrelevant data (red herring)', 'glittering generalities (virtue)', 'doubt', 'logos', 'justification' and 'distraction' classes. roberta models were the best-performing for 'repetition', 'band-wagon', 'ethos'. we also noticed a slight performance increase by adding the 2021 dataset during training, which was not necessarily true when adding the ptc corpus. this is probably due to the fact that the ptc corpus is about news articles and not memes. our dataaugmentation experiments on round-translation did not improve the results at all, while the gpt-4-turbo augmentation experiments provided a very slight boost, but not for the augmented classes. the hierarchical nature of the task and the evaluation metrics were reflected in the results, as most of our best performing models are outputting 28 classes by including the ancestors and/or are trained with hierarchical loss (hl). however, best models at detecting 'causal-oversimplification' are using bce loss. we can see in ## conclusion in this paper, we describe the system team eure-com used for sub-task 1 at semeval-2024 task 4: multilingual detection of persuasion techniques in memes. we explore multiple bert-based models, training datasets, losses, data augmentation procedures, and training process. our final submission consists of an ensembling model that performs majority voting between our top-3 best performing models for each persuasive technique. we find that some pre-trained models on harmful meme data are competitive, and that incorporating hierarchical information in the training process, such as outputting the whole 28 classes (including the ancestors) or using a hierarchical loss significantly improves the results. we obtain a hierarchical f1 score of 0.675 on the dev set and 0.655 (english), 0.345 (bulgarian), 0.442 (north macedonian), 0.177 (arabic) on the test set.