IITK at S em E val-2024 Task 4: Hierarchical Embeddings for Detection of Persuasion Techniques in Memes (2024)

Abstract: Memes are one of the most popular types of content used in an online disinformation campaign. They are primarily effective on social media platforms since they can easily reach many users. Memes in a disinformation campaign achieve their goal of influencing the users through several rhetorical and psychological techniques, such as causal oversimplification, name-calling, and smear. The SemEval 2024 Task 4 Multilingual Detection of Persuasion Technique in Memes on identifying such techniques in the memes is divided across three sub-tasks: (1) Hierarchical multi-label classification using only textual content of the meme, (2) Hierarchical multi-label classification using both, textual and visual content of the meme and (3) Binary classification of whether the meme contains a persuasion technique or not using it’s textual and visual content. This paper proposes an ensemble of Class Definition Prediction (CDP) and hyperbolic embeddings-based approaches for this task. We enhance meme classification accuracy and comprehensiveness by integrating HypEmo’s hierarchical label embeddings (Chen et al., 2023) and a multi-task learning framework for emotion prediction. We achieve a hierarchical F1-score of 0.60, 0.67, and 0.48 on the respective sub-tasks.

## introduction memes are popular among people of all age groups today through different social media platforms @xcite . these memes help people know about the trends around them and can influence their decisions. memes are one of the popular modes for spreading disinformation among people (examples in figure 1 ), as studies have suggested that people tend to believe what they see frequently in such memes spread over the internet @xcite . as evidenced by research @xcite during the 2016 us presidential campaign, nefarious actors, including bots, cyborgs, and trolls, leveraged memes to evoke emotional reactions and propagate misleading narratives @xcite . in this respect, semeval-2024 task 4 @xcite focuses on predicting the persuasive technique (from the visual and textual content) used in a meme across four different languages: english, arabic, north macedonian and bulgarian. the task is divided into three sub-tasks: (1) hierarchical multi-label classification using only textual content of the meme, (2) hierarchical multi-label classification using both textual and visual content of the meme and (3) binary classification of whether the meme contains a persuasion technique or not using it's textual and visual content. the training data is provided for each sub-task but only in english. taxonomy of various persuasion techniques (figure 2 ) and their respective definitions are provided. to address sub-task 1, we employed a dual approach involving definition-based modeling for each class and hierarchical classification using hyperbolic embeddings, as proposed in @xcite . based on hyperbolic embeddings, the method facilitates a nuanced classification of persuasion techniques by leveraging hierarchical structures. the incorporation of definition-based modeling allows for a dataset-agnostic approach, enhancing the precision of classification without reliance on hierarchical structures. for sub-task 2, we augmented our methodology by integrating clip embeddings @xcite to capture essential features from memes' textual and visual components. this fusion of textual and visual information enables a more comprehensive analysis of meme content. in addressing sub-task 3, we adopted an ensemble approach, leveraging transfer learning from both the distilbert @xcite and clip embeddings @xcite . this ensemble technique enhances the robustness and effectiveness of our classification system by amalgamating insights from both pre-trained models. we release the code via github. foot_0 ## background the goal of propaganda is to enhance people's mindsets @xcite , especially at the time of elections, where the trends in the media influ-ence the votes of the people @xcite . propaganda uses psychological and rhetorical techniques to serve its purpose. such methods include using logical fallacies and appealing to the audience's emotions. logical fallacies are usually hard to spot since the argumentation, at first sight, might seem correct and objective. however, a careful analysis shows that the conclusion cannot be drawn from the premise without misusing logical rules @xcite . another set of techniques uses emotional language to induce the audience to agree with the speaker only based on the emotional bond that is being created, provoking the suspension of any rational analysis of the argumentation @xcite . corpora development has been instrumental in advancing deception detection methodologies. @xcite introduced the tshp-17 corpus, providing document-level annotation across four classes: trusted, satire, hoax, and propaganda. however, their study on the classification task revealed limitations in the generalizability of ngram-based approaches. building on this, @xcite contributed the qprop corpus, which specifically targeted propaganda detection, employing a binary classification scheme of propaganda versus non-propaganda. similarly, habernal et al. (2018) developed a corpus annotated with fallacies, including ad hominem and red herring, directly relevant to propaganda techniques. bert-based variants have emerged as promising methodologies for classification tasks in tandem with corpus development. yoosuf and yang (2019) proposed a fine-tuning approach post-worldlevel classification using bert, while fadel et al. (2019) presented a pre-trained ensemble model integrating bilstm, bert, and rnn components. further extending the capabilities of bert, costa et al. (2023) advocated for a multilingual setup, employing translation to english before utilizing roberta. additionally, teimas and saias (2023) proposed a hybrid technique combining cnn with distilbert for improved detection accuracy. exploring multimodal content, glenski et al. (2019) delved into multilingual multimodal deception detection, mainly focusing on hateful memes. leveraging visual and textual content, they utilized fine-tuning techniques with state-of-the-art models like vilbert and visualbert and transfer learning-based approaches @xcite . ## data description the competition consisted of two different phases mainly the development phase which we refer to as the development set and for the development phase we were provided the training and validation sets for benchmarking our models all three sub-tasks have different sets of memes split across training, validation and develepmont sets as shown in our analysis used a dictionary to map various rhetorical techniques to numerical values for plotting. this dictionary is as follows: persuasion technique number mapped to presenting irrelevant data (red herring) 0 bandwagon 1 smears 2 glittering generalities (virtue) 3 causal oversimplification 4 whataboutism 5 loaded language 6 exaggeration/minimisation 7 repetition 8 thought-terminating cliché 9 name calling/labeling 10 appeal to authority 11 black-and-white fallacy/dictatorship 12 obfuscation, intentional vagueness, confusion (straw man) 13 reductio ad hitlerum 14 appeal to fear/prejudice 15 misrepresentation of someone's position (straw man) 16 flag-waving 17 slogans 18 doubt 19 the proposed system for all the sub-tasks involves task-specific modifications made to the bert model and earlier proposed works including clip model @xcite , class definition based emotion predictions @xcite and hypemo model @xcite ) (described below). ## results we conducted several experiments across all the sub-tasks, and the detailed information can be seen in for task 1, we started experimenting with the bert and roberta models, achieving a hierarchical f1 score of 0.55 and 0.60 on the test set of the english language. but, in this approach, we did not take the hierarchy and the definitions of the classes into consideration. we tried to accommodate that using the combination of hypemo and cdp models. for the hypemo model, the model was trained to prioritize higher-level labels in the directed acyclic graph (dag). during this process, we explored two options: eliminating children when the model predicted the parent label and retaining the children. we observed a significant impact on the hierarchical f1 score, with the first formulation yielding 0.45 f1 and the second approach resulting in 0.59 on the test set. we also tried to predict the labels utilizing only the definitions of the classes, using the cdp model, which yielded a hierarchical f1 score of 0.57 and 0.59 on the dev set and the test set, respectively. for constructing an ensemble, one approach considered concatenating embeddings or softmax predictions from both models for further classification using a neural network. however, this approach was not viable due to limited samples for generalization. the most effective model emerged from utilizing the ensemble with fine-tuning of hyperparameters. combining predictions from both models yielded a hierarchical f1 score of 0.60. for sub-task 2, we trained the model from scratch after including the two labels used in the ensemble used in sub-task 1 and changed the feature embeddings being trained by considering the features from the visual content. however, as you can see, there is very little to no difference between the results using clip and not using clip. we can also see that, unlike the first subtask, they perform better due to more data. we can see the f1-score analysis tables for each subtask, i.e., in ## conclusion detection of persuasion techniques in memes is seen in a multi-modal setting in this task, but the significant features are drawn from the textual cues in the memes, which can be seen in the results of sub-tasks 1 and 2. the clip and other visual language models still need considerable development, and visual cues are helpful for only specific input-output pairs. identifying whether a persuasion technique is present in the meme but does not apply to the multi-label classification task can be beneficial. also, we have used a basic ensemble of the latest works in this area and modified them for task-specific requirements. still, other complex architectures can be explored to get better results.