NLPNCHU at S em E val-2024 Task 4: A Comparison of MDHC Strategy and In-domain Pre-training for Multilingual Detection of Persuasion Techniques in Memes (2024)

Abstract: This study presents a systematic method for identifying 22 persuasive techniques used in multilingual memes. We explored various fine-tuning techniques and classification strategies, such as data augmentation, problem transformation, and hierarchical multi-label classification strategies. Identifying persuasive techniques in memes involves a multimodal task. We fine-tuned the XLM-RoBERTA-large-twitter language model, focusing on domain-specific language modeling, and integrated it with the CLIP visual model’s embedding to consider image and text features simultaneously. In our experiments, we evaluated the effectiveness of our approach by using official validation data in English. Our system in the competition, achieving competitive rankings in Subtask1 and Subtask2b across four languages: English, Bulgarian, North Macedonian, and Arabic. Significantly, we achieved 2nd place ranking for Arabic language in Subtask 1.

## introduction propaganda and advertising serve as examples of persuasive discourse, which aims to change another's behavior, feelings, intentions, or views through communication, often in a one-sided manner @xcite . hence, the context in which the communication occurs is crucial alongside the actual content being conveyed. memes, combining persuasive discourse on social media platforms, prove particularly effective. they spread ideas or emotions online and are a popular tool in misinformation campaigns, using various rhetorical and psychological techniques to influence users. memes' visual components either reinforce or convey persuasive tactics, thus playing a significant role in shaping public opinion and attitudes. to address these challenges, semeval-2024 introduced a shared task focusing on detecting 1 https://huggingface.co/sdadas/xlm-roberta-large-twitter persuasion techniques from multilingual memes @xcite . this task defines a hierarchy directed acyclic graph (hdag) to represent a meme's persuasive techniques and highlights the challenges and importance of understanding the nuances of digital persuasion. this study proposes exploring the effectiveness of multi-dimensional hierarchical classification (mdhc) strategies in identifying persuasive techniques in memes, based on previous research and the successful application of mdhc strategies in real-world hdags. the results from a competition show our approach's effectiveness, ranking first in a specific subtask and competitively across others, demonstrating the potential of mdhc strategies in analyzing persuasive discourse. ## background in this study, we explore the application of hierarchical multi-label classification (hmc) in the context of persuasive techniques, by structuring them within a hierarchical multi-label framework. this approach allows for the simultaneous handling of both textual and visual data through multimodal modeling. recent research @xcite has demonstrated the efficacy of the mdhc approach for hmc, noting its simplicity and ease of implementation. hmc, an advancement of multi-label classification (mc), is designed to predict multiple labels that are organized hierarchically from general to specific categories. the incorporation of hierarchical knowledge is found to significantly improve the performance of classifiers. the mc, which is applicable in a wide range of areas, involves the challenge of predicting multiple interrelated category variables. as highlighted by @xcite and @xcite , the complexity of mc compared to single-dimensional problems is primarily 1 1868 due to the vast combinations of class labels and the scarcity of relevant data. we transform persuasive techniques into the hmc framework, this approach transforms persuasive technique graphs for application in specific subtasks. given the necessity to analyze both textual and visual data for accurately identifying persuasive techniques, multimodal models become essential. clip (contrastive language-image pre-training) proposed by openai @xcite , stands out for its independent text and image encoding capabilities, offering flexibility for various subtask types. in a study, (kumar and nandakumar, 2022) have suggested a range of techniques that combine textual and visual embedding vectors, leading to the effective detection of hateful memes. they also conducted various fusion experiments by switching different text encoders. therefore, we refer to the authors' approach to combine the embedding vectors of the clip and the multilingual model with the aim of better adapting to subtask 2ab, which involves tasks belonging to the multilingual domain and including datasets in three non-english languages (bulgarian, north macedonian, arabic) in the test set. ## exploratory data analysis for datasets the dataset used in this study contains about 15,000 memes in english and other languages. we have examined the label distribution for each task, and it is apparent that the data sets for subtask 1 and 2a are imbalanced, which differing numbers of memes's persuasive techniques available for positive and negative are shown in figure 2 and figure 6 . further scrutiny, as delineated in attachment figure 5 , it's evident that the datasets for subtask 1 and 2a have a highly imbalanced distribution of data across 22 persuasion techniques, with subtask 2a, in particular, showing a significant imbalance between positive and negative samples. we will describe how to address these imbalances in subsequent sections. ## system overview in our research, we conducted an in-depth comparison of two mdhc strategies: stacking+gc and stacking+lcl, utilizing the same dataset for model training. the comparative analysis revealed that stacking+gc demonstrated superior performance over stacking+lcl. this superiority is attributed to its more effective handling of errors during the merging process of hierarchical levels, thereby enhancing the overall classification accuracy within the hierarchical structure of the data. xlm-roberta-large-twitter 1 for this system task, which is multilingual and specifically focused on the social media domain of memes, we finetuned the domain-specific language model xlm-roberta-large-twitter. this model was adjusted based on a corpus of over 156 million tweets in ten languages. clip uses two distinct architectures as the backbone for encoding visual and textual datasets: image encoder, which represents the neural network architecture responsible for encoding images (e.g., resnet or vision transformer), and text encoder, which represents the neural network architecture responsible for encoding textual information (e.g., bert or text transformer). this structure is flexibly adapted to the subtasks of this project. ## experiment and evaluate in our experiments, we employed three mdhc classification strategies on subtasks. all tasks utilize the hierarchyf 1 evaluation metric, which is the unified evaluation metric provided by the official source. in the experiment setting, refer to appendix a for details of the relevant parameters. in subtask 1, we compared the performance of two language models, xlm-roberta and xlm-roberta-twitter, across subtask 1 and 2a, to assess the impact of domain-specific pre-training. we explored three classification strategies: gc, stacking + gc, and stacking + lcl, to identify the most effective approach for subtask 1. in subtask 2a, we evaluated two multimodal models by combining clip with xlm-roberta-twitter from subtask 1. the goal was to improve the comprehension of meme's persuasive techniques by utilizing a fine-tuned text encoder. additionally, we employed gc, stacking + gc, and stacking + lcl strategies to determine which one is more effective for subtask 2a. our primary focus was on improving classification performance in a multimodal setting. in subtask 2b, we explored multimodal model and classification strategy using the clip + xlm-roberta-twitter combination from subtask 1, applied to a binary classification framework. utilizing a balanced dataset of hate memes collected by meta ai, as referenced in the harmful memes dataset @xcite , the goal was to train the model on balanced samples to prevent bias effectively. in the test set of the competition, we propose a method that has demonstrated competitive performance on the official competition leaderboard. as shown in in the multimodal task, as shown in finally, in subtask 2b, as shown in ## conclusion we conducted a detailed analysis of the hdag containing persuasive techniques, transforming it into an hmc task. we also explored two mdhc strategies and highlighted the importance of addressing the long-tail distribution issue, proposing the use of the dbl loss function to mitigate this issue in hmc tasks. regarding the models, we recommend utilizing domain-specific pre-training to detect memes containing persuasive elements and the effectiveness of domain-specific training was demonstrated across various experiments. finally, we achieve competitive results in the competition.