DUTIR 938 at S em E val-2024 Task 4: Semi-Supervised Learning and Model Ensemble for Persuasion Techniques Detection in Memes (2024)

Abstract: The development of social platforms has facilitated the proliferation of disinformation, with memes becoming one of the most popular types of propaganda for disseminating disinformation on the internet. Effectively detecting the persuasion techniques hidden within memes is helpful in understanding user-generated content and further promoting the detection of disinformation on the internet. This paper demonstrates the approach proposed by Team DUTIR938 in Subtask 2b of SemEval-2024 Task 4. We propose a dual-channel model based on semi-supervised learning and model ensemble. We utilize CLIP to extract image features, and employ various pretrained language models under task-adaptive pretraining for text feature extraction. To enhance the detection and generalization capabilities of the model, we implement sample data augmentation using semi-supervised pseudo-labeling methods, introduce adversarial training strategies, and design a two-stage global model ensemble strategy. Our proposed method surpasses the provided baseline method, with Macro/Micro F1 values of 0.80910/0.83667 in the English leaderboard. Our submission ranks 3rd/19 in terms of Macro F1 and 1st/19 in terms of Micro F1.

## introduction social networks play a significant role in our society. the development of social platforms has facilitated the dissemination of information, but it has also fueled the proliferation of disinformation @xcite . the dissemination mechanism of disinformation involves the use of propaganda techniques. "propaganda" is defined as a dissemination pattern referring to stakeholders influencing public opinion to support specific agendas and ideas by adopting persuasion techniques, such as disseminating one-sided, biased, or even fake news. research * corresponding author on detecting propaganda techniques contributes to combating network disinformation @xcite . among all types of content on social networks, memes play a significant role. memes typically exist in the form of images, possibly with overlapping text, and convey information in the form of jokes, irony, etc. in the current era of social media, they spread rapidly and can influence many people without awareness. memes are one of the most popular content types in online disinformation propaganda activities and serve as a powerful medium for promoting ideological and cognitive persuasion techniques @xcite . therefore, research on automatically detecting persuasion techniques hidden in memes is of significant importance, which contributes to understanding user-generated content and further aids in detecting network disinformation. subtask 2b of semeval-2024 task 4 aims to promote research on computational methods to detect persuasion techniques in memes @xcite , which is modeled as a binary classification problem. due to the complexity and subjectivity inherent in persuasive language, single multimodal model may struggle to capture all relevant features effectively. to address this issue, we propose a dual-channel model based on semisupervised learning and model ensemble in this paper. within the image channel, we use clip @xcite to extract image features from memes. concurrently, in the text channel, we employ diverse pretrained language models and conduct task-adaptive pretraining utilizing certain corpora provided by the task. we execute feature extraction from the pretrained language model, employing a variety of methodologies to capture sentence features, followed by a concatenation and fusion process of the extracted features, subsequently fed into the classification layer. we implement a semi-supervised pseudo-labeling ap-1 642 proach, wherein pseudo-labels are assigned to the test set, thereby augmenting the training data to achieve data augmentation. furthermore, to bolster each model's robustness, we employ fast gradient method(fgm) as our adversarial training strategy, introducing perturbations to the embedding layer of the model. lastly, we design a two-stage softvoting ensemble strategy to amalgamate the predictions of multiple models, augmenting the model's generalization capacity and performance. we applied our proposed method to the english dataset. our proposed method ranked 3rd/19 in terms of macro f1 and 1st/19 in terms of micro f1 in the english leaderboard for subtask 2b. ## related work transformer @xcite is a deep learning model based on the self-attention mechanism, which is known for its ability to effectively capture long-range dependencies in sequential data. based on transformer, pretrained language models such as bert (kenton and toutanova, 2019) have been proposed, which capture the contextual information of each token in the text through selfattention. subsequent pretrained language models have mainly been modified on pretraining tasks, such as roberta @xcite , deberta @xcite , and so on. these text models improve upon bert by enhancing semantic feature representation in text feature extraction. existing image feature extraction methods mostly rely on multiple convolutional neural networks @xcite , such as vgg @xcite and resnet @xcite . vision transformer (vit) @xcite is an image processing model based on the traditional transformer, dividing the input image into image patches and using multi-head self-attention mechanisms to capture global relationships between images. vit represents a significant advancement of transformer in the field of computer vision, bringing new ideas and methods to image processing tasks. clip @xcite ) is a multimodal model which is trained through contrastive learning on 400 million pairs of images and text. clip achieves high-performance cross-modal semantic feature extraction for images and text. ## system overview in this section, we will introduce the overall structure of our proposed system. our system is di-vided into two stages. (1) training of our text-image multi-modal classification model. during the training process, we introduce several training strategies, including task-adaptive pretraining, pseudo-labeling and adversarial training. (2) model ensemble. we use k-fold cross-validation for model training and design a two-stage soft-voting strategy to globally integrate the models obtained in the first stage. ## results the overview statistics of four different models on the new validation set are shown in regarding model ensemble, our analysis, illustrated in we employed all-model ensemble as the final model of our system and submitted the result file of the test set predicted by the final model. the official rankings are shown in ## conclusion the paper presents our system designed for subtask 2b of semeval-2024 task 4. we propose a dualchannel model based on semi-supervised learning and model ensemble. our framework leverages multiple pretrained models served as feature extractors for images and texts. we integrate a semisupervised pseudo-labeling approach for data augmentation, and introduce tapt and fgm adversarial training to significantly enhance the model's performance and robustness. finally, to enhance the generalization capability of our system, we design a two-stage soft-voting model ensemble strategy. our system achieves excellent performance in detecting persuasion techniques in memes, and we ranked 3rd in terms of macro f1 and 1st in terms of micro f1 in the test set for subtask 2b. our future research will be directed towards exploring the cross-modal fusion mechanisms within the model.