1213 L i at S em E val-2021 Task 6: Detection of Propaganda with Multi-modal Attention and Pre-trained Models (2021)

Abstract: This paper presents the solution proposed by the 1213Li team for subtask 3 in SemEval-2021 Task 6: identifying the multiple persuasion techniques used in the multi-modal content of the meme. We explored various approaches in feature extraction and the detection of persuasion labels. Our final model employs pre-trained models including RoBERTa and ResNet-50 as a feature extractor for texts and images, respectively, and adopts a label embedding layer with multi-modal attention mechanism to measure the similarity of labels with the multi-modal information and fuse features for label prediction. Our proposed method outperforms the provided baseline method and achieves 3rd out of 16 participants with 0.54860/0.22830 for Micro/Macro F1 scores.

## introduction the development of the internet and information technology promotes the generation and dissemination of information, but also fuels the proliferation of disinformation. as one of the most popular types of content in disinformation, memes attract readers easily and brought further challenges to the detection of disinformation @xcite . specifically, memes employ a number of techniques to influence users, which can be divided into the use of logical fallacies and appealing to the emotions of the audience @xcite . in practice, the former misuses logical rules to disguise wrong conclusions as correct and objective, figure 1 : examples of multi-modal samples, we rewrite the sentences on our own and collect the images from ‡ , § , ¶ , respectively. the first two rows illustrate the visual and the textual content, and in the last row, each line reveals the label (techniques) of the sample. while the latter utilizes emotional language to induce the audience to agree with the speaker emotionally and prevent their rational analysis of the argumentation. identifying the techniques used in memes contributes to the understanding of user-generated content and further helps to the detection of disinformation. the subtask 3 of @xcite is organized to stimulate the study of computational methods to detect persuasion techniques in memes that inhere in texts and images. as shown in figure 1 , each sample consists of a set of textual sentences and an attached image. according to the task description @xcite , the image and the sentence could convey the modality-specific persuasion techniques, respectively, and at the same time, images can be combined with sentence to express some techniques, which we named global techniques. based on the understanding of the task, we attribute the main challenges of subtask 3 to the following three aspects: 1) extracting essential features from each modality to predict modality-specific labels, 2) fusing multi-modal features to understand the content fully for predicting global labels, and 3) capturing the connections among multi labels. correspondingly, we present the methods to handle these challenges. specifically, our method employs the powerful feature extractor including the pre-trained roberta @xcite and resnet-50 @xcite to extract textual and visual features, respectively. besides, inspired by the work @xcite , our method adopts a label embedding layer to learn how semantically close the labels are to one another implicitly, and the embedding layer maps each label to a learnable fixed-size vector. before the label prediction, multi-modal features are fused according to their relevance with each label useing attention mechanism @xcite , and the final prediction is based on the fused features. ## related work subtask 3 is a multi-label classification task based on multi-modal data. as for the multi-label classification tasks, it was earlier handled by many machine learning methods. zhang et al. @xcite used a k-nearest neighbor-based algorithm to conduct experiments on real-world multilabel bioinformatic data. vens et al. @xcite proposed a hierarchical multi-label classification method based on decision trees. with the rapid development of deep learning, multi-label classification methods based on deep neural networks have become mainstream. wang et al. @xcite introduced and multi-label image classification network with the fusion of cnn and rnn. chernyavskiy et al. (chernyavskiy et al., 2020 ) used a roberta-based network combined with additional crf @xcite layers and transfer learning mechanism (pan and yang, 2010) to address a multi-label classification task in semeval-2020. however, these previous multi-label classification tasks were often based on single modality data. these approaches fall short when the task requires the use of multiple modal data. moreover, in the field of multi-modal tasks, we focus on task of multi-modal fake news detection. recent work @xcite @xcite mainly concern the fusion of multi-modal features and adopt a binary classifier, which is not applicable to current multi-label classification scenarios. ## results and discussion the result of the ablation study is shown in as we can see, the baseline method is very ineffective since it utilizes only the average-pooling features of visual and textual information, indicating that the lack of the interaction between modalityspecific features and label information hinder the model to select vital features for prediction and leads to poor performance. so we introduce the attention mechanism to selectively choose valid information from visual features and textual features, respectively. as shown in the second group of finally, the model that uses both visual features and textual features in combination with the attention mechanism has the optimal performance. during the test stage, we chose the model that performed best on the development set and got the final result through the ensemble. the final evaluation results are reported in ## conclusion this paper demonstrates the method that we proposed for subtask 3 in semeval-2021 task 6, which aims to identify which of 22 persuasion techniques are used in the textual and visual content of the specific meme. our method uses roberta and resnet-50 to extract multi-modal features, introduces the attention mechanism to fuse multi-modal features, and adopts the label embeddings to learn the representation of labels. our proposed model achieves noticeable improvements over the baseline method, and the official evaluation ranked our submission 3rd out of 16 teams.