M in D at S em E val-2021 Task 6: Propaganda Detection using Transfer Learning and Multimodal Fusion (2021)

Abstract: We describe our systems of subtask1 and subtask3 for SemEval-2021 Task 6 on Detection of Persuasion Techniques in Texts and Images. The purpose of subtask1 is to identify propaganda techniques given textual content, and the goal of subtask3 is to detect them given both textual and visual content. For subtask1, we investigate transfer learning based on pre-trained language models (PLMs) such as BERT, RoBERTa to solve data sparsity problems. For subtask3, we extract heterogeneous visual representations (i.e., face features, OCR features, and multimodal representations) and explore various multimodal fusion strategies to combine the textual and visual representations. The official evaluation shows our ensemble model ranks 1st for subtask1 and 2nd for subtask3.

## introduction with the recent interest in "fake news", the detection of propaganda or highly biased texts has emerged as an active research area @xcite @xcite . @xcite ) provides three subtasks aiming to detect persuasion techniques in texts and images. we participate in subtask1 and subtask3, which are defined as follows: for subtask1, we focus on using transfer learning to tackle problems related to the scarcity of data since deep learning models require a whole lot of data while it is difficult to obtain vast amount of the labeled data. especially, we first fine-tune the pre-trained language models on an external dataset from semeval-2020 task 11 @xcite and then continue to fine-tune them on the training dataset of semeval-2021 task 6. the probabilities of these tuned models are averaged to make the final prediction. for subtask3, we concentrate on multimodal fusion to combine textual and visual representation. heterogeneous visual representations are extracted, including face, ocr and multimodal representations. face representation consists of recognized human faces and facial expressions. ocr representation can capture the relations among snippets in an image. multimodal pre-trained model is capable of simultaneously processing multimodality inputs for joint visual and textual understanding. after that, we explore three multimodal fusion strategies (i.e., average, concat and mlp) to combine the textual and visual representations. the experimental results show that transfer learning can leverage knowledge from source data to tackle problems related to the scarcity of data, and heterogeneous visual representation (i.e., face, ocr, and multimodal representation) can be used as complementary features to better detect persuasion techniques. our ensemble model ranks 1st for subtask1 and 2nd for subtask3. ## system overview in this section, we provide a general overview of our systems for the two subtasks. we consider the propaganda detection task as multimodal multiclass multi-label classification task, predicting one or more labels given an input text and an input image. ## discussion more thorough studies and analyses are conducted in this section, trying to answer two questions: (1) how is the performance of transfer learning on less data? (2) how is the performance of multimodal fusion on multimodal data? moreover, we give ## conclusion in this paper, we adopt transfer learning to handle data sparsity problems for subtask1, and fuse heterogeneous multimodal representation for sub-task3. the experimental results show that transfer learning can leverage knowledge from source data to tackle problems related to the scarcity of data, and heterogeneous visual representation (i.e., face, ocr, and multimodal representation) can extract complementary features. in future work, we plan to explore fine-grained multimodal fusion with token representations in text and object features in images.