Alpha at S em E val-2021 Task 6: Transformer Based Propaganda Classification (2021)

Abstract: This paper describes our system participated in Task 6 of SemEval-2021: the task focuses on multimodal propaganda technique classification and it aims to classify given image and text into 22 classes. In this paper, we propose to use transformer based architecture to fuse the clues from both image and text. We explore two branches of techniques including fine-tuning the text pretrained transformer with extended visual features, and fine-tuning the multimodal pretrained transformers. For the visual features, we have tested both grid features based on ResNet and salient region features from pretrained object detector. Among the pretrained multimodal transformers, we choose ERNIE-ViL, a two-steam cross-attended transformers pretrained on large scale image-caption aligned data. Fine-tuing ERNIE-ViL for our task produce a better performance due to general joint multimodal representation for text and image learned by ERNIE-ViL. Besides, as the distribution of the classification labels is very unbalanced, we also make a further attempt on the loss function and the experiment result shows that focal loss would perform better than cross entropy loss. Last we have won first for subtask C in the final competition.

## introduction propaganda is usually adopted to influence the audience by selectively displaying the facts to encourage specific synthesis or perception, or using the loaded language to produce emotion rather than emotion itself. it was often associated with materials prepared by governments in the past century. in the internet era, activist groups, companies, religious organizations, the media, and individuals also * indicates equal contribution. with the recent research interest in detecting "fake news", the detection of persuasion techniques in the texts and images has emerged as an active research area. most previous work like @xcite and @xcite have performed the analysis at the language content level only. however, in our daily life, memes consist of images superimposed with texts. the aim of the image in a meme is either to reinforce a technique in the text or to convey one or more persuasion techniques. semeval-2021 task6-c offers a different perspective, multimodal multi-label classification @xcite , identify which of the 22 techniques are used both in the textual and visual content of memes. since memes are combinations of texts and images, for this propaganda classification task, we proposed to use transformer-based architecture to fuse the clues from both linguistic and visual modalities. two branches of finetuning techniques are explored in this paper. first, a text pre-trained transformer is applied with extended visual features. specifically, we initialize the transformer with pre-trained text transformers and fine-tune the model with extended visual features including grid features(e.g., resnet @xcite ) and region features(e.g., butd @xcite ) from an image feature extraction network and an object detector respectively. second, pre-trained multimodal transformers from ernie-vil @xcite are used due to its better multimodal joint representations characterizing crossmodal alignments of detailed semantics. our contributions are three-folds: ## approach we propose to use a transformer encoder to fuse the clues from both linguistic and visual modalities and our approach is summarized in two branches, the first one is fine-tuning a text pre-trained transformer with extended visual features, and the other one is fine-tuning a multimodal pre-trained model. for the first one, we try two different sets of visual features, grid features based on equally-split patches of the image and salient region features based on an object detector. for the second one, a sota multimodal model, ernie-vil @xcite is applied with a multi-label classification loss. a unified framework for the two branches is shown in figure 1 . we will introduce more details in this section. ## conclusion we explore two branches to fine-tune pre-trained transformers to jointly modelling texts and images for the propaganda classification task. the first branch, fine-tuning pre-trained text transformer with visual feature, obtain significant performance improvement compared to text classification which validate the importance of visual clues for this task. visual features from object detector yield slightly better results than grid features from resnet. importantly, fine-tuning pre-trained multimodal transformers obtain the best single model performance. and this improvement further validates the claim made by previous work that vision-language pretraining learned general joint representation needed for multimodal tasks. besides, since the distribution of the classification labels is extremely unbalanced, we also make a further attempt on the loss function. training models with focal loss can lead to a huge performance improvements than training with cross entropy loss.