LIIR at S em E val-2021 task 6: Detection of Persuasion Techniques In Texts and Images using CLIP features (2021)

Abstract: We describe our approach for SemEval-2021 task 6 on detection of persuasion techniques in multimodal content (memes). Our system combines pretrained multimodal models (CLIP) and chained classifiers. Also, we propose to enrich the data by a data augmentation technique. Our submission achieves a rank of 8/16 in terms of F1-micro and 9/16 with F1-macro on the test set.

## introduction online propaganda is potentially harmful to society, and the task of automated propaganda detection has been suggested to alleviate its risks @xcite . in particular, providing a justification when performing propaganda detection is important for acceptability and application of the decisions. previous challenges have focused on the detection of propaganda techniques @xcite , based on news articles. however, many use cases do not solely involve text, but can also involve other modalities, notably images. task 6 of semeval-2021 proposes a shared task on the detection of persuasion techniques detection in memes, where both images and text are involved. substasks 1 and 2 deal with text in isolation, but we focus on subtask 3: visuolinguistic persuasion technique detection. this article presents the system behind our submission for subtask 3 @xcite . to handle this problem, we use a model containing three components: data augmentation, image and text feature extraction, and chain classifier components. first, given a paired image-text as the input, we paraphrase the text part using back-translation and pair it again with the corresponding image to enrich the data. then, we extract visual and textual features using the clip @xcite image encoder and text encoder, respectively. finally, we use a chain classifier to model the relation between labels for the final prediction. our proposed method, named liir, has achieved a competitive performance with the best performing methods in the competition. also, empirical results show that the augmentation approach is effective in improving the results. the rest of the article is organized as follows. the next section reviews related works. section 3 describes the methodology of our proposed method. we will discuss experiments and evaluation results in sections 4 and 5, respectively. finally, the last section contains the conclusion of our work. ## related work this work is related to computational techniques for automated propaganda detection @xcite and is the continuation of a previous shared task @xcite . taks 11 of semeval-2020 proposes a more finegraind analysis by also identifying the underlying techniques behind propaganda in news text, with annotations derived from previously proposed propaganda techniques typologies @xcite . this current iteration of the task tackles a more challenging domain, by including multimodal content, notably memes. the subtle interaction between text and image is an open challenge for state of the art multimodal models. for instance, the hateful memes challenge @xcite was recently proposed, as a binary task for detection of hateful content. the recent advances in pretraining of visuolinguistic representations @xcite lead the model closer to human accuracy @xcite . more generally, propaganda detection is at the crossroad of many tasks, since it can be helped by many subtasks. fact-checking @xcite can be involved with propaganda detection, alongside various social, emotional and discursive aspects @xcite , including offensive language detection @xcite ) emotion analysis (dolan, 2002), computational study of persuasiveness @xcite and argumentation @xcite . ## methodology in this section, we introduce the design of our proposed method. the overall architecture of our method is depicted in figure 1. our model consists of several components: a data augmentation component (back-translation), a feature extraction component(clip), and a chained classifier. details of each component are described in the following subsections. ## negative results we also tried to use clip as a zero-shot classifier for propaganda technique detection. to do so, we constructed prompts such as : for each input image/text, we generated a prompt for each labels, and used clip to estimate the estimate an affinity score between the prompt and the image. clip is designed to predict relatedness between the input image and text, and we expected that an input text mentioning the relevant propaganda technique should be associated with higher probabilities that the others. however, this method did not seem to perform better than chance. this suggests that propaganda detection technique task might be too abstract for clip in zero-shot settings. ## conclusion we described our submission for the shared task of multimodal propaganda technique detection at semeval-2021. our system performances that are competitive with other systems even though we used a simple architecture with no ensemble, by leveraging non-supervised learning. we believe that further work on zero-shot learning would be a valuable way to improve propaganda detection techniques for the least frequent labels.