LT 3 at S em E val-2021 Task 6: Using Multi-Modal Compact Bilinear Pooling to Combine Visual and Textual Understanding in Memes (2021)

Abstract: Internet memes have become ubiquitous in social media networks today. Due to their popularity, they are also a widely used mode of expression to spread disinformation online. As memes consist of a mixture of text and image, they require a multi-modal approach for automatic analysis. In this paper, we describe our contribution to the SemEval-2021 Detection of Persuasian Techniques in Texts and Images Task. We propose a Multi-Modal learning system, which incorporates “memebeddings”, viz. joint text and vision features by combining them with compact bilinear pooling, to automatically identify rhetorical and psychological disinformation techniques. The experimental results show that the proposed system constantly outperforms the competition’s baseline, and achieves the 2nd best Macro F1-score and 14th best Micro F1-score out of all participants.

## introduction propaganda is a mode of communication by which the interested party pursues the aim of influencing public opinion in favour of a specific agenda or ideas. this is achieved by disseminating onesided, biased or even fake news. with the advent of social media networks, propagandist text can reach an enormous audience. given the overload of online text produced on a daily basis, it is not feasible to monitor this manually and researchers have started to investigate automatic methods to detect propaganda in text. in the semeval-2020 task 11 on detection of propaganda techniques in news articles @xcite , participants were asked to identify 14 different propagandist techniques in news articles. the task attracted a large interest, with 44 teams participating in the task. the best systems all used pre-trained transformers and ensemble techniques. we further refer @xcite for a comprehensive review of computational propaganda detection techniques. more recently, internet memes have emerged as a very popular mode of expression on social networks. while memes initially seemed to be used by users of specific online communities, they have gained popularity very rapidly and are today used by a very large and varied user base. memes can be used for very different purposes: they can be used as a form of visual rhetoric @xcite , for online bullying or trolling @xcite , but they can also function as a kind of persuasive device, while the intended message is wrapped in humour @xcite . as a result, they form an interesting object of study for automatically detecting propaganda techniques. the goal of the semeval-2021 shared task on the detection of persuasion techniques @xcite is to build models for identifying rhetorical and psychological techniques that are used to influence social media users in online disinformation campaigns. this paper reports on our participation in subtask 3, which is a multi-modal task conceived as a multi-label classification problem: given a meme, the system has to identify which of the 22 techniques are used both in the textual and visual content of the meme. to solve subtask 3, we propose a multi-modal multi-task learning system, which incorporates "memebeddings", viz. joint text and vision features combined by means of compact bilinear pooling, to automatically identify rhetorical and psychological disinformation techniques in memes. ## results while the mcb model is able to better pool the understanding of both modalities, it still fails when a complex concept or inference is involved. figure 3a represents a common meme format frequently found in the memes we were able to obtain from reddit. consequently, we expect the model has sufficiently learnt the corresponding visual features to come to a correct prediction. figure 3b , however, requires some complex visual ideas the model has to infer in combination with the text, which it fails to do frequently. we believe that jointly training visual and textual embeddings (making the learning more coherent and not disjointed between the two modalities), instead of simply attempting to combine independent vimacro-f1 micro-f1 baseline 0.051 0.070 feature combination with weighted averaging 0.112 0.248 multi-modal compact bilinear pooling (our system) 0.263 0.331 ## conclusion and future work this paper presents the multi-modal approach we proposed for automatically detecting persuasion techniques in memes. as memes combine text and images to obtain the desired effect, we built a system where visual and textual embeddings are combined to classify 22 different propaganda techniques. the experimental results show that combining textual and visual embeddings by weighted averaging already beats the baseline. these results, however, are considerably improved by combining both embedding sets by means of mcb pooling. in future work, we will investigate how we can incorporate additional semantic information in our model. a first step could consist of integrating more explicit argumentation information into our model. as these propaganda techniques use psychological and rhetorical techniques, we believe it might be interesting to include argumentation structures such as logical fallacies, where the reasoning is flawed, and by consequence the conclusion cannot be drawn from the premise(s) in the text. to this end, we will build on recent work on automatic fallacy detection @xcite . in addition, we also aim to include automatic emotion detection features, as writers of propagandist text often use emotional language to convince their readers. finally, we will investigate an approach to jointly train visual and textual embeddings, rather then combining separate embedding sets, as our error analysis showed that efficient analysis of memes often requires a combined approach.