AIMH at S em E val-2021 Task 6: Multimodal Classification Using an Ensemble of Transformer Models (2021)

Abstract: This paper describes the system used by the AIMH Team to approach the SemEval Task 6. We propose an approach that relies on an architecture based on the transformer model to process multimodal content (text and images) in memes. Our architecture, called DVTT (Double Visual Textual Transformer), approaches Subtasks 1 and 3 of Task 6 as multi-label classification problems, where the text and/or images of the meme are processed, and the probabilities of the presence of each possible persuasion technique are returned as a result. DVTT uses two complete networks of transformers that work on text and images that are mutually conditioned. One of the two modalities acts as the main one and the second one intervenes to enrich the first one, thus obtaining two distinct ways of operation. The two transformers outputs are merged by averaging the inferred probabilities for each possible label, and the overall network is trained end-to-end with a binary cross-entropy loss.

## introduction social networks play a critical role in our society. nowadays, most of the ideas, thoughts, and political beliefs are shared through the internet using social platforms like twitter, facebook, or instagram. although these online services enable information to be spread efficiently and effectively, it is non-trivial to understand if the shared contents are free of subtle meanings altering people's judgment abilities. among all the types of content living in a social network, memes acquire a significant role. memes are small yet effective units of information able to spread cultural ideas, symbols, or practices and usually exist under the form of pictures, possibly with overlayed text. memes are created so that they can propagate rapidly and reach a large number of users; for this reason, they are one of the most popular types of content used in an online disinformation campaign, influencing the users through several rhetorical and psychological techniques, such as causal oversimplification, name-calling, or smear. the automatic detection of these memes and the disinformation techniques they are possibly employing is a challenging yet crucial task for the proper management of a social network. in the last few years, machine learning and deep learning have defined remarkable milestones in automatic content extraction and reasoning from multimedia data. all these breakthroughs acquire a fundamental role in large-scale analysis of multimedia content from social networks. in this work, we tackle the problem of recognizing which kind of disinformation technique is used to forge memes for a disinformation campaign. in particular, we propose an architecture based on the well-established transformer architecture model @xcite for processing both the textual and visual inputs from the meme. this architecture, which we call dvtt (double visual textual transformer), comprises two full transformer networks working respectively on images and texts; however, each of these transformers is conditioned on the other modality. we consider this task as a multi-label classification problem, where text and/or images from the meme are processed, and probabilities of presence of each possible persuasion technique are returned as a result. in this paper, we tackle subtasks 1 and 3 of the semeval 2021 task 6 challenge foot_0 @xcite . subtask 1 consists of identifying which of 20 possible persuasion techniques are used in it given only the textual content; subtask 3 is very similar to subtask 1, but both textual and visual contents of the meme are used, and there are 22 possible persuasion techniques. our proposed models could reach the 5th position for subtask 1 and the 4th position for subtask 3 on the publicly available leaderboard. the code for replicating our results is available on github 2 . ## background recently, machine learning, and deep learning in particular, defined astonishing milestones in automatic content extraction and reasoning from multimedia data. in particular, concerning joint visual and textual analysis, many state-of-the-art approaches succeeded in tasks like visual question answering @xcite @xcite , image captioning @xcite @xcite , and image-text matching @xcite @xcite @xcite , often using structured reasoning using graph networks and graph convolutions. in the last few years, a graph-network related model, the transformer network @xcite , acquired increasing attention on the joint processing of images and texts. many works, inspired by the bert model @xcite , obtained remarkable results on word region alignments, visual-question answering, and image-text matching using transformer encoders @xcite @xcite . recently, the authors in @xcite ) used the full transformer stack to construct a powerful object detector, demonstrating these models' potential in pure visual contexts. given the enormous flexibility of the transformer architecture, in this work, we consider images and texts respectively as sets and sequences of vectors, and we ask the transformer to process them to produce probabilities of presence of each possible persuasion technique. ## system overview in this section, we first give a brief overview of the transformer architecture on which our proposal is based; then, we present in detail our system proposals for solving subtasks 1 and 3. ## experiments we used the data provided by the semeval 2021 task 6 challenge organizers to train and validate our model. although we mainly concentrated on subtask 3 (images + texts), we also tackled subtask 1, which is essentially equivalent to subtask 3, except that only the text is available. dataset the provided dataset comprises 687 memes for training, 63 memes for validating on the so-called development set, and 200 memes for the final testing. all the memes carry textual captions written in english. note that, in the end, we were allowed to use the annotations for the development set, so we had at our disposal a total of 750 annotated memes to use for the training and validation phases. the annotations consist of a list of persuasion techniques for every meme. in subtask 1 there are 20 possible persuasion techniques and 22 in subtask 3. ## results concerning subtask 3, we studied the performance of our dvtt model by comparing the f 1 -scores against the vtte baseline; furthermore, we tried also to train the model using a balanced sampling of the labels and to fine-tune the feature extractors (bert and the resnet-50), using a learning rate of 1/10 with respect to the one used for training the transformer models. using a lower learning rate during the fine-tuning process is a common procedure to avoid model overfitting. we also report the results of slightly different variants of the dvtt model obtained by increasing and decreasing the number of the transformer's encoder / decoder layers: the base architecture contains four layers; we also experimented with two and six. the results of these experiments are reported in for subtask 1, instead, we used the vtte model without visual input, trying out the same experiments performed for subtask 3. in this case, when varying the number of layers, we only considered the transformer encoder ones (there is no decoder in the vtte model). the ablation results on subtask 1 are reported in ## discussion looking at the subtask 3 results in figure 3 reports some examples of predictions from our model for subtask 3. we evidenced in green the true positives and in red the false positives. the model can correctly identify most of the persuasion techniques. however, there are cases where it is probably necessary to access more contextual information to solve the most complex labels. for example, in the second meme from the left, the model outputs the label exaggeration/minimisation probably due to the presence of vague quantities (killed thousands of innocents). it would be necessary to access external data to effectively reason on the complex common sense and historical facts hidden behind these complex memes. ## conclusions in this work, we proposed transformer-based models for tackling subtasks 1 and 3 of the semeval-2021 task 6, concerning the identification of persuasion techniques in memes. in particular, for subtask 3 which involves both images and texts from the meme, we proposed a double visual textual transformer (dvtt) model. this model uses the full power of the transformer architecture; it demonstrated better results than the baseline, which is composed of a single transformer encoder module fed with both images and text. on the public leaderboard, we reached 4th place on subtask 3. using the baseline model, which can process text alone without images, we also tackled subtask 1, reaching 5th place on the public leaderboard. in the future, we plan to improve our visual feature extraction pipeline, using face expression detection and classification and possibly employing ad-hoc trained object detectors suitable for meme images. also, it would be interesting to study the effective reasoning abilities of the proposed models, by leveraging the attention mechanisms of the transformer, possibly integrating the data with a knowledge base of historical facts that helps to create a more suitable context.