V olta at S em E val-2021 Task 6: Towards Detecting Persuasive Texts and Images using Textual and Multimodal Ensemble (2021)

Abstract: Memes are one of the most popular types of content used to spread information online. They can influence a large number of people through rhetorical and psychological techniques. The task, Detection of Persuasion Techniques in Texts and Images, is to detect these persuasive techniques in memes. It consists of three subtasks: (A) Multi-label classification using textual content, (B) Multi-label classification and span identification using textual content, and (C) Multi-label classification using visual and textual content. In this paper, we propose a transfer learning approach to fine-tune BERT-based models in different modalities. We also explore the effectiveness of ensembles of models trained in different modalities. We achieve an F1-score of 57.0, 48.2, and 52.1 in the corresponding subtasks.

## introduction memes are text superimposed on graphics that convey messages in the form of jokes, sarcasm, etc. in the current era of the internet and social media, they are very quick to spread. if used as a part of a disinformation campaign, it can be quite tricky to notice the agenda behind them and has the potential to influence a large mass of people without them realizing it @xcite @xcite . to this end, semeval 2021 task 6 @xcite focuses on identifying such persuasive techniques @xcite in a multimodal (visuallinguistic) setting. it consists of three subtasks that enable the participants to study the problem in each modality. only the english textual cues are used in tasks a and b, while the visual cues are also used in task c. task b is a modification of task a which further requires predicting the spans for each identified technique as well. we experiment with bert @xcite ) based unimodal models for tasks a and b. since they are state-of-the-art models for natural language understanding, they are a good choice for understanding the complex propaganda techniques in texts. transformers @xcite have limited sequence length, which limits their performance on longer data, but in the case of memes, the textual content is also very limited. for task c, we experiment with visual-linguistic (vl) models like uniter @xcite , @xcite , lxmert @xcite for the crossmodal understanding of memes. we further explore the effectiveness of ensembling models trained in different modalities. the code for all subtasks is available at http://github.com/kshitij98/ multimodal-propaganda.git ## background propaganda aims to push biased agendas to influence people's mindsets. it is successful in achieving its goal by hiding its way through any of the numerous media platforms available in the current world. a major factor behind the success of such campaigns is the boom of the internet and social media in recent years. another factor being the difficulty to spot such techniques manually because of the high volume of text produced and the unnoticeable nature of such content. with the recent interest of the research community in "fake news," the detection of persuasive techniques or highly biased texts has emerged as an active research area. some of the previous work in this direction analyzes the general pattern of propaganda @xcite , performs analysis at a document level @xcite and a fine-grained analysis of the text @xcite . however, most previous work analyses the techniques in a textual unimodal setting only. this work studies propaganda techniques in a new age domain like memes. meme classification task can be considered a combined vl multimodal problem. it is similar to the currently popular vl problems like visual question answering @xcite , visual commonsense reasoning @xcite and visual entailment @xcite , as we have to classify semantically correlated text with that of the visual content in the image. hence a crossmodal approach under vision and text should perform better than unimodal architectures. basic vl approaches are based on simple fusion techniques in the form of early or late fusion to correlate unimodally trained visual and textual models. however, in an ideal scenario, a multimodally trained model should be more effective in detecting persuasive techniques in memes. with the rising interest in vl problems, recent work attempts to study similar problems in a vl multimodal setting @xcite @xcite . ## system overview systems proposed for all the tasks use bert-based models with task-specific modifications. the proposed systems are explained in detail below: ## results we conduct several experiments to compare the performance of models on each of the tasks. detailed information can be seen in for task a, roberta large is the best performing model on the task. although the average performance of the model is worse than bert large on the test set, the maximum performance is still better. similarly, for task b, roberta large outperforms other models by a large margin. for task c, we trained the textual models on visual-linguistic labels for a fair comparison with the other vl models. unexpectedly, they also show decent performance and surprisingly perform better than the multimodal transformers. finally, the ensemble models are trained with their textual model trained on textual labels and the vl model trained on vl labels. they outperform their unimodal components and end up with a fair performance increase in each of the cases. during the post-evaluation phase, we conduct more experiments and report the corresponding performances on the test set for the best-validated models on the dev set. we realize that our submitted model performed worse on the test set in task c because the checkpoint used for test set predictions was different from the best validation checkpoint. the final values we achieve on the test set by using the best validation set checkpoints are 57.56, 48.23, and 55.68 by using roberta large , roberta large and roberta large + uniter, respectively on all the subtasks. a class-wise comparison is carried out for all models. since several of the classes have very low positive samples, it is difficult to draw conclusions because of the high variance in performance due to different model initializations, so we compare the most frequently occurring labels only. the difference in the performance of different models is illustrated in figure 4 . for this comparison, we train the textual model on multimodal and textual labels to compare the capability of the models. ideally, the textual models should be trained on textual labels only to learn more relevant features. the ensemble is trained with a textual model trained on textual labels and a vl model trained on vl labels. the comparison shows that the multimodal models are performing much better for detecting smears, which has various samples which require visual cues as well. another observation is that roberta is performing better on loaded language and name calling / labelling. training an ensemble of the textual model and the multimodal is helping the model perform better in all classes. to further study the models, we report the performance for each of them in different modalities. to measure the textual performance of the models, we compare the sets of labels in task a and c to shortlist labels which were identified only after the presence of visual cues. we do not consider those predictions and calculate the f1-micro score for the remaining subset of the predictions. similarly, for measuring the visual performance, we do not consider the labels which were identified by just using the textual cues. the performance of several runs of the models is compared in figure 5 . the comparison also supports the claim that ensembling models trained in different modalities help to learn from the best of both worlds. ## conclusion although detecting persuasive techniques in memes is a multimodal problem, often, most of the techniques can be identified by just using the textual cues from the meme. since vl models are still in their nascent stages, powerful textual models help with solving the problem at hand. future work can be done to improve these kinds of problems that are not multimodal in the truest sense. the ensembling method used in our model is very basic; better architectures can be explored to continue this line of work.