T eam X @ D ravidian L ang T ech- ACL 2022: A Comparative Analysis for Troll-Based Meme Classification (2022)

Abstract: The spread of fake news, propaganda, misinformation, disinformation, and harmful content online raised concerns among social mediaplatforms, government agencies, policymakers, and society as a whole. This is because such harmful or abusive content leads to several consequences to people such as physical, emotional, relational, and financial. Among different harmful content trolling-based online content is one of them, where the idea is to post a message that is provocative, offensive, or menacing with an intent to mislead the audience. The content can be textual, visual, a combination of both, or a meme. In this study, we provide a comparative analysis of troll-based memes classification using the textual, visual, and multimodal content. We report several interesting findings in terms of code-mixed text, multimodal setting, and combining an additional dataset, which shows improvements over the majority baseline.

## introduction social media have become one of the main communication channels for the propagation of information through textual, visual, or audio-visual content. while the content shared on social media creates a positive impact, however, there are also content that spread harm and hostility @xcite , including abusive language @xcite , propaganda @xcite cyberbullying @xcite , cyber-aggression @xcite , and other kinds of harmful content @xcite . the propagation of such content most often is done by an automated tool, troll, or coordinated groups, which target specific users, communities (e.g., minority groups), individuals, and companies. to detect such content there has been effort to develop automatic tools (see most recent surveys on disinformation @xcite , rumours @xcite , propaganda @xcite , and multimodal memes @xcite , hate speech @xcite , cyberbullying @xcite , and offensive content @xcite . in addition, shared tasks has also been organized in the past years addressing factuality, fake news and harmful content @xcite . among other social media content, recently, the uses of internet memes became popular and they are often shared for the purpose of humor or fun with no bad intentions. however, memes are also created and shared with harmful intentions. this include attack on people based on the characteristics such as ethnicity, race, sex, gender identity, disability, disease, nationality, and immigration status @xcite . there has been research effort to develop computational method to detect such memes, such as detecting hateful memes @xcite , propaganda @xcite , offensive @xcite , sexist meme @xcite and troll based meme @xcite . in this study, we focus on troll-based meme classification based on the dataset released in the shared task discussed in @xcite . while meme contains both textual and visual elements, hence, we investigate textual, visual content and their combination using different pretraind transformer models. in addition, we explored combining an external dataset, and use of code-mixed text (i.e., tamil and english) extracted using ocr. note that the text provided with the dataset is transcribed in latin. while prior work focuses on the text provided with the dataset, here, we also follow a different strategy, directly using the text from the ocr without any cleaning. our contributions include: prior work on detecting harmful aspects of memes include categorizing hateful memes @xcite , antisemitism @xcite and propaganda detection techniques in memes @xcite , harmful memes and their target @xcite , identifying protected category such as race, sex that has been attacked @xcite , and identifying offensive content @xcite . among the studies most notable effort that streamlined the research work include shared tasks such as "hateful memes challenge" @xcite , detection of persuasion techniques @xcite and troll meme classification @xcite . the work by @xcite investigates antisemitism along with its types by addressing the tasks as binary and multi-class classification using pretrained transformers and cnn as modalityspecific encoders along with various multimodal fusion strategies. @xcite developed a dataset with 22 propaganda techniques and investigates the different state-of-the-art pretrained models and demonstrate that joint vision-language models perform best. @xcite address two tasks such as detecting harmful memes and identifying the social entities they target and propose a multimodal model, which utilizes local and global information. @xcite goes one step further than a binary classification of hateful memes -more fine-grained categorization based on protected category (i.e., race, disability, religion, nationality, sex) and their attack types (i.e., contempt, mocking, inferiority, slurs, exclusion, dehumanizing, inciting violence) using the dataset released in the woah 2020 shared task. 1 fersini et al. (2019) studied sexist meme detection and investigate textual cues with a late-fusion strategy, which suggest that fusion approach performs better. the same authors also developed a dataset of 800 misogynistic memes covering different manifestations of hatred against women (e.g., body shaming, 1 github.com/facebookresearch/fine_ grained_hateful_memes stereotyping, objectification and violence), which are collected from different social media @xcite . in the "hateful memes challenge", the participants addressed the hateful meme classification task by fine-tuning the state-of-art multi-modal transformer models @xcite and best system in the competition used different unimodal and multimodal pre-training models such as vi-sualbert (li et al., 2019) vl-bert @xcite , uniter @xcite , villa @xcite and ensembles @xcite . the semeval-2021 propaganda detection shared task @xcite was organized with a focus on fine-grained propaganda techniques in text and the entire meme, and from the participants' systems, they conclude that multimodal cues are important for automated propaganda detection. in the troll meme classification shared task (suryawanshi and chakravarthi, 2021), the best system used resnet152, bert with multimodal attention, and the majority of the system used pretrained transformer models for text, cnn models for images, and early fusion approaches. ## results and discussion in for the image-only experiments, we obtain a comparative performance with resnet101 and ef-ficientnet (effnet (b1)). our experiments on multimodel experiments provide better results compared to text and image modalities as shown in the with additional data, results improved significantly for all image-only models. it confirms our observation that visual elements in memes share information across different datasets, and possibly different cultures too. our future plan is to apply the data augmentation technique for multimodality experiments and also we have a plan to explore the performance of popular multimodel algorithms. during the shared task participation, we submitted one run, using only image modality, where our system was not showing promising performance. however, our subsequent experiments and analysis show several promising directions in terms of original code-mixed data and additional data from the other task. ## conclusion and future work we present a comparative analysis of different modalities for the troll-based meme classification task. we show that the multilingual model capture more information for code-mixed text than its latin counterpart. we present higher performance with multimodality compared to unimodal models. our experiments also suggest that additional data from the other task helps in capturing visual information. in the future, we plan to further develop multimodal models that can capture information in code-mixed noisy conditions.