NLP - CUET @ D ravidian L ang T ech- EACL 2021: Investigating Visual and Textual Features to Identify Trolls from Multimodal Social Media Memes (2021)

Abstract: In the past few years, the meme has become a new way of communication on the Internet. As memes are in images forms with embedded text, it can quickly spread hate, offence and violence. Classifying memes are very challenging because of their multimodal nature and region-specific interpretation. A shared task is organized to develop models that can identify trolls from multimodal social media memes. This work presents a computational model that we developed as part of our participation in the task. Training data comes in two forms: an image with embedded Tamil code-mixed text and an associated caption. We investigated the visual and textual features using CNN, VGG16, Inception, m-BERT, XLM-R, XLNet algorithms. Multimodal features are extracted by combining image (CNN, ResNet50, Inception) and text (Bi-LSTM) features via early fusion approach. Results indicate that the textual approach with XLNet achieved the highest weighted f_1-score of 0.58, which enable our model to secure 3rd rank in this task.

## introduction with the internet's phenomenal growth, social media has become a platform for sharing information, opinion, feeling, expressions, and ideas. most users enjoy the liberty to post or share contents in such virtual platforms without any legal authority intervention or moderation @xcite . some people misuse this freedom and take social platforms to spread negativity, threat and offence against individuals or communities in various ways @xcite . one such way is making and sharing troll memes to provoke, offend and demean a group or race on the internet (mojica de la vega and @xcite . although memes meant to be sarcastic or humorous, sometimes it becomes aggressive, threatening and abusive @xcite . till to date, extensive research has been conducted to detect hate, hostility, and aggression from a single modality such as image or text @xcite . identification of troll, offence, abuse by analyzing the combined information of visual and textual modalities is still an unexplored research avenue in natural language processing (nlp). classifying memes from multi-model data is a challenging task since memes express sarcasm and humour implicitly. one meme may not be a troll if we consider only image or text associated with it. however, it can be a troll if it considers both text and image modalities. such implicit meaning of memes, use of sarcastic, ambiguous and humorous terms and absence of baseline algorithms that take care of multiple modalities are the primary concerns of categorizing multimodal memes. features from multiple modalities (i.e image, text) have been exploited in many works to solve these problems @xcite . we plan to address this issue by using transfer learning as these models have better generalization capability than the models trained on small dataset. this work is a little effort to compensate for the existing deficiency of assign task with the following contributions: ## related work in the past few years, trolling, aggression, hostile and abusive language detection from social media data has been studied widely by nlp experts @xcite @xcite . majority of these researches carried out concerning the textual information alone @xcite . however, a meme existence can be found in a basic image, text embedded in image or image with sarcastic caption. very few researches have investigated both textual and visual features to classify meme, troll, offence and aggression. suryawanshi et al. (2020a) developed a multimodal dataset to identify offensive memes. authors have combined text and image both modalities to detect an offensive meme. their model obtained f 1 -score of 0.50 in a multimodal setting with lstm and vgg16 network. a unimodal framework utilized the image features to detect offensive content in image (gandhi et al., 2019). a multimodal framework is proposed by (pranesh and shekhar, 2020) to analyze the underlying sentiment of memes. they used vgg19 model pre-trained on imagenet and bidirectional encoder representations from transformers (bert) language model to capture both textual and visual features. wang and wen (2015) have combined visual and textual information to predict and generate a suitable description for memes. du et al. (2020) introduced a model to identify the image with text (iwt) memes that spread hate, offence or misinformation. (suryawanshi et al., 2020b) built a dataset 'tamilmemes' to detect troll from memes. in 'tamilmemes' text is embedded in the image, each image is labelled as either 'troll' or 'not-troll'. to perform classification, the authors utilized imagebased features, but their method did not achieve credible performance. ## task and dataset descriptions meme as the troll is an image that has offensive or sarcastic text embedded into it and intent to demean, provoke or offend an individual or a group @xcite ). an image itself can also be a troll meme without any embedded text into it. in this task, we aim to detect troll meme from an image and its associated caption. task organizers 1 provided a dataset of troll memes for 1 https://competitions.codalab.org/competitions/27651 tamil language @xcite . the dataset contains two parts: an image with embedded tamil code-mixed text, and a caption. participants are allowed to use image, caption or both to perform the classification task. we used image, text, and multimodal (i.e., image + text) features to address the assigned task (detail analysis is discussed in section 4). ## methodology the prime concern of this work is to classify trolling from the multimodal memes. initially, the investigation begins with accounting only the images' visual features where different cnn architectures will use. the textual features will consider in the next and apply the transformerbased model (i.e. m-bert, xlm-r, xlnet) for the classification task. finally, we investigate the effect of combined visual and textual features and compare its performance with the other approaches. figure 1 shows the abstract view of the employed techniques. ## result and analysis from the above analysis, it is evident that visual models performed poorly compared to the other approaches. the possible reason behind this might be due to the overlapping of multiple images in all the classes. that means the dataset consists of many memes with the same visual content with different captions in both classes. moreover, many images do not provide any explicit meaningful information to conclude whether it is a troll or not-troll meme. ## conclusion this work present the details of the methods and performance analysis of the models that we developed to participate in the meme classification shared task at eacl-2021. we have utilized visual, textual and multimodal features to classify memes into the 'troll' and 'not troll' categories. results reveal that all the visual classifiers achieve similar weighted f 1 -score of 0.46. transformer-based methods capture textual features where xlnet outdoes all others by obtaining 0.58 f 1 -score. in the multimodal approach, visual and textual features are combined via early fusion of weights. f 1 score rose significantly after adding textual features in cnn and inception models. only bilstm method is applied to extract features from the text in this approach. in future, it will be interesting to see how the models behave if transformers use in place of bilstm. an ensemble of transformers might also be explored in case of textual approach. and evaluation (lrec-2018), miyazaki, japan. european languages resources association (elra). william yang wang and miaomiao wen. 2015. i can has cheezburger? a nonparanormal approach to combining textual and visual information for predicting and generating popular meme descriptions. in proceedings of the 2015 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 355-365, denver, colorado. association for computational linguistics. zhilin yang, zihang dai, yiming yang, jaime g. carbonell, ruslan salakhutdinov, and quoc v. le. 2019. xlnet: generalized autoregressive pretraining for language understanding. corr, abs/1906.08237.