CUET - NLP @ D ravidian L ang T ech- ACL 2022: Investigating Deep Learning Techniques to Detect Multimodal Troll Memes (2022)

Abstract: With the substantial rise of internet usage, social media has become a powerful communication medium to convey information, opinions, and feelings on various issues. Recently, memes have become a popular way of sharing information on social media. Usually, memes are visuals with text incorporated into them and quickly disseminate hatred and offensive content. Detecting or classifying memes is challenging due to their region-specific interpretation and multimodal nature. This work presents a meme classification technique in Tamil developed by the CUET NLP team under the shared task (DravidianLangTech-ACL2022). Several computational models have been investigated to perform the classification task. This work also explored visual and textual features using VGG16, ResNet50, VGG19, CNN and CNN+LSTM models. Multimodal features are extracted by combining image (VGG16) and text (CNN, LSTM+CNN) characteristics. Results demonstrate that the textual strategy with CNN+LSTM achieved the highest weighted f_1 -score (0.52) and recall (0.57). Moreover, the CNN-Text+VGG16 outperformed the other models concerning the multimodal memes detection by achieving the highest f_1 -score of 0.49, but the LSTM+CNN model allowed the team to achieve 4^{th} place in the shared task.

## introduction the meme refers to an element of a culture or system of behaviour conveyed from one individual to another by imitation or other non-genetic actions. memes appear in various formats, including but not limited to photographs, videos, tweets, and have a growing influence on social media communication @xcite . images with embedded text are the most widely used form of memes. memes facilitate transmitting ideas or feelings spontaneously. posting and sharing memes have recently become a popular way of disseminating information on social media since memes can propagate information humorously or sarcastically @xcite . propagation of malicious memes and other related activities via memes such as trolling, cyberbullying is rapidly rising @xcite . the implicit meaning of the memes, presence of ambiguous, humorous, sarcastic terms, and usage of attractive, comical, theatrical images have made meme classification even more complicated @xcite . for example, in figure 1 , text and image individually exhibit no means of attack. however, considering both modalities, it insults the persons by directing the age gap in their marriage. to facilitate research in this arena, this work presents our system to classify multimodal troll memes for the tamil language. tamil is a member of the southern branch of the dravidian languages, a group of about 26 languages indigenous to the indian subcontinent. it is also classed as a member of the tamil language family, which contains the languages of around 35 ethno-linguistic groups, including the irula and yerukula languages @xcite @xcite @xcite . tamil is an official language of tamil nadu, sri lanka, singapore, and the union territory of puducherry in india. significant minority speak tamil in the four other south indian states of kerala, karnataka, andhra pradesh, and telangana, as well as the union territory of the andaman and nicobar islands @xcite . it is also spoken by the tamil diaspora, which may be found in malaysia, myanmar, south africa, the united kingdom, the united states, canada, australia, and mauritius. tamil is also the native language of sri lankan moors. tamil, one of the 22 scheduled languages in the indian constitution, was the first to be designated as a classical language of india @xcite @xcite @xcite . we experimented with several deep learning models to extract visual and textual features. after investigating the outcomes, an early fusion approach is employed to combine the features from both modalities. the results indicate that the textual models acquired higher f 1 -score compared to the visual and multimodal counterparts. 2 related work over the past few years, trolling, hostility, offensive, and abusive language detection from social media data have been extensively studied by nlp professionals @xcite @xcite . the majority of these researches were carried out considering only textual information @xcite . however, a meme's existence can be found in an image and text embedded in an image. most previous studies focused on categorizing memes based on unimodal data: text or image. however, this work considers detecting memes from multimodal data: text and image in tamil. pranesh and shekhar (2020) proposed a multimodal framework (memsem) consisting of vgg19 for image features and bert for text features. memesem achieved a better result than all unimodal and multimodal baselines with 67.12% accuracy. gomez et al. (2020) developed a multimodal hate speech dataset containing images and corresponding tweets. the results indicate that the multimodal model (cnn+rnn) was not outperformed the textual model. bucur et al. (2022) employed a 3-branch network for sentiment analysis. they used efficientnetv4 and clip to extract image features, while a sentence transformer was used to get the text features. the system achieved a weighted f 1 -score of 0.5318 with the coral loss function. ## task and dataset descriptions a troll meme is an image with embedded offensive or sarcastic text which degrade, provoke, or offend a person or group @xcite . this work aims to classify troll memes by exploiting the visual and textual information. the task organizers foot_0 provided a dataset having two types of memes (troll and not troll) in tamil @xcite . ## methodology the objective of this work is to identify the troll from multimodal memes. initially, we exploit the visual aspects of the memes and develop several cnn architectures. subsequently, the textual information is considered, and deep learning-based methods (i.e., lstm, cnn, lstm+cnn) are applied for classification. finally, the visual and textual features are synergistically combined to make more robust meme classification inferences. figure 2 depicts the abstract process of the troll meme classification system. ## result and analysis the task's purpose is to categorize troll memes in tamil. we experimented with various visual and textual models to deal with each modality. furthermore, the features from both modalities were merged. the weighted f 1 -score determines the models' superiority. other evaluation criteria, such as precision and recall, are also considered to understand the model's performance better. ## error analysis a detailed error analysis is done on the best model for each modality to gain more insights. confusion matrices are used to analyze the performance (figure 3 ). figure 3c shows that, out of 395 troll memes, the cnn text + vgg16 model accurately categorized 373 images while misclassifying 22 as not-troll. however, this model's actual positive rate is lower than its true negative rate since it correctly classified just 21 not-troll memes and incorrectly classified 251 memes. the vgg16 model also performed well in the visual method, successfully detecting 354 troll memes out of 395. however, the model struggled to identify not-troll memes, correctly classifying only 31 of a total of 272 not-troll memes and incorrectly classifying 241 of the exact total. meanwhile, figure 3a shows that the cnn text model accurately categorized 294 of 395 troll memes, which is lower than the accuracy of other models. in comparison, the model accurately recognized only 72 non-troll memes out of 272. according to the results of the above investigation, all models are biased toward troll memes and incorrectly label more than 73% of memes as trolls. this improper detection is most likely due to the overlapping nature of memes across all classes. furthermore, 80 memes in the train set and 34 memes in the test set were missed embedded captions, making it challenging for textual and multimodal models to predict the actual class. ## conclusion this paper presented a deep learning model for detecting troll memes in tamil. we experimented with visual, textual, and visual-textual fusion techniques. results revealed that the visual approach obtained the highest weighted f 1 -score of 0.50, whereas the textual approach (lstm+cnn) achieved 0.52 f 1 -score. however, after aggregating features from both modalities, we noticed a slight drop in the model performance. the combined cnn-text+vgg16 model acquired the maximal weighted f 1 -score (0.49) with multimodal approach outperformed other models. it will be interesting to catch how the multimodal fusion performs after extracting the visual and textual features with state-of-the-art models. we aim to investigate transformer-based models (e.g., vision transformer, indicbert, mbert, xml-r, electra, muril) with the extended dataset in the future.