Multimodal Code-Mixed T amil Troll Meme Classification using Feature Fusion (2022)

Abstract: Memes became an important way of expressing relevant idea through social media platforms and forums. At the same time, these memes are trolled by a person who tries to get identified from the other internet users like social media users, chat rooms and blogs. The memes contain both textual and visual information. Based on the content of memes, they are trolled in online community. There is no restriction for language usage in online media. The present work focuses on whether memes are trolled or not trolled. The proposed multi modal approach achieved considerably better weighted average F1 score of 0.5437 compared to Unimodal approaches. The other performance metrics like precision, recall, accuracy and macro average have also been studied to observe the proposed system.

## introduction social media is a technology where people share information, idea and their opinions to the virtual group of people. these contents uses internet to reach the people via electronic medium, which includes photos, videos, and textual information @xcite . these electronic social media contents are accessed through computers, mobiles, tablets via the web based applications @xcite @xcite . government keep an eye on the conventional media contents, because the information shared in conventional media are monitored for trolled contents @xcite . but in case of social media there is no strict laws or methodologies to monitor the internet contents. one of the most important way of sharing our thought is either via text messages or via images. there are so many contributions for social media contents like text, emojis, info graphics, charts and photographs. in this, photographs and texts plays a key role. meme is an element of behaviour imitation passed from an individual to another. meme was first coined by british evolutionary biologist richard @xcite . it is an idea to mutate, replicate or imitate others behaviours to pass an information. it is an art of writing human creativity. memes with text and images could be detected for trolled or not trolled. visual question answering, image captioning @xcite and identifying the images are categorised as classification problem that depends both on the individual inputs. this type of classification task is portrayed in troll meme classification in tamil @xcite ) @xcite . this task focuses on memes as trolled or not trolled. the idea of trolled is determined by the text and image interaction as shown in figure 1 2 . authors handled the multi modal input data for different applications such as hateful meme detection @xcite and adversarial meme detection @xcite . particularly the figure 1 shows that the event happened in recent times with popular occurrences/offensive texts are get trolled. but in case of 2 is not an recent time activity happened does not contain any harm or offensive messages that are not get trolled. meme is a good imitation of a real world problem. here the task on troll meme classification in tamil is a task of classifying the memes into hate and non hate images. troll is nothing but a offensive message or disruptive message in the social media. the language mentioned in the task is tamil. for the text memes, corresponding image memes also given. by using the texts and the images, the proposed system need to classify whether the meme is trolled or not. the evaluation metric from the data set is taken as weighted f1 score. in this research, we have defined the research 1 figure 2: example image for non troll meme figure 1 is trolled and figure 2 is not trolled image. questions and the same have been addressed in the next upcoming sections. research question 1: to study the performance of the pretrained word embedding like glove for transliterated content. research question 2: analyse the performance of the glove embedding with other language based pretrained models research question 3: to study the performance of the system with the deep learning architectures with glove embeddings. to solve the research questions, extensive literature study have been conducted and reported. this section describes about the introduction and section 2 and 3 talks about the related works and methodologies used in the classification task. section 4 describes the results and discussion and last section deals with conclusion of the paper. ## related works multimodal representation have recently gained good attention due to the uni modals @xcite poor performance on the applications such as image captioning @xcite , visual reasoning @xcite , memes classification and visual question answering @xcite . multimodal task involves visual and language understanding between the two unimodalities. maximum works carried on multimodal systems have either one is late fusion (lf) @xcite or early fusion (ef) @xcite . the other fusion techniques are hybrid multimodal fusion, model-level fusion, rule-based fusion, classification-based fusion, estimation-based fusion are reported in the literature survey @xcite . late fusion process involves two unimodal system independently till before the last layer and fuse the their decisions for further processing. early fusion approach uses two modalities with complex approaches within the model architectures. early fusion of the features provide better representation for further processing in accomplishing the task. some of the steps are followed in nlp type of task are pre-processing , feature engineering, and dimensionality reduction. preprocessing involves stop word removal, tokenization, spelling correction, noise removal, remove numbers, stemming and lemmeatization. for english memes @xcite these types of pre processings are applicable, but in case of non english memes some other type of pre processing be used to clean the input data. applied stop word removal and special characters removed on transliterated dataset content. feature engineering is used to extract useful features from the input data. some of the feature engineering word embedding approaches are glove, word2vec, ngram and term frequency and inverse document frequency. dimensionality reduction technique used to reduce the dimensionality of the large data sets into a smaller data set which may contain the most important features from large data set. for social media comments @xcite @xcite , different transformer based approaches and attention based approach are proposed. many researchers have implemented classifiers such as knn, naive bayes,svm and ensemble classifiers @xcite . recently deep learning (dl) methods such as rnn, cnn and transformer models @xcite @xcite , bilstm-crf @xcite and hybrid convolutional bidirectional recurrent neural network for sentiment analysis @xcite attains better results compared to machine learning models due to ability to model complex representations inside the data. image classification on meme classification starts with the pre processing steps like resize the image, noise removal, rgb2gray scale conversion and segmentation process. after pre processing feature extraction involves color extraction, texture extraction, shape and deep feature extraction. for short text classification task @xcite , proposed cnn with bi-gru on open directory project (odp) dataset and obtained 82.04% accuracy. @xcite @xcite studied the performance of transformers on the code-mixed social media contents. @xcite proposed translation based offensive content identification on tamil text using pretrained word embedding. muril pretrained embeddings were used by the translated content for classification. in (rajalakshmi and agrawal, 2017), authors proposed relevance based metric for code-mixed language by using statistics based approach. @xcite proposed transformer based approach for identification of offensive content on social media tamil comments. in @xcite , the authors proposed transfer learning approach for movie review by using bidirectional gate recurrent unit(bgru). the features from bert embeddings are used as features for transfer learning approach. @xcite proposed muril based approach for youtube comments for offensive content identification. @xcite created dataset for offensive span identification for code-mixed social media tamil contents. @xcite @xcite @xcite traditional machine learning algorithms and text embedding methods @xcite have been proposed on short text classifications. transformer based approach @xcite and xgboost @xcite ) based approaches were used on depression detection using signs. aspect-based approach @xcite is studied on sentiment analysis. most of the image classifier models uses convolutional neural network (cnn) architecture for feature extraction, which automatically extracts the features from the data inputs. mobilenetv2 @xcite pre-trained model uses these automatic feature extraction of deep learning model with depth wise convolution and point wise convolution for reducing the parameters. in our multimodal classifier approach, the proposed system used cnn with bidirectional long short term memory (bi-lstm) as text classifier and mo-bilenetv2 as image classifier for troll meme classification with multimodal approach. a detailed experimental study has been conducted to explore the role of cnn, bi-lstm and combination of both. cnn works well for short text analysis in english @xcite . to explore the role of cnn for our application we adapted the same to our approach. mobilenetv2 is very effective feature extractor for image classification problems better than vgg and resnet. so we have adopted both to our proposed architecture. ## results and discussion from from our results, the recall value for troll text and images shown higher results compared to precision on all the unimodal approaches. because all the images on the data set contains shown very few false negatives. it means that troll categories are classified almost correctly. we need to concentrate more on non troll contents of both text and images to improve the performance of the system. on comparing the results of the recall on unimodal and multimodal approach, the image classifier classified more troll images in terms of true category. for non troll category all the modals have higher precision score than recall score. statistical significance test, fried man test is conducted for our proposed architecture with other two unimodal approaches. null hypothesis h0 and different alternate hypothesis h1 is defined and computed the score. for 10 instances and level of significance as 0.01 is observed on fried table and obtained fr as 9.60. we have populated the table with respective rankings and calculated the score as 72.9. this is greater than 9.60 of table value. so we can accept the alternate hypothesis. this assessment tool is used the test the performance of the proposed system with other systems. ## conclusion combination of sequence based model and pretrained image model showed better performance. cnn-bi-lstm classifier on text input and mo-bilenetv2 on image input combination obtained some better performance on multimodal approach. the glove embedding performance on troll classification shown better performance with deep learning architectures. in order to obtain better result, the multimodal approach shown some better performance than the unimodal approaches. the performance metrics weighted f1 score is chosen to balance the results on the both class labels. weighted average on meme classification gives important to both the classes. by using multimodal classification approach on memes attained a result of 0.5437 weighted f1 score. the performance of the multimodal classifier can be improved with more number of input data and feature extraction on images. the dataset is created on low-resource language tamil and the cultural adaptation details can be considered for future scope.