T roll M eta@ D ravidian L ang T ech- EACL 2021: Meme classification using deep learning (2021)

Abstract: Memes act as a medium to carry one’s feelings, cultural ideas, or practices by means of symbols, imitations, or simply images. Whenever social media is involved, hurting the feelings of others and abusing others are always a problem. Here we are proposing a system, that classifies the memes into abusive/offensive memes and neutral ones. The work involved classifying the images into offensive and non-offensive ones. The system implements resnet-50, a deep residual neural network architecture.

## introduction social media are the interactive platforms that are in and around the daily life of most of the people @xcite . memes has become integral part of daily life and they play a crucial role in sociopolitical, cultural and behaviour of the people @xcite . memes are not only media for conveying information or disrupting the socio-political situation but, they serve as the main source of sharing humour and laughter. internet humour can create a logical bonding between people and can act as a common platform for like minded people @xcite . memes with colorful/even much dull background powered by splashy texts can create amazing sense of humour @xcite . on the other hand, meme trolling is considered as online activism (a p k et al., 2020), creating awareness and new kind of marketing too. troll is a person who starts the flame of insulting/ upsetting the feelings of another person on internet. this can be done by posting messages in the online community such as social media or even in private chats. trolling has caused in one's personal and professional life even @xcite . the content in any mainstream media are closely monitored but, for social media there are no such monitoring systems. internet has opened the road to post anything and everything on social platforms. offensive content memes are spreading the emotion of fear, misguided phobia and they are misleading the population as they spend most of their time on the internet @xcite . considering the psychological, socio-political and social impacts of memes, especially memes we are proposing a deep learning based meme classifier that can differentiate the trolling one's from non-trolling counterparts. ## dataset the dataset used here, 'tamil troll memes' dataset, which was made available by @xcite . the dataset contained total of 2,969 im-ages of memes, most which contained text embedded images. 4 methodology residual networks are playing a crucial role in image classification. they find a way to overcome the gradient loss, which are a major problem in deep neural networks. residual networks, which uses residue from the previous layers to prevent loss of gradients due to depth of the network @xcite . here we are using a deep residual neural network based on resnet architecture. the architecture has 50 layers, with which the network can grasp the features from different parts and levels of the image. the images are resized to 64x64, with all three color channels, are fed into the algorithm, which then passed to sub-sequent layers of the neural network to finally give the output. softmax activation in the last layer gives the probability value for each class and class with max probability is selected as output. we used fastai library, with which we are able to train our model much faster @xcite . the framework provides easy to access layered api architecture, which has the apis to access most of the deep learning algorithms. pause and train architecture @xcite helps to improve the control over model training as one can have a look at loss, accuracy and other parameters. manual early stopping mechanism helps to monitor the model improvement easily @xcite which is integrated as a part of fastai library, which employs cycle training that allows to monitor each cycle and freeze the weights if required. ## results and evaluation the work was performed to classify the memes in to trolling and non-trolling. the resnet-50 model which was trained on the dataset containing 2349 images belonging to two different classes, for 50 epochs. the variation of loss and accuracy for image classification are shown in the figure below. to analyze the results of the model, a confusion matrix was constructed, and the weighted f1 is calculated. the model trained with binary cross entropy loss function for 10 cycles with imagenet weights and performed with the following results, the values for validation are more than that of training, even though it looks like overfitting the values remained same for the k-fold validation also, therefore the model is not overfitting over a set of data but it suffers lack of data, which we think can be improved with accumulation of data from different sources. however the values in the test are much lesser compared to that of training and validation, which can be result of model overfitting. ## limitations during the course of work we faced many challenges and observed some limitations. it is hard to identify sarcasm. most of the times text and the expression in the image contradict each other and create a wave of sarcasm. an example of such meme is where a character is associated with "ahaan". this is highly contextual as this was part of a tamil movie comedy scene. the text/caption in the image is in tamil but written in italic, and mixture of tamil and english will also pose a huge problem while building the model. ## conclusion memes poses as a medium which are powerful enough to disrupt politics, induce and spread humour. in the mean time they are capable of causing considerable harm to socio-political and personal health of individuals via trolls. our work, meme classification using deep learning sheds light towards filtering the memes to trolling and nontrolling ones with training weighted f1 of 0.87, validation weighted f1 of 0.92 and overall testing accuracy of 0.48. there's work needed to be done as a part of building model with better generalization. the work helped us to understand the impacts and effects of the memes in day today life of the people and how they evolved to be an integral part of the internet and social media.